Identity: Entra ID, groups and service principals

Every grant starts with a name

A privilege is a sentence: this principal may do this to that object. If the principal is a loose list of individual users, the sentence goes stale the day someone changes team. Governance at scale needs identities that come from one source, change when the source changes, and are grouped by role.

In this lesson we set up identity the way we do on Azure Databricks: Microsoft Entra ID as the source, groups as the unit of access, service principals for automation, and privileged roles that exist only on demand. We do not cover network access or conditional access policies in Entra ID itself.

Three kinds of identity

The Azure Databricks documentation lists three identity types.

Identities are managed at account level and then assigned to workspaces. The documentation states that all new workspaces are enabled for identity federation by default, and it cannot be disabled. That means you manage identities once at the account, and assign them to as many workspaces as needed. Legacy workspaces without federation use workspace-local groups, which are not account groups and do not work for Unity Catalog grants. Migrating those is the first governance task in an older estate.

An account has limits on identities and groups. The documentation lists defaults of 10,000 combined users and service principals and 5,000 groups, with higher limits for newer accounts and by request. Design groups for roles, not for individuals, and you will not meet them.

Entra ID as the source of record

The documentation recommends syncing identities from Microsoft Entra ID using automatic identity management, enabled by default for accounts created after 1 August 2025. With it you can search Entra ID users, service principals and groups directly in workspace admin settings and add them to the workspace and the account. Entra ID is the source of record, so a change in group membership there is respected in Azure Databricks.

That is the model we use: no manual user management in Databricks.

• Create the groups in Entra ID, with an owner and a naming convention.
• Let automatic identity management bring them in. Where an older setup uses SCIM provisioning, the principle is the same: the identity provider is the source and Databricks is the consumer.
• Do not add users directly to workspaces, catalogs or schemas. A user gets access by being a member of a group.
• Remove people in Entra ID. Offboarding then happens in one place, and the effect reaches Databricks without a second step.

Single sign-on with Entra ID is available for both the account console and workspaces by default, and just-in-time provisioning can create a user on first login. JIT is convenient for users, but access still comes from groups, so a new user sees nothing until a group grants it.

Group naming and structure

A convention makes grants readable and auditable. Ours:

Roles are reader, contributor and owner. We use them in lesson 3. One group for each access level on each scope keeps the model small: you can read a group name and know what it grants.

Nested groups are a trap when they grow without a map. Prefer a flat structure of role groups, and put people in the groups they need. If you need a "finance analysts" group, make it a member of the reader groups it needs and document why.

Service principals for everything that runs alone

Jobs, pipelines, CI/CD and apps run as service principals. The documentation describes service principals as identities for jobs, automated tools and systems such as scripts, apps and CI/CD platforms. Rules we apply:

• One service principal per environment and purpose. For example a deployment principal per environment, and a runtime principal per domain.
• Least privilege. The pipeline principal for silver has contributor on silver and reader on bronze, and nothing else.
• No secrets where federation works. Use workload identity federation for CI/CD, and managed identities where possible, instead of long-lived client secrets.
• Name an owner. The documentation describes service principal managers who can add, update and remove roles on a service principal. Make the manager a group.

When an agent or app acts for a person, it should act with that person's rights, so it cannot read what they cannot read. Our post on on-behalf-of access in Databricks Apps covers the pattern.

Admin roles and PIM

The documentation lists roles with the power to manage identities: account admins, workspace admins, group managers and service principal managers. Account admins can manage every user, service principal and group, and workspace admins have more limited rights within their workspaces.

Admin roles are the highest-risk identities in the platform, so our standards add one rule: privileged access is eligible, not standing. In Entra ID Privileged Identity Management (PIM), a person is eligible for an admin role and activates it for a limited time, with a reason and multi-factor authentication. Acceptance and production activations need approval. Only break-glass accounts have standing access, and they have alerts.

Pair that with the Databricks side: keep the number of account admins small, and give workspace admin to a group that holds the role through PIM, not to named individuals.

Identity attributes

The documentation notes that Azure Databricks can store identity attributes, such as title, department and cost centre, from your identity provider on account users, for use in access policies. It labels this feature Beta. We treat Beta features as pilots: useful for planning, but not the base of a production access model until the status changes.

A quick audit

Check these five things in any estate you inherit:

1. Are there workspace-local groups that hold grants?
2. Are individual users granted privileges directly?
3. Do production jobs run as a person?
4. Who has account admin and workspace admin, and is it standing or eligible?
5. When did someone last review group membership?

The answers tell you where the risk is, usually within an hour.

Summary

Use Entra ID as the source, groups as the unit of access, service principals for automation and PIM for privileged roles. Manage identities at account level, never grant to individuals and name an owner for every group and service principal.

Next lesson: privileges and the reader and contributor pattern.