Most enterprise data projects don't stall because of the technology. They stall because of the foundation underneath it. In this series we walk through the ten phases of a production-ready Databricks deployment, in the order you would actually build them, written for the people who sign it off, not only the people who build it.

Phase 1 is identity: who gets in. It comes first for a simple reason. Every later phase, from workspaces and Unity Catalog to cost control, assigns rights to identities. If the identities themselves are messy, every layer built on top inherits the mess.

The question for leadership

Can you say, today, who has access to your data platform, and why?

In a healthy setup that answer comes from one place in minutes. In many organisations it takes a week of spreadsheets, emails and screenshots, and even then nobody is fully sure. If that sounds familiar, identity is your blind spot.

Where it goes wrong

Identity rarely breaks on day one. It erodes, quietly, while the platform grows from a pilot into something the business depends on.

  • Users are added by hand, workspace by workspace. Every new workspace becomes another list to maintain. Leavers are removed from one, forgotten in another, and keep access for months.
  • Rights are given to people instead of roles. A quick grant to an individual is easy to make and impossible to audit. After a year nobody can explain who sees what, or why.
  • Production jobs run on personal accounts. A pipeline scheduled under a colleague's name works until that colleague changes role or leaves. Then it breaks, or worse, keeps running with rights nobody intended.

Five building blocks

Databricks' own identity guidance points to the same foundation we implement for clients.

1. One source of truth

Your identity provider, such as Microsoft Entra ID, owns every user and group. Identities are managed once at the Databricks account level and assigned to workspaces from there, instead of being maintained separately in each workspace.

2. Single sign-on with MFA, for everyone

All users sign in through your identity provider, with multi-factor authentication enforced there. With unified login, that includes account and workspace admins. No separate passwords, no exceptions for the people with the most power.

3. Automatic provisioning

Joiners, movers and leavers flow from the identity provider into the Databricks account automatically, through automatic identity management or account-level SCIM provisioning. When someone leaves, their access ends the same day, without anyone having to remember a list of workspaces.

4. Groups, not individuals

Workspace access and data permissions in Unity Catalog are granted to groups that mirror real roles, such as finance analysts or data engineers. Changing what someone can do means changing their group membership, in one place, with an audit trail.

5. Service principals for production

Production pipelines run as service principals: machine identities with exactly the rights the job needs. Interactive users don't need write access to production data, and a resignation never breaks a pipeline.

Two quick wins

  • Keep the number of admins small. A handful of account and workspace admins, all behind SSO and MFA. Day-to-day group management can be delegated without handing out full admin rights.
  • Make tokens expire. Databricks recommends OAuth over long-lived personal access tokens. Where tokens are still needed, admins can restrict who may create them and cap their lifetime; Databricks suggests keeping it under 90 days.

Four checks for your leadership team

You don't need to read configuration to know whether this phase is in order. Ask these four questions:

  • Does a leaver lose access the same day?
  • Can every permission be traced back to a group?
  • Does any production job still run under a personal account?
  • Can we count our admins on one hand, and does every token expire?

If you can tick off all four, your front door is in order. If not, this is the cheapest phase to fix, and the most expensive one to postpone.

Next up: Phase 2, Workspaces

With identity in place, the next question is how teams stay separated: which workspaces you need, what lives in each, and how development, test and production are kept apart. That is Phase 2.

This article is part of our series Databricks in Production. Want to know where your platform stands? Get in touch with Ferdinand on LinkedIn.