In Phase 1 we put the front door in order: who gets in. Phase 2 is about what is behind that door. How do you organise the platform so that teams can work freely, while a mistake in development can never reach production?

In Databricks that starts with workspaces. A workspace is the environment where people write code, run jobs and build dashboards. How many you have, and what you put in each, decides how well your teams and environments are separated.

The question for leadership

Can a mistake in development ever touch production data?

If the honest answer is “it depends on who is logged in”, your environments are separated in name only. That is a risk for data quality, for compliance, and for the trust the business puts in its reports.

Where it goes wrong

We see two extremes, and one trap that looks like a solution.

  • Everything in one workspace. It starts as a pilot and grows. Experiments, ad-hoc analysis and production pipelines end up side by side, with the same clusters and the same permissions.
  • A new workspace for every team or project. It feels like clean separation, but after a year there are dozens of environments with different settings, no clear owners and costs nobody can explain.
  • Separation in name only. There is a dev and a prod workspace, but both can read and write the same production data. The walls are there, the doors are open.

Five building blocks

Databricks' own workspace guidance points to the same pattern we implement for clients.

1. Development, test and production apart

At minimum, development and production each get their own workspace, often with a test or acceptance environment in between. New code and new settings are proven before they reach production.

2. Split only for a real reason

A new workspace needs a reason that cannot be solved otherwise: strongly regulated data that needs its own controls, a region where data must stay, or a business unit that must be fully isolated. Databricks explicitly advises against creating workspaces for individual teams or small projects.

3. Catalogs for teams, not workspaces

Teams share a workspace and get their own catalog or schema in Unity Catalog. That gives each team its own space for data, without multiplying environments, settings and costs.

4. Data bound to its environment

With workspace-catalog binding, a production catalog can only be reached from the production workspace. Even a user with the right permissions cannot touch production data from a development workspace.

5. Created from code, named by rule

Workspaces are provisioned through automation, such as Terraform, with clear naming conventions. Every workspace is reproducible, has an owner, and nobody creates one with a click in a console. We come back to this in Phase 7, Infrastructure as Code.

Four checks for your leadership team

You don't need to open the platform to know whether this phase is in order. Ask these four questions:

  • Can development touch production data?
  • Does every workspace have an owner and a reason to exist?
  • When a new team starts, does it get a catalog, or a new workspace?
  • Are workspaces created from code, or by hand?

If the answers are no, yes, a catalog and from code, your environments are properly separated.

Next up: Phase 3, Unity Catalog

With identities and workspaces in place, the next question is who sees which data, down to the table and column. That is Phase 3.

This article is part of our series Databricks in Production. Want to know where your platform stands? Get in touch with Ferdinand on LinkedIn.