Bundles and CI/CD through four environments

Clicked work is unreviewable

A pipeline built in the workspace UI works on the day it is built. Six months later nobody knows who changed the schedule, why the cluster size doubled, or how to rebuild the whole thing in another environment. Bundles solve that by describing the project as code: jobs, pipelines, dashboards and their settings live in files, go through pull requests and deploy through a pipeline.

The documentation now calls the feature Declarative Automation Bundles, and says it was formerly known as Databricks Asset Bundles. Many teams, and our own posts, still say Asset Bundles or DABs. They are the same thing. The command line stays databricks bundle.

In this lesson we structure a bundle, define the four environments, and set the pipeline rules that move a change from a feature branch to production. We do not cover Terraform for the surrounding infrastructure. That has its own place in the governance course.

What a bundle is

A bundle packages source files, definitions of Databricks resources (such as jobs, pipelines and dashboards), workspace configuration and tests into one project that deploys to a target environment. It is defined in YAML and is a feature of the Databricks CLI, version 0.218.0 or later according to the documentation.

The lifecycle has four commands you will use daily:

The main file is databricks.yml at the project root. According to the documentation, its top-level keys are:

• bundle: core settings. name is required.
• include: other configuration files or path patterns to load.
• variables: custom values reused through the bundle.
• resources: the workspace objects to deploy, such as jobs, pipelines and dashboards.
• targets: deployment environments such as dev and prod. Exactly one can set default: true.
• workspace: settings such as host and root_path.
• permissions: permissions applied to the defined resources.
• run_as: the user or service principal that runs the bundle.

The documentation's own minimal example shows how a target overrides the base definition. The dev target is the default, so databricks bundle deploy goes to dev, and databricks bundle deploy -t prod goes to production.

bundle:
  name: hello-bundle
resources:
  jobs:
    hello-job:
      name: hello-job
      tasks:
        - task_key: hello-task
          existing_cluster_id: 1234-567890-abcde123
          notebook_task:
            notebook_path: ./hello.py
targets:
  dev:
    default: true
  prod:
    workspace:
      host: https://<production-workspace-url>
    resources:
      jobs:
        hello-job:
          name: hello-job
          tasks:
            - task_key: hello-task
              existing_cluster_id: 2345-678901-fabcd456

A NexOps-style layout

For a domain pipeline, we use one bundle with a pipeline resource, a job resource and four targets. The listing is schematic: the exact keys of the pipeline and job resources depend on your CLI version, so run databricks bundle validate against the current schema.

# databricks.yml (schematic). Replace hosts and identities for each environment.
bundle:
  name: sales-orders
 
include:
  - resources/*.yml          # pipeline and job definitions live in their own files
 
variables:
  catalog:
    description: Unity Catalog catalog for this environment
 
targets:
  dev:
    default: true
    variables: { catalog: dev_sales }
    workspace: { host: https://<dev-workspace-url> }
  test:
    variables: { catalog: test_sales }
    workspace: { host: https://<test-workspace-url> }
  acc:
    mode: production
    variables: { catalog: acc_sales }
    workspace: { host: https://<acc-workspace-url> }
    run_as: { service_principal_name: <acc-service-principal-application-id> }
  prd:
    mode: production
    variables: { catalog: prd_sales }
    workspace: { host: https://<prd-workspace-url> }
    run_as: { service_principal_name: <prd-service-principal-application-id> }

Three details carry the design. The catalog is a variable, so the same pipeline code writes to dev_sales in dev and prd_sales in production. Resource definitions live in resources/*.yml, which keeps databricks.yml short. And acc and prd run as service principals, never as a person.

The four environments and four branches

Our standard uses four environments, dev, test, acc and prd, each with its own workspace, catalogs, Key Vault and service principal. It uses four long-lived branches: dev, test, acc and main, where main deploys to production.

1. Create a feature branch from dev.
2. Open a pull request to dev. The build runs validation and tests. A reviewer approves. Squash merge.
3. The pipeline deploys the bundle to the dev target.
4. Promote by pull request from dev to test, then test to acc, then acc to main. Never skip an environment. Each step deploys to that environment's target.
5. Acceptance and production need two reviewers and an approval gate before the deploy.

For hotfixes, branch from main, deploy through the same pipeline, and merge the fix back down.

Our post on declarative CI/CD with Asset Bundles walks through a full flow, and the one on deploying SAT with bundles through dev, test, acceptance and production shows the same pattern on a real product.

What the pipeline must check

Run these on every pull request:

• Lint and format checks, and unit tests with a coverage gate.
• databricks bundle validate for every changed target.
• Static analysis, dependency and licence scan, and secret scanning.
• Infrastructure checks if Terraform changed.

Deploy only from the CI/CD pipeline, with workload identity federation instead of stored secrets. Do not run databricks bundle deploy from a laptop to test, acc or prd. A deploy from a laptop skips every check you just built.

Keep logic out of the YAML

Bundles describe where and when things run. Put business logic in Python packages or SQL files, and keep notebooks thin: they call the logic. That way you can unit test the logic, and the bundle stays readable. Reuse shared include files for common resources, such as policies, permissions, notifications and tags, instead of copying them between bundles.

Common mistakes

• Deploying by hand "just once". The environments drift, and nobody can explain why.
• Environment names inside code. Use variables and targets.
• A person as run_as. The deployment breaks when that person leaves.
• Permissions to users. Set permissions in the bundle, to groups.
• One bundle for everything. Split by domain, so a change has a small blast radius.

Summary

Describe jobs and pipelines as a bundle, with one target per environment and the catalog as a variable. Promote through four branches by pull request. Deploy only from the pipeline, validate on every pull request and run as a service principal.

Next lesson: data quality and monitoring once the pipelines run.