Governance as code with Terraform

Click once, explain forever

By now you know the pieces: namespace, groups, grants, policies, shares. If someone creates them by clicking, you have a platform that works and cannot be explained. Nobody knows who granted what, dev and production differ in ways nobody chose, and a fix in one environment never reaches the others.

Governance as code puts every one of those objects in a repository. A change is a pull request with a reviewer and a plan, a deployment is a pipeline run, and the state of the platform is whatever the main branch says. In this lesson we use Terraform with the Databricks provider to describe catalogs, schemas and grants, apply our module rules, and run the pipeline that deploys them.

We do not repeat the Azure infrastructure side: network, storage and workspace. The posts linked below cover it.

What belongs in code

The boundary is lifecycle. Terraform owns the stable platform structure and access. Bundles own the data products that run on it.

The grants resource

The Terraform provider for Databricks has two grant resources, and the difference matters.

• databricks_grants is authoritative for the grants on one securable. Configure it, and Terraform overwrites whatever else exists on that object. Changes made outside Terraform are reset.
• databricks_grant is additive. It manages a single principal's grants and leaves the rest alone.

Each databricks_grants resource needs exactly one securable identifier, such as metastore, catalog, schema, table or volume, and one or more grant blocks, each with a principal (a user name, group name or service principal application ID) and privileges written with underscores, for example CREATE_TABLE.

These examples come from the provider documentation:

resource "databricks_catalog" "sandbox" {
  name    = "sandbox"
  comment = "this catalog is managed by terraform"
  properties = {
    purpose = "testing"
  }
}
 
resource "databricks_grants" "sandbox" {
  catalog = databricks_catalog.sandbox.name
  grant {
    principal  = "Data Engineers"
    privileges = ["USE_CATALOG", "USE_SCHEMA", "CREATE_SCHEMA", "CREATE_TABLE", "MODIFY"]
  }
}

resource "databricks_schema" "things" {
  catalog_name = databricks_catalog.sandbox.id
  name         = "things"
  comment      = "this schema is managed by terraform"
  properties = {
    kind = "various"
  }
}
 
resource "databricks_grants" "things" {
  schema = databricks_schema.things.id
  grant {
    principal  = "Data Engineers"
    privileges = ["USE_SCHEMA", "MODIFY"]
  }
}

The authoritative behaviour is useful for governance: a grant someone adds by hand disappears on the next apply, and you can see it as drift before it does. It is also a hazard. If two resources both declare databricks_grants for the same securable, they fight. Make the choice once per securable.

A module for the reader and contributor pattern

Lesson 3 gave the pattern: per schema, a reader group, a contributor group and an owner group. In Terraform that becomes one module that every domain calls. The listing is schematic: it shows the structure, and you should adapt the resource arguments to your provider version.

# modules/uc-schema/main.tf (schematic)
variable "catalog_name" { type = string }
variable "schema_name"  { type = string }
variable "env"          { type = string }
variable "owner_group"  { type = string }
 
locals {
  prefix = "grp-uc-${var.env}-${var.catalog_name}-${var.schema_name}"
}
 
resource "databricks_schema" "this" {
  catalog_name = var.catalog_name
  name         = var.schema_name
  owner        = var.owner_group           # a group, never a person
}
 
# Authoritative for the schema: only these two groups hold privileges here.
resource "databricks_grants" "schema" {
  schema = databricks_schema.this.id
 
  grant {
    principal  = "${local.prefix}-reader"
    privileges = ["USE_SCHEMA", "SELECT", "READ_VOLUME"]
  }
  grant {
    principal  = "${local.prefix}-contributor"
    privileges = ["USE_SCHEMA", "SELECT", "MODIFY", "CREATE_TABLE", "WRITE_VOLUME"]
  }
}
 
# Additive on the catalog, because several schema modules grant USE_CATALOG on the same catalog.
resource "databricks_grant" "catalog_reader" {
  catalog    = var.catalog_name
  principal  = "${local.prefix}-reader"
  privileges = ["USE_CATALOG"]
}
resource "databricks_grant" "catalog_contributor" {
  catalog    = var.catalog_name
  principal  = "${local.prefix}-contributor"
  privileges = ["USE_CATALOG"]
}

Note the split. The schema grants are authoritative, because the module owns the schema. The catalog grants are additive, because many schema modules share one catalog, and an authoritative resource there would erase the grants of the others.

A root configuration then stays thin, as our standards require: it calls the module and sets variables.

module "finance_gold" {
  source       = "git::https://example.invalid/nexops-modules.git//uc-schema?ref=v1.4.0"  # pinned version
  catalog_name = "prd_finance"
  schema_name  = "gold"
  env          = "prd"
  owner_group  = "grp-uc-prd-finance-gold-owner"
}

Module and root rules

Our Terraform standards apply to governance code as to any infrastructure.

• Reuse before you recreate. One module for one responsibility, such as the schema module above. Extend it with a variable. Do not copy it.
• Version and pin. Tag modules with semantic versions and pin every reference. Pin the Terraform version and the provider versions, and commit the lock file.
• Thin roots. Roots call modules and set variables, and contain no resource logic.
• One state per environment, stored remotely with locking, with restricted access, because state can contain secrets.
• No hard-coded names, IDs or secrets. Use variables and naming modules.
• Tag resources with environment, owner, cost centre and data classification where the resource supports tags.

Our posts on Azure Terraform best practices for a data platform and the open-source Lakehouse Terraform boilerplate show the layout in detail.

The pipeline

Governance code is only as safe as the pipeline that deploys it.

1. Pull request. The author opens a pull request to dev. The build runs terraform fmt -check, terraform validate, a linter, a security scan and terraform plan for each changed environment.
2. Plan in the review. The plan is posted to the pull request. The reviewer reads the plan, not only the diff. A line that removes a grant or replaces a catalog gets attention.
3. Apply only from the pipeline, with the plan from the same run. Nobody applies from a laptop to test, acc or prd.
4. Approval gates before apply in acc and prd. If the plan deletes or replaces a stateful resource, stop and get explicit approval. Dropping a catalog deletes data.
5. Promotion follows dev, test, acc, main, one pull request per step, as in lesson 5 of course 2.
6. Drift detection on a schedule. A scheduled plan that finds changes means someone clicked. Fix the drift in code, not in the portal.

Authenticate the pipeline with workload identity federation to a deployment service principal per environment, with the minimum rights it needs, instead of a stored client secret.

Test the policy

Treat governance like software. Add checks that run on the plan or after apply:

• A policy test that fails if any grant has a person as principal.
• A test that every schema has a reader group, a contributor group and a group owner.
• Tests for row filters and masks with test identities (lesson 4).
• The scheduled governance checks of lesson 5, which verify the live platform against the same rules.

Code that defines rules and queries that verify them is a strong pair: the first states the intent, and the second proves the platform follows it.

Common mistakes

• Two authoritative resources on one securable. They fight on every apply.
• Authoritative grants on a shared catalog. They erase other modules' grants.
• Applying from a laptop. It skips the plan review and the approval.
• Copy-pasted modules. The copies drift.
• State access for everyone. State can hold secrets.
• Hand fixes in production. Fix in code, apply through the pipeline.

Summary

Put catalogs, schemas, grants, ownership and shares in Terraform. Use authoritative grants where one module owns the object, additive ones where several share it, and one tested module for the reader and contributor pattern. Deploy only through a pipeline with plan review, approvals and drift detection.

You now have the whole governance model. The final project asks you to apply it end to end.