The lakehouse and medallion layers

One platform, three layers of trust

Most data platform failures start with the same question nobody can answer: "which table do I trust?" The medallion architecture answers it by giving every table a layer, and every layer a contract. It is the pattern behind nearly every pipeline we build, so we start the course here.

In this lesson we define the layers, set rules for what belongs in each, and show how the layers map to Unity Catalog names. We do not cover ingestion mechanics (lesson 2) or pipeline syntax (lesson 3).

The lakehouse in one paragraph

A lakehouse keeps data in open table formats on cloud object storage and puts warehouse features on top: transactions, schema enforcement, governance and fast SQL. On Databricks, tables are Delta tables, and Unity Catalog governs them. You get one copy of the data for engineering, analytics and AI, instead of a lake for files and a warehouse for reports with a copy job in between.

Unity Catalog uses a three-level namespace: catalog.schema.object. The object level covers tables, views, volumes, functions and models. Tables and volumes can be managed, where Unity Catalog governs access and the storage lifecycle, or external, where it governs access only. For new work, start with managed tables unless an existing storage contract forces external ones.

The medallion layers

The Databricks documentation describes the medallion architecture as a data design pattern that organises a lakehouse into three layers of increasing quality: bronze, silver and gold. It calls the pattern a recommended practice, not a requirement. We use it because it makes responsibilities obvious.

Three rules keep the layers honest.

1. Bronze is a faithful record. If you can rebuild silver from bronze, a bug in your cleaning logic is a re-run, not a data loss. This is also why bronze keeps fields loose: a source that adds a column or changes a type must not break ingestion.
2. Silver is built from bronze, not from the source. The documentation advises streaming reads from bronze where possible and no writes to silver directly from ingestion. That keeps one path for lineage and one place to fix a mistake.
3. Gold answers questions. A gold table exists because someone asks a business question. If nobody reads it, delete it.

Mapping layers to Unity Catalog

Naming is a design decision, because grants attach to names. Our standard keeps one catalog per environment and domain, and one schema per layer:

dev_sales.bronze.orders_raw
dev_sales.silver.orders
dev_sales.gold.orders_daily
prd_sales.bronze.orders_raw
prd_sales.silver.orders
prd_sales.gold.orders_daily

The environment prefix (dev, test, acc, prd) gives every environment its own catalog, so a pipeline in dev cannot write into production by accident. Access follows the schema: a reader group and a contributor group per schema. Analysts get reader on gold. Pipelines run as a service principal with contributor on the layer they write. Nobody gets write access to bronze except the ingestion identity. We cover the grant pattern in the governance course.

A small example

The following SQL builds the three layers from a raw orders table. It uses standard Databricks SQL on Delta tables. In production you would use the declarative pipelines of lesson 3, but the layer logic is identical.

-- Bronze: keep what arrived, add load metadata. No business rules here.
CREATE OR REPLACE TABLE dev_sales.bronze.orders_raw AS
SELECT *, current_timestamp() AS _loaded_at
FROM   dev_sales.landing.orders_export;
 
-- Silver: enforce types, remove duplicates, drop rows that cannot be used.
CREATE OR REPLACE TABLE dev_sales.silver.orders AS
SELECT DISTINCT
  CAST(order_id AS BIGINT)           AS order_id,
  CAST(customer_id AS BIGINT)        AS customer_id,
  CAST(order_ts AS TIMESTAMP)        AS order_ts,
  CAST(amount AS DECIMAL(12,2))      AS amount
FROM   dev_sales.bronze.orders_raw
WHERE  order_id IS NOT NULL AND amount IS NOT NULL;
 
-- Gold: one table for one business question: revenue per day.
CREATE OR REPLACE TABLE dev_sales.gold.orders_daily AS
SELECT DATE(order_ts) AS order_date, COUNT(*) AS orders, SUM(amount) AS revenue
FROM   dev_sales.silver.orders
GROUP  BY DATE(order_ts);

Note what the example shows: the silver query removes unusable rows. In lesson 3 you will see how to record those drops instead of hiding them.

Where teams go wrong

• Skipping bronze. Loading straight into silver saves a table and costs you the ability to replay.
• Business logic in bronze. The first time a rule changes, you have lost the original.
• Gold per dashboard. Fifty near-identical gold tables is a maintenance burden. Model per domain and question.
• No owners. Every schema needs an owner group, and every table a documented purpose and data owner.
• Ingestion cadence by habit. The documentation notes the trade-off: continuous ingestion has the lowest latency and highest cost, triggered ingestion balances the two, batch is cheapest and slowest. Ask the business what freshness it needs, then pick.

Summary

Use bronze for fidelity, silver for trust and gold for answers. Name layers as schemas inside environment-specific catalogs, so governance follows the structure. Build silver from bronze only.

Next lesson: getting data into bronze with Auto Loader and Lakeflow Connect.