ENNL
Book a call
Courses

Intermediate / Self-paced, Online

Databricks data engineering foundations

From the lakehouse to governed pipelines you deploy as code

Start courseSee the syllabus
Level
Intermediate
Total time
about 6 h
Lessons
7
Labs
7
Last verified
2026-11-11

What you will learn

  • Design bronze, silver and gold layers and map them to Unity Catalog names
  • Choose Auto Loader or Lakeflow Connect per source and handle schema changes
  • Build a declarative pipeline with expectations that warn, drop or fail
  • Orchestrate it with a job that has triggers, dependencies and a reconciliation gate
  • Deploy through dev, test, acceptance and production with a bundle and a pipeline
  • Choose compute and watch spend with policies, tags and billing tables

Who it is for

Engineers who want one coherent way to build, run and ship pipelines on Databricks, from the first table to production.

Data engineers who write SQL and Python and build production pipelines on Databricks

What you need first

SQL and basic Python. You have used a notebook and know what a table is. Free Edition is enough for most labs.

Syllabus

Lessons build on each other, so follow them in order the first time. Under each title you see what you can do by the end of the lesson.

  1. The lakehouse and medallion layers

    Lesson 1 · 45 min · Lab included

    • Explain the lakehouse and the Unity Catalog three-level namespace
    • Assign rules to bronze, silver and gold
    • Map layers to environment catalogs and layer schemas

  2. Ingestion with Auto Loader and Lakeflow Connect

    Lesson 2 · 50 min · Lab included

    • Choose Auto Loader, Lakeflow Connect or a streaming source per source type
    • Explain how the checkpoint gives exactly-once file processing
    • Design a landing zone as a contract with a supplier
    • Decide in advance what each schema change does

  3. Pipelines with Lakeflow Spark Declarative Pipelines

    Lesson 3 · 55 min · Lab included

    • Choose between streaming tables and materialised views
    • Translate older dlt code to the dp interface
    • Add expectations and pick warn, drop or fail per rule
    • Structure a medallion pipeline that you can test

  4. Jobs and orchestration

    Lesson 4 · 40 min · Lab included

    • Model a workflow as a graph of small tasks with dependencies
    • Choose time-based or event-based triggers
    • Make tasks idempotent and run as a service principal
    • Gate publication on a reconciliation task

  5. Bundles and CI/CD through four environments

    Lesson 5 · 55 min · Lab included

    • Describe the top-level keys of a bundle and the four main CLI commands
    • Structure targets for dev, test, acceptance and production with the catalog as a variable
    • Promote a change through four branches by pull request
    • List what the pipeline must check on every pull request

  6. Data quality and monitoring

    Lesson 6 · 45 min · Lab included

    • Separate validity, uniqueness, freshness, completeness and consistency checks
    • Place each check at the layer where it is cheapest to understand
    • Describe what anomaly detection and data profiling do
    • Write a reconciliation query and route alerts to an owner

  7. Cost and compute choices

    Lesson 7 · 45 min · Lab included

    • Compare serverless and classic compute on control and effort
    • Apply a decision rule for choosing compute for new work
    • Describe what compute policies control
    • Use tags and billing system tables to attribute spend

Final project

Capstone: a governed orders pipeline from landing to gold

Build a pipeline that ingests daily order files into bronze, cleans them into silver with expectations, aggregates revenue per day into gold, runs as a job with a reconciliation task and deploys through a bundle with dev and test targets. Use Databricks Free Edition with generated sample files. Document each decision in a short README in plain, short sentences.

The final project applies the whole course. Mark it done when you can tick every item.

What good looks like

  • Bronze keeps raw fields with load metadata and silver is built from bronze only
  • Names follow environment catalogs and layer schemas, with a reader and a contributor group planned per schema
  • At least three expectations exist and each has a documented action: warn, drop or fail
  • The job has dependencies, a timeout, a notification and a reconciliation task that stops publication on a mismatch
  • The bundle has dev and test targets, the catalog is a variable and databricks bundle validate passes
  • The README states which compute type is used and why, and which tags carry owner and use case

Keep going

Learning paths with this course

How we do this for clients

Learn this with your team

We run this course as a live workshop for teams, on-site or online, and adapt the labs to your platform.

See the workshops

Next step

Want your team to learn this together?

We give workshops to company teams and students, on-site or online. Tell us who the participants are and what they should be able to do afterwards.

Book a callSend a messageor email info@nexops.nl directly

Accepting projects for Q4 2026