Jobs and orchestration

From pipeline to process

A declarative pipeline refreshes tables. A business process does more: it waits for a delivery, refreshes the tables, checks the result, publishes a dashboard and tells someone when it fails. Lakeflow Jobs is the Databricks service that orchestrates that sequence.

In this lesson we model a workflow as a graph of tasks, choose triggers, and set the rules that make a job safe to run unattended. We do not cover external orchestrators. If your organisation already runs one, the same design rules apply to it.

What a job is

A job contains one or more tasks. Tasks can depend on each other and form a directed acyclic graph (DAG), which the Jobs interface shows as a graph. The documentation names three task types you will use first: a notebook task, a pipeline task (which runs a Lakeflow pipeline) and a Python script task. It links to a longer list of task types. The authoring interface also supports control flow such as if/else branching and for-each loops, and jobs can pass run-time parameters to tasks.

A typical daily workflow:

1. A pipeline task refreshes bronze, silver and gold.
2. A notebook or script task runs a reconciliation check against the source totals.
3. A downstream task refreshes a dashboard or publishes a share, but only if step 2 succeeded.

Because step 3 depends on step 2, a failed check stops the publication. That dependency is the cheapest safeguard in the whole platform.

Triggers

A trigger starts a job. The documentation describes two families.

Prefer an event-based trigger when the source delivers irregularly. A schedule that runs at 02:00 and finds no file wastes a run. A schedule that runs before the late file arrives produces yesterday's report as today's.

Always pick the time zone explicitly for a schedule, and write it in the job description. Daylight saving time has ended careers.

Design rules for unattended jobs

Apply these rules to every job. They map onto our delivery standards.

Idempotent tasks. A task that runs twice must leave the same result as running once. Declarative pipelines and MERGE statements give you this. Append-only statements with no key do not. Idempotency is what makes a retry safe.

Run as a service principal. Jobs run as a service principal with the least privileges for the work, never as a person. A person leaves the company, and the job stops. Define the identity in the bundle (lesson 5).

Small, named tasks. One task, one purpose. A task that does ingestion, cleaning and publishing in a single notebook cannot be re-run in part, and its failure message is "something failed". Split tasks along the layers of lesson 1.

Timeouts and retries. Give every task a timeout so that a hung run does not block the next one. Use a limited number of retries for tasks that fail for transient reasons, such as a network error, and none for tasks that fail for logical reasons, since a retry will fail the same way.

Notifications. The documentation lists notifications by e-mail or webhook when a job fails or runs long. Send failures to a channel that a person watches, and name the job owner in the message. A notification to a mailbox nobody reads is no notification.

Parameters, not copies. Pass the environment, the date range or the source name as parameters. Do not copy a job per environment. The bundle gives every environment the same job with different values.

No secrets in parameters. Read secrets from a Key Vault-backed secret scope, or avoid them with managed identities.

A dependency example

The graph below is the shape of most of our daily workflows.

wait_for_files (file arrival trigger)
   -> run_pipeline (pipeline task: bronze, silver, gold)
        -> reconcile (script task: compare row counts with the source report)
             -> publish_share (only if reconcile succeeded)
             -> notify_owner (runs on failure of any upstream task)

The reconcile task compares totals with an independent number: the supplier's control file, or a count from the source system. A pipeline that succeeds technically but loads half the rows is the failure that expectations alone do not catch.

Know the limits

The documentation lists limits for Lakeflow Jobs: 2,000 concurrent task runs per workspace, 1,000 tasks per job and 12,000 saved jobs per workspace. Most teams never meet them. If one job grows towards 1,000 tasks, the problem is the design, and you should split it into jobs per domain.

Compute for jobs

Jobs can run on serverless compute or on classic job clusters. We cover the choice in lesson 7. For now, the rule is: never run a scheduled job on an all-purpose cluster that a person also uses. A person closing a notebook should never be able to affect a nightly load.

Monitoring a job

Track three numbers per job: success rate, duration against its usual range, and time since the last success. A job that has silently stopped being triggered shows no failures, only a growing gap since the last success. Alert on that gap. We expand on this in lesson 6.

Common mistakes

• Cron thinking. A schedule that assumes the data has arrived.
• One giant task. You cannot see, retry or reason about it.
• A personal identity as run-as. The job dies with the account.
• No downstream gate. The dashboard refreshes on top of a failed load.
• Retrying logical failures. It burns compute and delays the alert.

Summary

Model the workflow as a graph of small tasks with dependencies, choose event-based triggers for irregular sources, make tasks idempotent, run as a service principal, set timeouts, retries and notifications, and gate publication on a reconciliation task.

Next lesson: bundles, which turn all of this into code that moves through dev, test, acceptance and production.