Cost and compute choices

Compute is where the money goes

Storage is cheap. Queries and pipelines are not. On a lakehouse the compute bill follows decisions made by engineers: which compute type a job uses, how large it is, how long it idles, and who may create more. This lesson gives you a way to make those decisions on purpose.

We compare serverless and classic compute, set out what cluster policies control, and describe how to see where spend goes. We do not quote rates. They differ per agreement and change, and the lesson must stay true when they do.

Two ways to run compute

Serverless compute is managed by Databricks. The documentation says Databricks automatically allocates and manages the compute resources, so you do not provision compute in your own cloud account. It covers serverless notebooks, jobs and Lakeflow pipelines, and streaming on serverless. The runtime is versionless: it is upgraded automatically and workloads run on the latest version. Serverless is available by default in most workspaces, and workspaces need Unity Catalog. Billing follows serverless usage, which you can read from system tables such as system.billing.usage.

Classic compute runs in your own cloud subscription: you choose the node types, the number of workers, the runtime version and the autoscaling range. You gain control and carry the operational work: sizing, runtime upgrades, idle time and network integration.

The documentation links to a separate limitations page for serverless, and notes that some features that use serverless infrastructure, such as SQL warehouses and data quality monitoring, are configured separately. Read the limitations page before you move a job that uses unusual libraries or connectors.

A decision rule that works

We default to serverless for new pipelines and jobs, and leave it only for a reason. Valid reasons are specific:

• The workload needs a library, driver or custom data source that serverless does not support.
• A network or compliance requirement demands that compute runs in a network you control, and the serverless networking controls do not meet it. Check the current network connectivity options first. Our post on the private network and serverless egress redesign explains where the boundaries sit.
• You need a fixed runtime version for reproducibility. Our serverless dependency and reproducibility playbook shows how to pin environments on serverless first, since that is often enough.

"We have always used classic clusters" is not a reason. Neither is "serverless costs more". Measure first, with the same job on both, using the billing tables.

Classic compute: policies decide

If you run classic compute, compute policies are your main control. The documentation says policies let workspace admins limit a user's or group's ability to create compute using a set of rules. A policy can require prescribed settings, cap the number of clusters a user can create, hide or fix settings so that more users can create their own compute, limit attributes that drive hourly cost, and enforce cluster-scoped library installs.

Cost-relevant settings include Max compute resources per user and Max DBUs per hour. Lowering a limit does not terminate existing resources, so an admin must stop those by hand.

Policy access works through permissions. Workspace admins have access to all policies. Other users need permission on a policy, and users without access to any policy do not see the policy dropdown. Give permissions to groups, in line with our standards.

A policy definition is JSON. The listing below is schematic: it shows the intent of three rule types, and you should write real definitions against the compute policy reference.

{
  "autotermination_minutes": { "type": "fixed", "value": 20 },
  "autoscale.max_workers":   { "type": "range", "maxValue": 8 },
  "node_type_id":            { "type": "allowlist", "values": ["<approved-node-type-a>", "<approved-node-type-b>"] }
}

Read the intent: idle clusters stop after twenty minutes, nobody can scale beyond eight workers, and only approved node types are available. Put one policy per use case in the bundle, and give your engineers the policy instead of free configuration.

See where spend goes

You cannot manage spend you cannot see. Serverless usage is tracked in system.billing.usage. Use it with tags so that every job and pipeline carries an owner and a use case. Our standards require tags on every resource, such as environment, owner, cost centre and data classification.

-- Schematic: check the system table schema in your workspace for exact column names.
SELECT usage_date,
       sku_name,
       custom_tags['owner']    AS owner,
       SUM(usage_quantity)     AS dbus
FROM   system.billing.usage
WHERE  usage_date >= current_date() - INTERVAL 30 DAYS
GROUP  BY usage_date, sku_name, custom_tags['owner']
ORDER  BY usage_date, dbus DESC;

Review a view like this monthly with the owners. Ask three questions for each top consumer. Is the work still needed? Is it as frequent as the business needs? Is it the right compute type?

Levers that rarely hurt

In our experience these changes lower spend without a trade-off in correctness:

1. Trigger by event, not by habit. A pipeline that runs every ten minutes for data that changes daily wastes compute.
2. Use availableNow style incremental runs instead of a continuously running stream, when latency allows (lesson 2).
3. Stop full refreshes that are not needed. Incremental processing is the point of streaming tables.
4. Right-size dev and test. Small samples, small compute, short timeouts. Do not copy production scale into dev.
5. Remove the unused. Tables nobody reads and jobs nobody uses still cost storage, compute and attention.
6. Pin cost-relevant defaults in the bundle, so a new job starts cheap and a reviewer sees any change.

Guardrails

Add budgets and alerts where your platform supports them, per workspace or per tag, and send them to the owner group. Put an approval step on changes that raise a policy limit. A cost alert is a quality signal: an unexpected rise usually points at a loop, a full refresh or a missing filter.

Common mistakes

• One policy for everyone. Different workloads need different limits.
• All-purpose compute for scheduled work. A person's session should never carry a nightly load.
• No tags. You cannot attribute spend, so nobody owns it.
• Optimising before measuring. Check the billing data first.
• Moving to serverless without reading limitations.

Summary

Default to serverless for new work and leave it for a documented reason. On classic compute, enforce policies for autotermination, scale and node types. Tag everything, read the billing system tables monthly, and use the low-risk levers first.

This completes the seven lessons. The final project asks you to build a governed pipeline end to end.