Bring your own model

The model is a dependency, so treat it like one

Models change faster than any other part of your stack. A provider retires a version, a new open-weight model beats your current one on your task, a client asks for data to stay in a region. If swapping the model means rewriting the agent, you have built a lock-in.

At NexOps, we design the harness so that the model is one line of configuration per task. Prompts, tools, guardrails and the evaluation set stay where they are. In this lesson we explain how to get there and how to swap a model without drama. We do not rank models. Rankings age in weeks, and your task decides the winner.

Where a model can come from

Choose per task, not per company. A classifier step and a drafting step rarely need the same model.

One interface in front of every model

The swap is only cheap if every model is reached the same way. On Databricks there are two relevant building blocks.

Model Serving external models. Model Serving can expose third-party models behind a unified endpoint. You add an external_model block under served_entities with the model name, the provider and the task (llm/v1/chat, llm/v1/completions or llm/v1/embeddings). The documentation lists providers such as OpenAI, Anthropic, Cohere, Amazon Bedrock and Google Cloud Vertex AI, plus a custom provider for OpenAI-compatible proxies. Databricks stores the provider credentials encrypted. If an endpoint hosts several external models, you must define a traffic_config with a share for each.

Unity Gateway. The Agent Bricks documentation describes Unity Gateway as access to frontier and open models through one API, so you can switch models without changing agent code. Gateway features you will use for swaps: access control through Unity Catalog privileges, rate limits and budgets, logging to Delta tables, and traffic splitting with failover across model destinations.

Two caveats from the documentation. The MLflow Deployments CRUD API used in the external-model examples is described as Public Preview, and network connectivity configuration for external models is also Public Preview. Check the current status of each feature before you plan a production rollout around it.

Keep the model out of your code

Whatever platform you use, apply the same pattern in the agent: ask for a role, not a model.

# Schematic configuration, not real syntax of any product.
tasks:
  classify_request:
    model_endpoint: small-model-a        # a role name that the platform resolves
  draft_reply:
    model_endpoint: large-model-b
  extract_fields:
    model_endpoint: our-fine-tuned-model
 
release_gate:
  evaluation_set: real-cases
  quality: must_not_drop
  cost_per_task: within_budget
rollout: shadow_run_then_traffic_split

The agent reads model_endpoint and calls it through the common interface. Changing the model means changing one value and running the gate.

The safe swap, step by step

1. Pin the baseline. Run the current model on the evaluation set and store quality and cost per task. You compare against this.
2. Run the candidate on the same set. Same prompts, same tools, same scorers. Look at the per-group results, not only the average. A model that improves the average and breaks the safety group has failed.
3. Expect prompt differences. Models respond differently to the same instruction. Allow one round of prompt adjustment per candidate, and record it, so the comparison stays fair.
4. Check cost per task. A larger model may need fewer retries. Measure instead of assuming (lesson 6).
5. Shadow run. Send real traffic to the candidate in the background, compare answers, and let users see only the current model's answer.
6. Split traffic. Move a small share of live traffic to the candidate through the gateway. Watch quality scorers on production traces and cost. Increase the share in steps.
7. Keep the exit. Rollback is one step: the task points at the previous model again. Keep the previous configuration deployable for a release cycle.

What breaks during a swap

• Tool-calling format. Models differ in how reliably they produce valid tool calls. Your strict schemas (lesson 2) catch this.
• Context length. A smaller window can truncate history silently. Trace token counts.
• Output style. A new model may be more verbose, which changes cost and may break format checks.
• Safety behaviour. Refusal and injection behaviour differ per model. Re-run the adversarial cases from lesson 5.
• Latency. Measure, since a larger model on a busy endpoint can break a response-time target.

Open-weight and sovereign options

When data must stay in a region you control, an open-weight model on your own infrastructure is an option, and so is a fine-tuned smaller model for a narrow task. The rule from above holds: the self-hosted model must match your evaluation set before it takes traffic. Self-hosting adds work, such as capacity, patching and monitoring, so choose it for a reason: data location, steady high volume or a narrow task.

Where regulation or contracts forbid sending data to a provider, put that rule in the gateway policy, so it applies to every agent and not only to the ones the team remembered to configure.

Summary

Treat the model as a replaceable dependency. Reach every model through one interface, choose per task, and swap with the same evaluation set, the same cost gate, a shadow run, a traffic split and a one-step rollback.

Next lesson: running agents in production, where all of these pieces meet.