ENNL
Book a call
Courses

Advanced / Self-paced, Online

AI agents in production

Tools, memory, evaluation, guardrails, cost and operations, with the model of your choice

Start courseSee the syllabus
Level
Advanced
Total time
about 6 h
Lessons
8
Labs
8
Last verified
2026-11-11

What you will learn

  • Design a bounded agent loop with scoped tools and human approval
  • Separate session state from long-term memory and delete both on request
  • Build an evaluation set, check LLM judges against people and gate releases on it
  • Defend against prompt injection in layers and test the attacks
  • Measure cost per task and use it as a release gate
  • Swap models safely and run agents with a gateway, alerts and runbooks

Who it is for

Teams that move an agent from a demo to production and need the engineering around the model: tools, state, evaluation, safety, cost and operations.

Engineers and architects who build or run AI agents for real users

What you need first

Comfortable with Python and JSON. You have called an LLM API at least once. SQL basics help in lessons 3 and 8.

Syllabus

Lessons build on each other, so follow them in order the first time. Under each title you see what you can do by the end of the lesson.

  1. What an agent is in production

    Lesson 1 · 40 min · Lab included

    • Describe the six steps of an agent loop and why it needs bounds
    • Name the eight building blocks and what you own in each
    • Sketch a bounded loop with an allow list of tools and two handover exits

  2. Tools and MCP

    Lesson 2 · 45 min · Lab included

    • Explain how MCP lists and calls tools and where its security rules apply
    • Split tools into read and write and add approval to the write class
    • Give each tool a scoped identity and grant it the minimum in Unity Catalog
    • Write a strict tool schema with a useful error path

  3. Memory and state

    Lesson 3 · 45 min · Lab included

    • Separate session state from long-term memory and decide what goes where
    • Design memory tables with subject, source and expiry
    • Write memory through a gate to resist poisoning
    • Build deletion on expiry, on request and on correction

  4. Evaluation that gates releases

    Lesson 4 · 55 min · Lab included

    • Build an evaluation set from real cases with expectations and a held-out part
    • Choose between code scorers and LLM judges
    • Measure agreement between a judge and human labels
    • Turn results into a regression gate

  5. Guardrails and safety

    Lesson 5 · 50 min · Lab included

    • Tell direct from indirect prompt injection and name the realistic attack path
    • Apply five defence layers from least privilege to adversarial testing
    • Write deterministic output checks and wrap untrusted content
    • Design an escalation path for blocked or suspicious cases

  6. Cost and tokens

    Lesson 6 · 50 min · Lab included

    • Name where tokens go in an agent request
    • Measure cost per task and find the heaviest step
    • Pull cost levers in an order that protects quality
    • Make cost per task part of the release gate

  7. Bring your own model

    Lesson 7 · 40 min · Lab included

    • Choose a model source per task: provider API, licensed model, open-weight or fine-tuned
    • Reach every model through one interface so a swap is a configuration change
    • Run a safe swap with a baseline, a shadow run and a traffic split
    • Name what typically breaks in a swap and how to catch it

  8. Running agents in production

    Lesson 8 · 50 min · Lab included

    • Run an agent as a service with crash recovery, versioning and separate environments
    • Use a gateway for budgets, limits, policy and logging
    • Choose observability signals and alerts that lead to an action
    • Write runbooks and a kill switch and rehearse them

Final project

Capstone: a governed support-triage agent

Design and build an agent that reads incoming support requests, looks up order status through one read tool, and drafts a reply that a person approves. Deliver the agent loop, the tool definition, the memory design, an evaluation set of at least 30 cases with a regression gate, an injection test set, a cost per task measurement and an operations pack with alerts and runbooks. Use Databricks Free Edition or a local Python environment, with a stub model if you have no model access.

The final project applies the whole course. Mark it done when you can tick every item.

What good looks like

  • The loop has a step limit, a time limit and a token budget, and both exits hand over to a person
  • The tool list is an allow list, every tool has a strict schema and the write action needs approval
  • Session state and long-term memory are separate stores with subject, source and expiry, and one call deletes a subject from both
  • The evaluation set has per-group results, a held-out part and a release gate that fails when quality or safety drops
  • At least five injection cases are in the set and the agent passes all of them
  • Cost per task and the heaviest step are reported, and a budget gate is part of the release rule
  • Runbooks cover at least four situations and name an owner and a kill switch

Keep going

Learning paths with this course

How we do this for clients

Learn this with your team

We run this course as a live workshop for teams, on-site or online, and adapt the labs to your platform.

See the workshops

Next step

Want your team to learn this together?

We give workshops to company teams and students, on-site or online. Tell us who the participants are and what they should be able to do afterwards.

Book a callSend a messageor email info@nexops.nl directly

Accepting projects for Q4 2026