AI drift testing in production repeatedly evaluates live AI workflows to detect changes in their behaviour and outcomes. It also measures associated changes in token consumption, context size, cost and performance. These changes can originate in the model, prompt, underlying APIs, MCP tools or retrieved information.
At APIContext, we extended our capability around our existing outside-in synthetic monitoring. We want teams to see when a production workflow changes, evaluate whether it still meets their requirements and receive an alert with enough evidence to investigate.
Several sessions at the Nordic APIs Summit in Stockholm gave practical advice for this direction. Speakers working on API contracts, enterprise tools, observability and agent evaluation described different parts of the same operational problem: the environment around an AI application keeps changing after deployment.
An unchanged prompt does not guarantee unchanged behaviour.
A routine API release might add more information to a response. An MCP server might expose additional tools. A revised error message might cause an agent to retry rather than stop. Each change can affect the information the model receives and the decisions it makes.
That is why production drift needs to be assessed across the complete workflow.
Dr. Miriam Greis of adorsys described how API implementations diverge from their OpenAPI descriptions. Her session demonstrated schema-based contract testing to detect differences in responses, status codes and headers. Schema-Based Contract Testing: Preventing API Drift in Practice
These checks establish whether an implementation continues to honour its documented contract. For an AI consumer, there is a further dependency to test: whether the interface still supports the task the agent is trying to complete.
Even a compatible change can affect that task. A larger response can increase context consumption. Different pagination can require additional calls. A changed tool description can influence which operation the agent selects.
The API remains available while the workflow becomes slower, more expensive or less reliable.
Context growth was a recurring concern in Stockholm.
Hugo Guerrero of Solo.io described how injecting tool schemas from multiple MCP servers increases latency and token costs, and can reduce reasoning quality. As teams expand their tool catalogues, the model has more information to process and more possible actions to consider. Solving Context Bloat
Adrien Poupa and Saif Kurdi-Teylouni of MaintainX, presenting with Ahmet Soormally of WunderGraph, showed the scale this can reach. Their static tool definitions could occupy roughly half of a 200,000-token context window.
For me, this creates a clear operational requirement. Teams need to measure how the cost and quality of an established task change as its supporting environment grows.
Token consumption becomes more useful when it is attached to a defined task and an evaluated outcome. A cheaper run that produces an unacceptable result has not improved the service. A more expensive run may be justified if it reliably completes work that previously failed.
The comparison needs to include both.
Hongliu Cao of Amadeus demonstrated why the definition of success also needs attention.
His research examined "corrupt success", where an agent completes a task while violating the procedure required to complete it correctly. In a flight-rebooking example, the replacement flight was booked, but the agent communicated the wrong price and cancelled the original booking without consent.
Across the research scenarios tested, 27 to 78 percent of transactions classified as successful violated at least one procedural constraint. Beyond Status Codes: Enforcing Process Integrity When AI Agents Call Your APIs
A production evaluation therefore needs to examine the outcome alongside the recorded actions. It may need to check consent, permitted access, the order of operations and whether statements to the user agree with the information returned by the tools.
Those requirements also change over time.
A policy update might introduce an approval step. An API might expose a new operation. A tool might become available to a different group of users. The evaluation needs to reflect the current requirement.
Andi Mann of Apica connected these concerns to active synthetic testing. His session described using known transactions to generate evidence about behaviour, then combining that evidence with observability data about performance and operation.
That approach fits our existing monitoring model. A defined scenario provides a repeatable basis for comparison. Running it through the production interface exposes it to the dependencies and controls the service actually uses.
Outside-in execution also establishes whether the intended caller can reach and authenticate to the service. Evaluating the returned evidence then establishes whether the task met its requirements.
For AI drift testing in production, I would start with a small operational scorecard:
- Task completion and compliance with required procedures
- Tool selection, arguments, retries and errors
- Context size, token consumption and cost per accepted outcome
- Workflow latency, including slower runs
- Evaluation scores and their variation across repeated tests
Each measure needs a declared scenario and baseline. Comparisons become difficult to interpret if the task, model, test data and evaluator all change together.
A drift alert should identify the affected scenario, the previous baseline and the evidence that moved outside the expected range. Where available, it should include changes in tool definitions, API responses, model versions or context size. That gives the receiving team a starting point for investigation.
We are also thinking about eval drift.
An evaluation can lose relevance as production changes. Its test cases may omit a new tool or approval requirement. Its expected results may become outdated. If it uses a model as a judge, a change to that judge can alter scores even when the application behaves consistently.
This is a further implication I take from the sessions in Stockholm. The speakers described the changing contracts, tool surfaces and procedures. Our roadmap needs to account for how those changes affect the tests themselves.
That means retaining the versions of the scenario, rubric and judge, checking evaluations against known examples and reviewing coverage when dependencies change. Repeated runs help distinguish persistent changes from normal variation.
The work we are building follows that sequence: execute the production scenario, evaluate its evidence, compare results over time and alert on meaningful change. The evaluation itself remains part of what must be maintained.
For a team changing an underlying API, the useful outcome is specific. They should be able to see whether dependent AI workflows still complete correctly, follow the required controls and operate within acceptable cost and performance limits.
That is the production evidence we want APIContext to provide.