Upgrading An LLM Version Requires Data Migration Thinking

We moved our internal agency lead-qualification bot from Claude 3 Sonnet to Claude 3.5 Sonnet on a Tuesday. Anthropic called it an upgrade. We treated it like a patch release — bump the model string, redeploy, move on with our day. By Friday, we had dropped 14% of inbound leads in that three-day window, measured by comparing web-form submissions against successful CRM writes in the internal postmortem.

Nothing crashed. No error logs, no 500s, no alerts firing at 2am. The bot kept answering. The CRM pipeline kept humming along like everything was fine. That’s what made it dangerous — a silent failure doesn’t page anyone. It just costs you money until someone in sales finally asks why the pipeline looks thin.

Silent Regressions in JSON Output

The bot’s job was simple: parse an inbound message, extract lead attributes, emit a flat JSON object, hand it to a Python function that writes to the CRM. Claude 3 Sonnet had reliably produced {"name": ..., "company": ..., "intent": ..., "budget_band": ...} for months. Our parser was written against that exact shape, and only that shape.

3.5 Sonnet produced something structurally different often enough to matter. It nested budget_band inside a qualification object. It occasionally added a confidence key nobody asked for. Sometimes it wrapped the whole thing in a lead envelope, because the new model’s training had given it a different instinct for what “helpful” formatting looks like. None of this was wrong, exactly. It just wasn’t what our parser expected — and that gap is where the damage happened.

Our Python code did what brittle glue code does when an expected field moves behind an extra layer of nesting: it threw a KeyError reaching for budget_band where a flat top-level key used to be. That exception got swallowed by a broad try/except further up the stack — our mistake, not the model’s — and the lead silently dropped instead of writing to the CRM. The postmortem matched three days of inbound submissions against CRM write records and found a 14% drop caused by swallowed parser exceptions — with zero signal until sales compared pipeline volume against CRM entries.

The line we wrote into our own postmortem: a model upgrade inside a multi-agent system is not a library dependency bump. It’s a breaking structural change to a contract you didn’t know you had.

The Architecture of Schema Drift

Multi-agent systems run on contracts. Agent A emits a JSON object. Agent B parses it and calls a tool. Agent C reads the tool’s output and decides what happens next. That’s the same basic shape as a service boundary in any distributed system — except the thing generating the payload on one side is a probabilistic model, not a compiler.

LLMs don’t owe you backwards compatibility on output formatting. There’s no semver contract that says “3.5 will format JSON the way 3.0 did.” When a provider fine-tunes a new version, it changes the token probability distribution end to end — which reshapes formatting habits, reasoning paths, and which edge cases get handled inline versus flagged separately. None of that shows up in the model card. Almost all of it shows up in your parser, eventually, at the worst time.

Teams that build multi-agent crews around rigid prompt engineering — “the model will always return exactly this shape because I told it to in the system prompt” — are building on a foundation that can shift under them with any new checkpoint. The better mental model isn’t “I called a function and got a return value.” It’s “I ran a query against a table, and the table’s schema can change without my permission, on someone else’s release schedule.”

Treat it that way, and the rest of the operational discipline follows on its own.

Shadow Traffic and Parallel Staging

We ran into a second, higher-stakes version of this problem on the WaterDoctor project — a sensor anomaly detection system for water quality monitoring, where the output isn’t a CRM record but an alert to a facility manager who may or may not shut down a pump based on what the model just told them.

We needed to move the classification layer from GPT-4-Turbo to GPT-4o. After the lead-bot incident, hot-swapping the model string in production was simply off the table. Instead, we built a parallel staging path: every real sensor reading from production got routed to both models simultaneously, for two weeks, before GPT-4o was allowed anywhere near a customer.

We diffed the two models’ classification boundaries daily. They didn’t agree on everything. GPT-4o grouped certain water quality anomalies differently than Turbo did — some borderline readings that Turbo classified as noise, 4o flagged as early-stage anomalies worth a second look. That’s not necessarily wrong. It might even be an improvement. But “might be an improvement” isn’t something you want to discover for the first time in production, on a system where a false negative means a water-quality anomaly goes unflagged and a false positive means a facility manager starts tuning out alerts from fatigue.

Two weeks of shadow traffic against real sensor data gave us the actual delta between models on our specific distribution of edge cases — not on whatever benchmark the provider used to announce the upgrade. Synthetic test suites are good at catching regressions you already anticipated. They’re close to useless against reasoning drift you didn’t anticipate, because you can’t write a test for a failure mode you don’t know exists yet. In our experience, real production data run in parallel is still the most reliable way to surface that kind of drift before it costs you something.

The Operational Playbook for Model Migrations

A few rules came out of both incidents. We run them as a checklist now, before any model swap, no exceptions:

Lock model versions in production code. Never deploy against a floating alias like gpt-4 or claude-3-sonnet. Pin the dated release string — a floating alias means the provider can change your production behavior on their schedule, not yours.

Treat models as immutable dependencies in your eval pipeline. Change the model, run the full regression suite — the same one you’d run for a database engine version bump. “It’s just a text model” isn’t an exemption.

Write defensive parsers on every agent-to-agent boundary. Missing keys, unexpected nesting, extra fields — all of it should degrade gracefully, not throw and get silently caught three layers up. A payload that doesn’t match the expected schema should trigger a loud, logged, alertable event. Not a quiet drop.

Instrument the parsing layer, not just the API layer. Most observability setups watch for HTTP errors and latency spikes. They don’t watch for “the JSON parsed fine but the values are semantically wrong” — which is exactly the failure mode that got us. Log agent payload failures, including the ones that parsed successfully but drifted from the expected schema, to a central dashboard. Alert on volume anomalies the same way you’d alert on a sudden drop in database write throughput.

The Deprecation Window Reality

Here’s the part that doesn’t go away once the parsers are fixed: providers deprecate models on their own timeline, not yours. When an API provider announces deprecation, the published shutdown date is a hard migration deadline, not an opening offer — the exact window varies by provider and by model family, but the operational problem is the same either way: the moment the notice goes out, you’re on a countdown.

Treat that window the way you’d treat a PostgreSQL major-version migration — budget an actual engineering sprint for it, not a Friday afternoon. Rewrite prompts against the new model’s formatting habits, re-run the regression suite, and stand up shadow traffic if the system is high-stakes enough to warrant it. And if it touches a customer’s water supply or their CRM revenue, it is.

Agentic systems built on rented model weights don’t get to be set-and-forget. The intelligence is rented, and the lease terms change without notice. Budgeting for that churn — as a recurring line item, not a one-off firefight — is what separates a multi-agent system that survives its third model upgrade from one that quietly drops 14% of something important on the way there.