If you've shipped an AI agent in production, you may have noticed a pattern: it looks great in staging, earns praise in the first week, then quality quietly decays. Tickets start coming in around day 10. By day 20, CSAT has moved. The team is confused — nothing obvious changed.
This pattern is common enough that we think of it as the "launch peak." Quality at launch is an artifact of attention, not an accurate picture of steady-state performance. Here are the five reasons why.
1. Pre-launch testing creates a selection bias
The test set you used to evaluate the agent before launch was built by your team. It reflects the cases you thought to include — common user intents, known edge cases, representative queries. What it doesn't include is the full distribution of what real users actually ask.
In production, users ask things your test set didn't anticipate. They provide underspecified context, use domain jargon, mix topics mid-conversation, and trigger edge cases that looked extremely unlikely in planning. The agent's real-world distribution is almost always broader than your eval distribution. Quality at launch looks good because you tested for what you could imagine — not for everything users do.
2. Your LLM provider makes silent updates
All major LLM API providers — OpenAI, Anthropic, Google, Mistral — update their models on a rolling basis without announcing every change. The model behind gpt-4o today is not identical to the model behind gpt-4o last month. Fine-tuning datasets change. Safety filters tighten or relax. System prompt handling is adjusted. Tokenization updates shift the effective context window.
These changes are typically improvements on average — they make the model better for the median use case. But your agent is not the median use case. If your prompts were engineered to work around a specific behavior, a model update can silently break that workaround. You won't know unless you're tracking quality continuously.
3. RAG indexes become stale
For agents using retrieval-augmented generation, the quality of retrievals degrades over time. Product documentation gets updated but the index doesn't. New support content is added to the knowledge base but the embeddings weren't recomputed. The underlying data changes while the retrieval layer remains frozen.
When a user asks about a feature that was updated last week and your agent retrieves a document from three months ago, the answer is wrong. Not because the agent is broken — because the context it's grounding in is outdated. This is a systematic quality issue that worsens monotonically over time as the gap between your index and reality widens.
4. Prompt configuration drift accumulates
After launch, the product team inevitably needs to change the agent. A new feature needs a new instruction. A bug in the prompt's handling of a specific case gets patched. The tone needs to match a rebrand. Each of these changes is small and intentional — but the cumulative effect on quality is rarely evaluated end-to-end.
Post-launch prompt changes are almost never subjected to the same evaluation rigor as the initial launch. The system prompt that was evaluated against 200 test cases in pre-production has evolved through six ad-hoc tweaks, none of which were evaluated against more than a few spot-check responses.
The compounding problem: Each individual change to your agent's configuration is evaluated in isolation. The cumulative effect — the current system prompt + current model + current index — is rarely measured as a whole. This is where quality silently erodes.
5. Team attention shifts post-launch
The week before launch, your team is intensely focused on the agent's quality. After launch, attention moves to other priorities. Quality monitoring often drops to a periodic CSAT review, if it exists at all. Problems that would have been caught immediately pre-launch now take days or weeks to surface because no one is actively watching quality signals.
This isn't a failure of process — it's a resource reality. You can't have someone manually reviewing agent output quality every day indefinitely. The solution is automated quality monitoring: statistical signals that tell you when quality has shifted, so your team only needs to engage when something meaningful happens.
What you can do about it
The good news: all five of these causes are detectable with continuous statistical quality monitoring. Run your agent against a stable golden test set daily. Apply SPC rules to the resulting quality scores. You'll see the selection bias close as real production data accumulates, catch silent model updates the day after they happen, detect index staleness through grounding score drift, and notice prompt configuration drift immediately after it's deployed.
The launch peak is not inevitable. It's the result of intense pre-launch attention without a continuous equivalent. Statistical process control provides that continuous attention — automatically.