Teams still shop for models as if the logo on the API is the product. In production, the thing that usually decides whether an agent is useful is not which model you called. It is what you put in the window, in what order, and how much noise you asked the model to ignore.
That sounds like a small operations detail. It is not. A 2024 study evaluating 18 leading models found that every one of them got worse as the input got longer. Not some of them. All of them. The window is a ceiling. Treating it as a floor — filling it because you paid for it — is how agents start missing the one paragraph that mattered.
Advertised context is not working memory
Vendors advertise 128K, 200K, a million tokens. That number means the request will not be rejected. It does not mean the model attends evenly across every token. Rotary position embeddings, which most modern models use, decay over distance: tokens far from the current position get less weight. A million-token window is a million-token intake, with highly uneven attention inside it.
The practical gap is brutal. Teams load fifty retrieved chunks because the window can hold them, then wonder why the model answers from the system prompt and ignores the source in position 14. The model did not "forget." It never weighted that passage enough to use it.
Lost in the middle is a U-shape, not a rumour
Stanford's Lost in the Middle paper, published in TACL in 2024, measured this directly. Models are strong on the beginning of a long prompt, strong on the end, and unreliable in between. Early and late evidence often lands in the 85–95% range. The same evidence moved to the middle can drop well into the 70s, and in some setups closer to a coin flip. That U-shaped curve shows up across model families, including models trained specifically for long context.
Chroma later named the broader pattern context rot: accuracy declines as context grows even when the relevant document is present. With roughly twenty retrieved documents, they saw accuracy fall from the 70–75% range into the 55–60% range. That is not a rounding error. It is a product that looks fine in a short demo and fails once you give it a real corpus.
Length hurts even when retrieval is perfect
A common reply is "then retrieve better." Retrieval helps, and you should do it. It is not sufficient. Follow-up work in 2025 showed that even when the model can find the right span — and even when you place that span in the best positions, the start and the end — sheer length still taxes performance on math, question answering, and coding. Distractors make it worse. Length alone is enough to make it worse.
So the job is not "fit more into the window." The job is to keep the window short, ranked, and honest about what the model is being asked to do.
The cost curve is not a side issue
Every extra token is billed, and every extra token is processed. Long prompts are slower and more expensive in a way that is not linear once you count retries, tool calls, and the human time spent checking answers the model should have had in context. A cheaper model with 4,000 well-chosen tokens will often beat an expensive model with 80,000 unsorted ones, both on quality and on the invoice.
If the business case for the agent depends on margin, window design is a pricing decision, not just an accuracy one.
What to do instead of shopping models
Put the instruction and the question where the model actually looks: the start, and a short recap at the end. Retrieve a small set of passages, not a dump. Rank them. Drop near-duplicates. If two chunks say the same thing, keep one. If a chunk is only loosely related, it is a distractor, not "more context."
Use a long window when the artefact is one thing the model must hold together — a contract, a codebase file, a transcript. Do not use it as a substitute for search over thousands of pages. And measure the agent on the task, at the context length you will actually ship, with the evidence in the middle as well as at the edges. A needle-in-a-haystack demo at 32K is not a production test.
Model choice still matters for reasoning, tools, and tone. It starts to matter after the window is clean. Until then, swapping models is how teams spend a quarter without changing the failure mode.