Every frontier lab now sells a context window somewhere between 200k and 2M tokens. The sales narrative writes itself — "just paste the whole codebase in the prompt." We hear it in almost every discovery call. It is almost never the right move.
The three costs nobody puts on the slide
Tokens. A million-token input, priced at the current mid-tier rate, is single-digit dollars per call. That's per call. A product that answers a hundred user questions a day burns hundreds a day in inference before it renders a single UI pixel.
Latency. Time-to-first-token on a million-token prompt is in the tens of seconds, sometimes minutes. That is a UX death sentence for anything that isn't a batch job. Prompt caching helps if the leading section is stable, but the moment you interleave user query with document, the cache stops paying.
Rot. The most-quoted long-context result of the last two years is that models genuinely struggle with information buried in the middle of long inputs. Needle-in-haystack results have improved, but "improved" is not "solved" — precision drops measurably past 64k for most tasks, and the drop is not symmetric across positions.
What still wins
For almost every RAG-shaped product: retrieval into a small context, cited sources, a re-rank step if precision matters. The upgrade path is a better retriever, not a bigger window. Retrieval-augmented generation was declared dead by three separate essays in 2025 and quietly kept working through all of them.
When we do use long context — and we do — it is for tasks where the information genuinely doesn't fit a retrieval abstraction:
- Reading a whole legal contract to answer a whole-document question ("does this instrument have any change-of-control clauses at all").
- Reasoning over a single large codebase where the interaction between files is the point.
- One-shot document QA where the document changes every call (no cache benefit anyway).
Notice what those have in common: single-document, one-off, low-frequency. They're research tasks, not product paths.
A cheap decision rule
Before you reach for the giant window, ask three questions:
- Can retrieval bring the top-k relevant chunks into a 16-32k window? If yes, do that.
- Is the leading section of the prompt stable across calls? If not, prompt caching won't save you.
- Can the user wait 30 seconds for an answer? If not, latency alone kills the design.
Long context is a tool. It's a good one. It is not a strategy, and any pitch that treats "our context window is bigger than theirs" as a moat is a pitch you can safely ignore.