Every model ships with a context window measured in tokens, and the number keeps going up — so the intuition forms early: more room means more context means better answers. That intuition is wrong in a specific, measurable way. Attention over a long sequence is not uniform. Material in the middle of a long prompt is recalled worse than material at the edges, and every marginal paragraph dilutes the instructions you actually care about.
The better mental model is a budget. Every token you spend on boilerplate is a token of attention you can’t spend on the task. A 200-line system prompt where 30 lines do the work isn’t thorough — it’s noise wearing a suit. The craft is deciding what earns a place in context at all, and where it sits.
01The three layers of a working prompt
Prompts that survive contact with production tend to converge on the same shape:
- Instructions — short, imperative, at the top or bottom, never buried. What to do, what never to do.
- Reference — schemas, examples, style rules. Compressed to the minimum that changes behavior.
- Retrieved material — the variable payload. Clearly delimited, labeled as data rather than instructions.
When output quality drops after you add material, suspect placement before you suspect the model. Move the instruction block to the end, rerun, and compare — the fix is often free.
“The model didn't forget your instruction. You buried it under forty paragraphs it had no reason to prioritize.”
— Placeholder Creator, “Long context is a lie (sort of)” · 12:41