LLM Reasoning Budget: How Developers Should Spend Thinking Tokens Without Wasting Latency

Author(s): Anna Jey Originally published on Towards AI. LLM Reasoning Budget A reasoning model can feel brilliant on one task and painfully slow on the next. The model did not suddenly get worse. You probably gave the same thinking budget to a simple lookup, a tricky code fix, and a risky production decision. That mistake is getting easier to make because reasoning controls are moving into normal developer workflows. Anthropic’s Claude Opus 5 release notes describe thinking on by default, a full effort ladder, and support for long-context agent work. OpenAI’s reasoning model docs explain that invisible reasoning tokens are billed as output tokens and consume context space. Google’s Gemini API changelog points in the same direction from the other side: newer Flash models are being tuned for token efficiency, lower latency, and agentic planning. GitHub is also making more models available inside Copilot, including Claude Opus 5 and Gemini 3.6 Flash. The practical lesson is simple: model choice is no longer enough. Developers now need a reasoning budget policy. This guide shows how to design one. You will learn when to use low, medium, high, and maximum effort; how to route tasks by difficulty; how to measure quality instead of guessing; and how to avoid paying for deep thinking when the app only needs a clean answer. What an LLM Reasoning Budget Really Controls An LLM reasoning budget is the amount of inference-time work you allow the model to spend before it returns an answer. Depending on the provider, this may appear as reasoning.effort, effort, a thinking budget, adaptive thinking, a deep reasoning mode, or a model tier that implicitly does more internal work. Do not treat this as a style setting. It is a resource allocation setting. Higher reasoning effort can help a model plan, inspect alternatives, use tools more carefully, or recover from ambiguity. It can also add latency, raise output-token cost, crowd the context window, and make simple tasks worse by overthinking them. The right budget depends on the task, not the prestige of the model. The best reasoning budget is the cheapest setting that still passes your quality bar for that specific class of work. That quality bar matters. A customer-support tagger, a code migration planner, a security triage agent, and a financial analysis assistant should not share the same default. They have different failure costs, latency expectations, tool needs, and rollback paths. Why This Became a Production Problem Older AI apps usually had one big decision: which model should answer? A team might pick a fast model for chat, a stronger model for code, and a cheap model for batch tasks. That still matters, but reasoning models add another dimension. Now you can choose the model and how hard that model should think. You can run a frontier model at lower effort for routine work, or a smaller model with more structured verification for a hard task. You can use one model for planning, another for tool execution, and another for final review. You can also burn a surprising amount of money while doing all of this badly. Research on test-time compute supports this messy reality. One study on compute-optimal scaling found that the best way to spend extra inference compute changes with problem difficulty and the base model. Easier problems may benefit from refinement, while harder problems may require broader search or stronger models. Another infrastructure-focused paper notes that reasoning-heavy workloads generate many output tokens, which can make decoding a dominant latency cost. That lines up with what developers complain about in practice. Reddit threads around Claude, OpenAI, and local models repeatedly mention the same pain: reasoning modes can improve hard answers, but they can also waste tokens, slow down chat, hide cost in output billing, and make migrations confusing when defaults change. The Four-Bucket Policy Start with four buckets. They are simple enough for a product team to understand and specific enough for an engineering team to implement. Low: Fast Answers for Low-Risk Work Use low effort when the task is clear, narrow, and easy to verify. Good examples include classification, short transformations, search query rewriting, formatting, simple extraction, light summarization, and small code edits with strong tests. Low effort should be your default for high-volume automation. If a support workflow tags 50,000 tickets a day, high effort on every ticket is usually a tax, not a feature. Use low effort first, then escalate only when confidence is low or downstream validation fails. Medium: The Default for Normal Product Work Medium effort fits tasks that need several steps but do not require deep exploration. Use it for moderate code generation, API mapping, product copy analysis, data cleaning, normal RAG answers, and workflow planning where errors are recoverable. Medium is also a good fallback when your router is unsure. It is rarely the cheapest path, but it gives you a balanced baseline for early production tests. High: Expensive Attention for Ambiguous Tasks Use high effort when the task has real ambiguity, hidden constraints, or a meaningful failure cost. Examples include debugging a race condition, comparing architecture options, planning a data migration, reviewing security-sensitive code, or deciding whether an agent should take an irreversible action. High effort should be intentional. If every request lands here, you do not have a reasoning strategy. You have a premium default. Max: Capability-Critical Work With a Human Gate Maximum effort belongs to rare cases: incident response analysis, major architecture decisions, risky tool actions, legal or compliance-sensitive reasoning, and final checks before production changes. Use it where the cost of a bad answer is clearly higher than the cost of slower inference. Do not send max-effort results straight into production side effects. Treat them like senior recommendations: valuable, but still subject to review, tests, approvals, and audit logs. A production reasoning policy routes task classes into budget tiers, then measures whether the chosen tier actually improved the outcome. How to Route Requests by Difficulty A reasoning budget router does not need to be fancy at first. Begin with […]