Reasoning Models: When to Use Them and When Not To

The $340,000 Lesson in Model Selection

A financial services team I know well built a document processing pipeline last year. The system was impressive in demos: it received complex regulatory documents, extracted structured data, flagged compliance issues, and generated summary reports. The quality was excellent. The engineers were proud of what they had built. The product team was already planning the next integration.

Then the bill arrived. Monthly inference costs had hit $340,000 and were climbing. The pipeline was processing about 18,000 documents a day — a volume that had seemed ambitious when they planned capacity, and had since become routine. A post-incident audit revealed the problem immediately: the team had used their highest-capability reasoning model for every single operation in the pipeline. Document classification. Named entity extraction. Field normalization. Date parsing. Schema validation. All of it routed through a model designed for complex multi-step reasoning tasks.

The remediation took six weeks and reduced monthly inference costs by 87% without any measurable degradation in output quality. The fix was not technical wizardry. It was taxonomy. The team built a classification layer that matched each task type to the cheapest model that could meet the quality threshold. Reasoning-intensive tasks — identifying whether a clause constituted a material disclosure obligation under three overlapping regulatory frameworks — stayed on the high-capability model. Deterministic extraction tasks moved to a smaller, faster, dramatically cheaper model. The system got faster and cheaper at the same time.

That story is not unusual in 2026. Reasoning models are now available from every major AI provider, and the pressure to deploy them is real. They produce better answers. They handle ambiguity more gracefully. They make engineering teams look capable in demonstrations. But the decision to use them is not a quality decision — it is a cost engineering decision. And most teams are making it incorrectly, in both directions.

This post builds the framework I use to make that decision deliberately.

What Reasoning Models Actually Do Differently

To route tasks correctly, you need to understand what reasoning models are doing mechanically that standard models are not.

Standard language models generate tokens by predicting what comes next, conditioned on everything that came before. They are extremely fast at this because the computation is a single forward pass through the model. Given a prompt, the model produces output. If the answer requires twenty reasoning steps, the model either encodes those steps implicitly in its weights or it fails.

Reasoning models — the class of models that first became broadly available in late 2024 and that every major AI lab had shipped by early 2026 — work differently. They generate extended chains of intermediate reasoning before producing a final answer. This chain-of-thought generation is not just a prompt-engineering trick. It is built into the model architecture and training objective. The model literally produces more tokens before answering, using those tokens to reason through the problem.

The practical consequences are three-fold.

Extended inference compute. A reasoning model asked a complex question spends more compute generating its answer because it generates more tokens. This is why reasoning models cost more per request. You are paying for the intermediate reasoning steps, not just the final output.

Self-verification loops. Because the reasoning chain is made explicit, the model can check its own work mid-generation. It can catch contradictions, recognize when an approach is failing, and course-correct before committing to a final answer. Standard models cannot do this systematically — they lack the explicit intermediate representation.

Multi-constraint satisfaction. Complex tasks often involve satisfying multiple constraints simultaneously — legal requirements, format specifications, business rules, domain knowledge. Reasoning models can hold more constraints in active consideration across their generation process because the chain-of-thought gives them working memory.

The corollary of these strengths is equally important: for tasks that do not require multi-step reasoning, self-verification, or multi-constraint satisfaction, the reasoning model’s extended compute is pure waste. The task completes correctly with a standard model. The reasoning tokens are generated and discarded. The cost is paid for nothing.

The Task Taxonomy: When Reasoning Earns Its Cost

The most useful tool I have developed for routing decisions is a task taxonomy that maps task characteristics to model requirements. It has three categories.

Category 1: Reasoning-Required Tasks

These are tasks where the quality difference between a standard model and a reasoning model is consistently measurable and operationally significant. Using a cheaper model here produces worse outcomes — not marginally worse, but meaningfully worse in ways that affect decisions, downstream systems, or user trust.

Multi-step planning under uncertainty. A task like “given these seven project dependencies, this team capacity, and these three external deadlines, produce a delivery sequence that minimizes risk” requires the model to hold multiple interdependencies in consideration simultaneously, evaluate trade-offs across scenarios, and produce a coherent plan that accounts for constraint conflicts. A standard model will produce a plan. It will often be wrong in subtle ways that are difficult to catch until execution reveals the problem.

Ambiguous constraint satisfaction. Legal interpretation, compliance analysis, and contract review are the canonical examples. The task is not “extract the clause” — standard models do that well. The task is “determine whether this clause, read in conjunction with these two other sections, satisfies this regulatory requirement in this jurisdiction under these conditions.” That is genuine multi-constraint reasoning.

Adversarial and edge-case inputs. When the input is designed to be ambiguous, contains conflicting information, or is a genuine edge case not well-covered by common patterns, reasoning models handle it more gracefully. Standard models tend to either pattern-match to the nearest familiar case (which may be wrong) or fail ungracefully.

Multi-hop information synthesis. Tasks requiring the model to retrieve, connect, and synthesize information across multiple documents, contexts, or reasoning chains — where each step depends on the results of previous steps — benefit from the explicit chain-of-thought.

High-stakes low-volume decisions. When the cost of a wrong answer is high and the volume is low enough that per-request cost is not the binding constraint, use the reasoning model. The error rate reduction justifies the cost at low volume even when it would not justify it at high volume.

Category 2: Standard Model Tasks

These tasks have clear quality thresholds that standard models meet reliably. Using a reasoning model here produces no measurable quality improvement. The extra cost is pure waste.

Structured data extraction. Pulling named entities, dates, amounts, addresses, and other typed fields from documents is largely a pattern-recognition task. A well-instructed standard model with clear output schema guidance produces correct results at high accuracy. The failure modes are coverage failures — the model misses an edge case format — not reasoning failures.

Classification at established categories. When the category set is well-defined and the examples are clear, standard models classify accurately and quickly. Document type classification, intent classification, sentiment classification — these are pattern-matching tasks.

Summarization with defined format. Producing a summary that hits specific format requirements and length constraints is a generation task where the quality criterion is adherence to format and coverage of key points. A standard model with good format instructions handles this well.

Translation and reformatting. Converting between data formats, languages, or structural representations does not require multi-step reasoning. The transformation is either correct or it is not.

High-frequency, low-stakes operations. Any task that runs thousands of times per hour, where a small per-request error rate is acceptable because downstream validation exists, should default to the cheapest model that meets the acceptable error rate threshold.

Category 3: Borderline Tasks — Requiring Empirical Routing

These are the tasks that neither category clearly covers. They are genuinely ambiguous — sometimes requiring reasoning, sometimes not, depending on input characteristics that cannot be determined until inference time. Managing this category is where sophisticated routing infrastructure earns its value.

Domain-specific question answering with variable complexity. A user asking about company policies might ask “what is the remote work policy?” (simple retrieval) or “does my situation qualify for the exception under section 4.2 given that I have both a home office and a satellite office in a different country?” (genuine constraint reasoning). The same pipeline must handle both, and routing them to different models requires classifying complexity at request time.

Code generation at variable complexity. Writing a simple utility function is a pattern-completion task. Designing an algorithm that satisfies five specific performance constraints across multiple input distributions is a reasoning task. Both arrive through the same code generation endpoint.

Multi-document analysis where depth varies. A task described as “analyze these contracts for risk” might involve a straightforward review against a standard checklist (standard model) or a complex analysis of interaction effects between clauses across multiple documents (reasoning model). Input length and stated complexity are reasonable signals, but they are imperfect.

The Three-Dimensional Routing Decision Framework

After working with several engineering teams on this problem, the clearest decision framework I have found operates across three dimensions. Each dimension is independently assessable, and their intersection produces a routing recommendation.

Dimension 1: Task Complexity

Single-step: The task can be completed in one cognitive operation. Pattern recognition, retrieval, simple transformation. Route to the cheapest model that meets quality threshold.

Multi-step: The task requires sequencing multiple cognitive operations where each depends on the prior result. Still feasible for capable standard models if the steps are predictable and the state space is small.

Multi-constraint: The task requires satisfying multiple constraints simultaneously where the constraints may conflict, interact, or require interpretation to prioritize. This is the clearest signal for reasoning model routing.

Dimension 2: Stakes

Low-consequence errors: The system has downstream validation, human review, or correction mechanisms. A wrong answer will be caught before it causes harm. The correct optimization is minimizing cost per correct outcome across the full validation loop, not eliminating errors at the model level.

High-consequence errors: A wrong answer causes irreversible harm: a compliance failure, a financial decision, a patient safety issue, a reputational event. Here the reasoning model’s error rate reduction justifies its cost even at low volume.

Dimension 3: Frequency

Single or low-frequency use: Per-request cost is not the binding constraint. Use the model best suited to the task regardless of cost, especially if stakes are high.

High-frequency at scale: Per-request cost compounds dramatically. A task running 100,000 times per day with a $0.05 cost difference between the routing options costs $5,000 per day or $1.8 million per year extra if the cheaper option is available and meets quality requirements.

The routing matrix produced by these three dimensions:

The most dangerous cell in this matrix is “Multi-constraint, Low Stakes, High Frequency.” This is where teams most often over-invest in reasoning models because the complexity appears to justify it, while the combination of low stakes and high frequency means the cost is very high and a cheaper model with empirical validation would suffice.

The Plan-and-Execute Pattern

The routing framework above handles task-level decisions. The Plan-and-Execute pattern handles a structural problem that arises in complex workflows: when a task is inherently complex but the complexity is concentrated in planning, not execution.

The pattern works as follows:

Phase 1 — Plan: A reasoning model receives the full task description, the available tools or sub-tasks, and any constraints. It produces a structured execution plan: a sequence of steps, the inputs and expected outputs for each step, the dependencies between steps, and the contingency rules for step failures.

Phase 2 — Execute: Each step in the plan is executed by the cheapest model that can reliably perform that specific step. The execution model does not need to understand the full task. It only needs to execute one well-defined step correctly.

The quality improvement comes from concentrating reasoning at the planning stage, where it is genuinely needed. The cost reduction comes from using cheaper models for execution, where reasoning is not needed.

In a document processing pipeline using this pattern:

• The reasoning model receives a complex regulatory document and produces a processing plan: extract these fields, apply these validation rules, flag these clause types for human review, generate this summary with these specific constraints.
• A fast standard model executes each extraction step using the plan as a structured prompt.
• A small specialized model handles format validation.
• A classification model routes the output to the appropriate downstream system.

The overall quality matches or exceeds the all-reasoning-model pipeline. The cost reduction at scale is typically 70 to 90 percent. The latency for common cases is lower because the execution steps are fast.

The failure mode to design against: execution models that encounter something the plan did not anticipate. When an execution step fails or produces an unexpected result, the system needs a well-designed escalation path. Either the step is retried with more context, or the task is escalated back to the reasoning model for re-planning. Teams that implement the pattern without this escalation path find that their cost savings are partially eroded by silent failures that require human correction downstream.

A well-implemented escalation path:

execution_step(task, plan_step) 
  -> success: continue pipeline
  -> low-confidence output: escalate to standard model with expanded context
  -> failure: escalate to reasoning model with failure context for re-planning
  -> second failure: route to human review queue

The escalation logic does not need to be complex. A confidence threshold on the execution model’s output is often sufficient. What matters is that the path exists and is tested before production deployment.

The Hybrid Model Architecture

Most production AI systems in 2026 do not use a single model. They use a hierarchy of models, each matched to a specific class of task. The architecture I recommend has four layers.

Layer 1 — Reasoning models sit at the top of the hierarchy. They handle planning, complex constraint satisfaction, high-stakes decisions, and tasks that have empirically failed at lower layers. They are used sparingly — targeted at exactly the tasks where their extended compute produces measurable quality improvement.

Layer 2 — Full-capability standard models handle the majority of generation tasks: summarization, question answering with clear domain scope, code generation at moderate complexity, multi-document analysis where the synthesis is straightforward. These models are fast and capable. Most tasks should land here by default.

Layer 3 — Small language models (SLMs) handle classification, structured extraction, format validation, and other high-frequency pattern-recognition tasks. SLMs running locally or at the edge can process these tasks at a fraction of the cost of full API calls. Teams using SLMs for appropriate task classes routinely achieve 60-80% cost reduction on those task classes with no quality degradation.

Layer 4 — Deterministic functions handle tasks that should not use a model at all. Date parsing, schema validation, regex extraction, lookup operations — these are places where teams sometimes reach for a model because the model is available. A deterministic function is faster, cheaper, and more reliable for deterministic tasks.

The routing infrastructure that connects these layers is what makes the architecture work in practice. It must:

• Classify incoming tasks accurately enough to route them to the right layer
• Handle misclassification gracefully with escalation paths
• Track per-task quality metrics to identify routing errors empirically
• Support reconfiguration without redeployment as task distributions shift

The routing classifier itself is an interesting design decision. In most cases, a lightweight classification model or rule-based system handles routing with sufficient accuracy. The routing classifier does not need to be a reasoning model — it needs to assess whether a task requires reasoning, not perform the reasoning itself. Complexity signals that work well as routing features include input length, keyword presence (terms like “determine whether,” “given that,” “considering,” “in conjunction with”), the presence of conflicting information in the input, and explicit complexity indicators from upstream systems.

What Most Teams Get Wrong

The most common failure I see is not choosing the wrong model for a specific task. It is never building the infrastructure to make the choice deliberately.

Teams deploy a pipeline in a hurry, use their best model for everything because it works and the demo is the deadline, and then face a billing problem six months later when volume has scaled. By that point, the pipeline architecture does not expose clean task-level boundaries where routing can be inserted. The refactoring cost is high, and it competes with feature work that looks more valuable.

The second most common failure is optimizing the wrong metric. Teams that do tackle this problem often focus on token count. They optimize prompts to reduce token length, choose smaller models, and measure success by tokens saved. The correct optimization target is cost per correct outcome. A shorter prompt that increases the error rate produces more expensive correct outputs even if it produces fewer tokens. A smaller model that increases human review requirements adds more cost than it saves. Measurement must track quality outcomes alongside cost, and the optimization must treat them jointly.

The third failure is treating the routing decision as a one-time architecture decision rather than an ongoing empirical practice. Task distributions shift as products evolve, as user behavior changes, as underlying models are updated. A routing configuration that is optimal today may be significantly suboptimal in six months. Teams need monitoring that surfaces routing quality — tracking what fraction of escalations succeed, what fraction of low-layer executions produce correct outputs, and where the model hierarchy is being underused or overused.

Warning: The three failure modes most teams hit

1. Deploying the best model for everything and discovering the cost problem on a billing cycle, not in architecture review.
2. Optimizing for token count instead of cost per correct outcome — a metric that looks right but leads to wrong decisions.
3. Treating model routing as a static architecture decision instead of an empirical practice with ongoing measurement.

Benchmark-Validated Routing: The Empirical Process

The routing matrix and the task taxonomy give you a starting point. The empirical process gives you a defensible configuration.

For each task type in your pipeline, the empirical process is:

Step 1 — Build a task-specific evaluation dataset. One hundred examples minimum, drawn from real production traffic or synthetically generated to cover the distribution of inputs you actually receive. Include edge cases. Include the cases that previously failed.

Step 2 — Define the quality threshold. What accuracy, coverage, or faithfulness score constitutes acceptable performance for this task type? This threshold is a product and business decision, not a technical one. Get explicit agreement on it before running benchmarks.

Step 3 — Run the evaluation across your model hierarchy. From the cheapest model upward, test each model against the evaluation dataset and measure performance against the quality threshold. Stop when you find the cheapest model that meets the threshold.

Step 4 — Measure cost per correct outcome. At the volume this task type will run in production, calculate what each model costs per correct output at the quality threshold. This is your routing cost basis.

Step 5 — Set routing configuration with a quality buffer. Do not set the routing threshold at exactly the measured quality threshold. Set it with a buffer that accounts for distribution shift. If your threshold is 90% accuracy and the model achieves 93% on your evaluation set, the buffer is 3 percentage points. Monitor production performance against the buffer; escalate routing review when performance approaches the threshold.

Step 6 — Repeat periodically. Whenever your input distribution shifts meaningfully, whenever a model is updated, and at least quarterly as a scheduled practice.

This process seems like a lot of work. It is. But it is significantly less work than the alternative, which is an unplanned cost crisis or a quality failure discovered in production.

Cost Governance: Ceilings, Alerts, and Budgets

Technical routing alone is not cost governance. Cost governance requires explicit controls that prevent drift and catch anomalies before they become billing surprises.

Per-task-type cost ceilings: For each task type in your system, define an acceptable cost per request. Implement this as a soft ceiling (triggers an alert) and a hard ceiling (routes to a cheaper model or queues for batch processing). Cost ceilings should be defined in terms of cost per correct outcome, not cost per request.

Model-tier budget allocation: Define what fraction of your total inference budget each model tier should consume. If your reasoning model budget is 20% of total inference spend, an alert should fire when it approaches 25% before you hit 30%. Budget drift is always easier to address early.

Anomaly detection on usage patterns: A sudden spike in reasoning model usage without a corresponding spike in the task types that require reasoning is a signal worth investigating. It may mean a routing classifier has degraded, that a new feature is routing incorrectly, or that a client is abusing a high-complexity endpoint.

Fine-tuning as a cost governance lever: When a task type runs at high volume and high frequency, fine-tuning a smaller model on reasoning model outputs is worth evaluating. The fine-tuned small model captures some of the reasoning capability at a fraction of the inference cost. This is not appropriate for all task types — fine-tuning has its own cost and maintenance overhead — but for stable, high-volume task types with well-defined quality requirements, it can reduce per-request cost by 80-90% relative to the reasoning model baseline while maintaining acceptable quality.

The fine-tuning decision follows its own framework:

• Volume: is the task type running at least 10,000 requests per day?
• Stability: is the task type definition stable enough to justify the fine-tuning investment?
• Quality headroom: does the smaller model get close enough to the quality threshold to make fine-tuning plausible?
• Maintenance cost: can the team sustain periodic re-tuning as requirements evolve?

If the answer to all four is yes, fine-tuning is worth pursuing. If any is no, the routing framework is the better tool.

Latency Implications and Real-Time Applications

One dimension that cost analysis alone underweights is latency. Reasoning models are slower than standard models by a margin that is operationally significant for real-time user-facing applications.

Extended inference compute means more tokens generated, which means more time to first token and more time to completion. In early 2026, reasoning model latency at typical task complexity is 3 to 8 seconds for non-trivial requests. Standard models handle similar requests in under a second. SLMs at the edge handle pattern-recognition tasks in under 100 milliseconds.

For batch processing, background analysis, and async workflows, this latency difference is irrelevant. Use the best model for the task. The pipeline runs when it runs.

For real-time user-facing applications — conversational interfaces, interactive document editing, live code assistance — this difference determines whether the product feels responsive or frustrating. A 6-second wait for a response that could have been produced in 800 milliseconds by a capable standard model is not a quality improvement; it is a user experience failure.

The latency routing rule I apply: for any user-facing interaction with a latency budget under 2 seconds, treat the reasoning model as unavailable regardless of task complexity. Either the task can be handled by a standard model within budget, or the user experience needs to be redesigned to accommodate async processing.

Streaming responses partially address this by reducing time-to-first-token perception, but they do not change total generation time. A streaming reasoning model response that takes 8 seconds to complete still imposes 8 seconds of wait time for tasks that require the full response before the user can proceed.

Where This Is Heading

The routing problem will become more complex before it becomes simpler, for two reasons.

First, the model landscape is fragmenting, not consolidating. In 2024, there were a handful of capable models. In early 2026, there are dozens of production-grade models across a wide spectrum of capability, cost, and latency profiles. Each new model adds a routing decision: where does this model fit in the hierarchy, and which task types does it displace? Engineering teams that build routing infrastructure now will be able to absorb new models quickly. Teams that route by hand or by convention will find each new model arrival is an expensive reconfiguration project.

Second, specialized models are proliferating. Domain-specific models — trained or fine-tuned for legal analysis, medical documentation, code generation, financial modeling — can outperform general reasoning models on their target domain at lower cost. The routing decision for a legal document pipeline in 2027 may be between a general reasoning model and a specialized legal model rather than between a reasoning model and a general standard model. The framework is the same. The model options are wider.

The teams building empirical routing infrastructure today — the evaluation datasets, the quality monitoring, the cost-per-outcome tracking — are building infrastructure that compounds in value as the model landscape evolves. The teams routing by hand or by convention are building debt.

The underlying discipline is not new. Every software system routes requests to the cheapest resource that meets the service level requirement. CPU cores, database replicas, CDN nodes — the logic is the same. AI models are now part of that resource hierarchy, and they need to be managed with the same discipline that the rest of the stack receives.

Model selection is a cost engineering discipline. The teams that treat it as one will ship better systems at lower cost. The teams that treat it as a preference will keep being surprised by their billing cycles.

Summary: The Routing Principles

After working through this framework with multiple teams, these are the principles I come back to:

Match the model to the task, not to the team’s comfort level. The best model for a task is the cheapest model that meets the quality threshold. Period.

Measure cost per correct outcome, not cost per token. The metric that looks obvious is often the wrong optimization target.

Build the routing infrastructure before you need it. The right time to design task classification and model hierarchy is during architecture, not during a billing crisis.

Treat routing as an empirical practice, not a static decision. Task distributions shift. Models change. What was optimal six months ago may not be optimal today.

Use the Plan-and-Execute pattern for complex workflows. Concentrate reasoning at the planning stage. Use cheap models for execution. Design escalation paths.

Implement cost governance explicitly. Ceilings, alerts, and budgets prevent the drift that turns a well-designed routing layer into an expensive one over time.

The question your team should be answering this week is not “which model is best?” The question is: “for each task type in our system, what is the cheapest model that meets our quality threshold, and do we have the measurement infrastructure to know whether that threshold is being met?”

If you cannot answer the second part, that is where to start.