Looking at the trace of a long-running coding or ops agent, you will see the same pattern. A significant share of the steps are mechanical: construct a tool call, check the output format, retry on a transient error, summarise a log, classify a result. None of that requires frontier reasoning. None of it requires a model that scored 90% on GPQA Diamond. It requires a model that follows instructions reliably, responds quickly, and does not cost much per call.

The problem is that most teams route every one of those steps through the same expensive model they use for the planning steps, because it is the model they already have an API key for. When NVIDIA launched Nemotron 3.5 Lightning on August 11, they named this practice directly: a frontier reasoning model is the wrong tool for most of what an agent actually does. The argument is worth taking seriously not because Nemotron 3.5 Lightning is the only execution model worth considering, but because NVIDIA building an entire model class and a routing library around this premise is a signal that the industry has converged on the same structural conclusion.

This piece makes the case for the two-tier agent architecture, explains where the cost is actually going in a single-model stack, and shows what the routing split looks like in practice.

Where the tokens are going

The intuition behind most frontier model deployments is that the model earns its cost by being smart. That is true for the planning steps. It is not true for the bulk of an agent’s token budget.

Token prices dropped a lot between 2025 and 2026, yet most enterprises still exceed their AI budgets. The explanation is that agentic workflows multiply token consumption in a way that single-turn pricing intuitions do not capture. A user asking a chatbot one question generates one inference call. An agent completing the equivalent task may generate thirty: planning steps, tool calls, result reads, validation loops, sub-agent delegations, error handling, context compression, and a final synthesis pass. Each of those is a separate inference call, each billed at output rates.

The LangChain benchmark published alongside the Nemotron 3.5 Lightning launch made this concrete. Across 145 multi-turn agentic tasks, routing between Lightning and Claude Opus 4.8 with an escalation router cut cost 74% compared to a frontier-only baseline. Only 7% of calls went to the frontier model. The other 93% were handled by the execution model. The accuracy tradeoff was roughly 6 percentage points.

That 7% figure is worth sitting with. If 93% of the calls in a real multi-turn benchmark can be handled by a cheap execution model with a 6-point accuracy tradeoff, the question is not whether two-tier routing is worth pursuing. It is why the default is still one model for everything.

What goes to the frontier model and what does not

The split is more intuitive than it might first seem. The execution layer is not a compromise on quality. It is recognition that most of what an agent does is not a quality problem.

Task type	Examples	Route to	Why
Planning and orchestration	Decompose a goal, decide which tools to invoke, synthesise conflicting results, handle unexpected failure	Frontier model	Requires genuine reasoning, judgement, and context across the full task
Tool call formatting	Construct a JSON payload, fill a function schema, format a search query	Execution model	Deterministic structure; correctness is verifiable; no reasoning required
Result validation	Did the tool return an error? Is the output within expected bounds? Does the format match?	Execution model	Pattern matching and schema checking; repeatable and fast
Retry logic	Re-issue a failed API call, adjust parameters on a transient error	Execution model	Conditional logic with no judgement. Better handled by code, or a tiny model
Context summarisation	Compress a long tool output before passing it back to the planner	Execution model	Extraction task; does not require frontier-level comprehension
Classification and routing decisions	Is this tool output an error or a result? Which next step applies?	Execution model	Bounded decision with enumerable outcomes; cheap to run reliably
Final synthesis and output	Produce the user-facing answer from accumulated results	Frontier model	Quality and coherence of output matters; this is what the user sees

The useful heuristic for drawing the line: if you can verify the output programmatically, against a schema, a format, a range, or a known error code, the task does not need a frontier model. Verification is cheap. Generating something that passes verification does not need frontier-level reasoning if the generation space is well-constrained.

The tasks that genuinely need the frontier model are the ones where correctness cannot be verified cheaply: synthesising conflicting information from multiple tool outputs, deciding between genuinely ambiguous next steps, producing user-facing output that needs to be coherent and well-reasoned. These are also the tasks that are sparse in a long agent loop. A well-designed agent might make a hundred tool calls across a complex task while making five genuine planning decisions.

The case Paul Gauthier ran in 2025

The empirical case for this architecture did not start with Nemotron 3.5 Lightning. In January 2025, Aider founder Paul Gauthier published results showing that combining DeepSeek R1 for reasoning and planning with Claude Sonnet for code editing hit state-of-the-art results on Aider’s polyglot benchmark at 14x less cost than using OpenAI o1 alone.

The architecture was simple. R1 handled the thinking: analysing the codebase, identifying which files were relevant, drafting an implementation plan. Sonnet handled the precision editing: taking a clear specification and executing it in code. R1’s reasoning was cheap per token. Sonnet’s editing was precise per step. Neither was doing the other’s job. The result was better than a single frontier model at a fraction of the cost.

That pattern has generalised. Research from UC Berkeley, Anyscale, and Canva published at ICLR 2025 showed that trained routing systems deliver 85% cost reduction while maintaining 95% of GPT-4 performance across mixed-complexity workloads. The core finding was that a small classifier can route the majority of traffic to smaller models without measurable quality degradation on those tasks. It is the same insight Gauthier ran manually in production.

What the execution model needs to be

Not every small model is suitable for the execution layer. The requirements are different from those of a general-purpose model.

Instruction following over reasoning breadth. The execution model’s job is to do what it is told precisely and consistently. It does not need to reason about whether the instruction is correct. The planner has already determined that. It needs to follow the schema, call the tool correctly, and return structured output. A model that is distilled or fine-tuned for agentic harnesses will often outperform a larger general model on these tasks despite having far fewer active parameters. Nemotron 3.5 Lightning was specifically trained against OpenClaw and Hermes Agent for exactly this reason.

Latency. In a long agent loop, the execution model’s response time accumulates. A model that adds 2 seconds to each tool call adds 200 seconds to a 100-step agent. Nemotron 3.5 Lightning’s 4x speed advantage over similar-sized models is specifically relevant here. Its Mamba-2 plus MoE plus Attention hybrid architecture with multi-token prediction baked in at pretraining is designed for throughput rather than maximum reasoning depth.

Predictability. The execution model needs to fail gracefully and predictably. A model that sometimes produces correct JSON and sometimes produces something structurally different is more expensive to work around than a slightly less capable model that fails in a known way. The failure modes of the execution model should be enumerable and catchable by the validation step, which then triggers escalation to the frontier model.

Fine-tunability. The execution model’s tasks are narrow and repetitive enough that domain-specific fine-tuning is practical. A customer support agent’s tool call formats, a coding agent’s error classification schema, an ops agent’s log parsing patterns: all of these are learnable by a small model fine-tuned on a few hundred examples. Nemotron 3.5 Lightning ships with LoRA and full SFT support through NeMo Automodel specifically to enable this. A fine-tuned execution model on your specific task distribution will almost always outperform a general-purpose small model at the same cost.

Model pairings that work in practice

The choice of execution model depends more on your existing infrastructure and what the execution tasks look like than on abstract capability rankings.

Planner (frontier)	Executor (cheap)	Cost split (est.)	Good for
Claude Opus 4.8	Nemotron 3.5 Lightning	~10% calls frontier / ~90% execution	Complex reasoning agents where synthesis quality is critical
DeepSeek V4 Pro	DeepSeek V4 Flash	~15% calls frontier / ~85% execution	Cost-sensitive pipelines; both models share the same API format
Kimi K3	Qwen3 30B-A3B	~10% calls frontier / ~90% execution	Open-weight-only stacks requiring a self-hostable execution layer
GPT-5.6 Sol	Nemotron 3.5 Lightning or GPT-5.6 Luna	~7% calls frontier / ~93% execution	OpenAI-native pipelines; Luna shares the same API surface as Sol

The V4 Pro and V4 Flash pairing deserves a specific note. Both models share the same API format on DeepInfra: same endpoint structure, same parameter interface, same function calling schema. Implementing the routing split is a one-line model-name change rather than an integration change. You point planning calls at deepseek-v4-pro and execution calls at deepseek-v4-flash, and the rest of the code stays identical. For teams already running on DeepSeek, this is the lowest-friction path to the two-tier architecture.

The Opus 4.8 and Lightning pairing is the one backed by the LangChain benchmark data. The 74% cost reduction on 145 multi-turn tasks is the most rigorous public result available for this specific architecture. The accuracy tradeoff of roughly 6 points is meaningful but manageable for most production workloads where the frontier model still handles the decisions that matter.

How NeMo Switchyard frames the routing problem

NVIDIA released NeMo Switchyard alongside Nemotron 3.5 Lightning. It is an open-source routing library that directs each agent step to the most efficient model available. The design is worth understanding because it makes explicit what most teams are currently handling ad hoc.

Switchyard treats routing as a first-class concern in the agent architecture rather than a deployment detail. Each step in the agent loop is classified by type, whether planning, tool execution, validation, or delegation, and routed to the appropriate model tier. The planner and executor do not need to be aware of each other. The router handles the handoff. The library is designed to be composable with LangGraph, LlamaIndex, and other popular agent frameworks, so the routing layer sits between the framework and the model API rather than inside either.

The key architectural point Switchyard makes explicit: the routing policy is a separate artifact from the agent logic. What the agent does, its tools, goals, and prompts, is defined once. Which model handles each step is a configuration decision that can be changed, experimented with, and monitored independently. That separation is what makes it possible to tune the cost-quality tradeoff by adjusting routing thresholds rather than rewriting agent code.

You do not need Switchyard specifically to implement this architecture. LangGraph’s conditional edges, LlamaIndex’s routing abstractions, or a simple if-else on task classification all work. What Switchyard provides is a structured vocabulary for the routing problem and a pre-built integration with the Nemotron model family. The underlying pattern, classify each step, route to the cheapest model that can handle it reliably, escalate on failure, is framework-agnostic.

The escalation path matters as much as the routing

A two-tier architecture that routes correctly under normal conditions but fails silently when the execution model is out of its depth is worse than a single-model stack. The escalation path from executor to planner needs to be explicit, fast, and observable.

The practical pattern: the execution model returns its output alongside a confidence signal. That might be an explicit score, a structured uncertainty field, or simply a flag in the response schema. The router checks the signal before passing the output to the next step. If the execution model flags low confidence, or if the output fails format validation, the router re-routes the same step to the frontier model. The frontier model’s output replaces the execution model’s failed attempt, and the agent loop continues.

What not to do: let failures accumulate silently, infer confidence from output structure alone, or escalate every validation failure to the frontier model. The first produces degraded results without any billing signal. The second produces false confidence on structurally valid but semantically wrong outputs. The third eliminates most of the cost saving the architecture was intended to produce.

Observable routing is the operational requirement that makes the architecture maintainable. Log which model handled each step, the execution model’s confidence signal, and whether escalation occurred. That trace is what tells you whether your routing thresholds are calibrated correctly, which task types are escalating more than expected, and where the execution model needs fine-tuning. Without it, you are flying blind on the cost-quality tradeoff the architecture is supposed to optimise.

What this means for per-task cost

The practical arithmetic is straightforward once the routing split is defined. Take a coding agent that makes 80 model calls per complex task: 6 planning decisions routed to Opus 4.8, and 74 execution steps routed to Nemotron 3.5 Lightning or V4 Flash. At Opus 4.8’s $25/M output rate and Lightning’s $0.40/M, with an average of 500 output tokens per planning step and 200 output tokens per execution step:

Single-model (all Opus 4.8): 80 calls x 200 tokens average x $25/M = $0.40 per task

Two-tier (6 frontier + 74 execution): 6 x 500 x $25/M + 74 x 200 x $0.40/M = $0.075 + $0.006 = $0.081 per task

That is an 80% cost reduction on the model bill. At 100,000 tasks per month, the single-model stack costs $40,000. The two-tier stack costs $8,100. The quality tradeoff depends entirely on whether the 74 execution steps are correctly classified, whether they are genuinely the kind of structured, verifiable work the execution model handles reliably. Getting that classification right is the engineering work the architecture requires.

The gap between falling unit prices and rising invoices is almost entirely explained by the multiplier effect of agentic workflows on token consumption. The two-tier architecture is the most direct available lever for closing that gap, not by reducing what the agent does, but by running most of what it does on infrastructure sized for the actual task.