Decision Models: The Fast Layer for AI Agents

There is a decision your agent makes dozens of times per session that you are probably paying a frontier model to handle. Should this request go to the fast cheap model or the expensive one? Is this tool call safe to execute? Is the task actually finished, or does the agent think it is finished? Which of these six retrieved documents is relevant to the question?

Every one of those is a bounded choice with a fixed option set. None of them requires generating prose. None of them requires chain-of-thought. None of them should cost what a Claude or GPT-5.5 call costs. But for most teams building agent pipelines today, that is exactly what they are paying because the LLM already in the loop is the path of least resistance for any decision that cannot be handled by a rules table.

A new model category arrived in September 2026 to solve this problem, and it spread faster than anything in the agent tooling space in the past year. TypeSafe AI released Jev on September 15. Laya, an open-source alternative, followed three days later. Kev, an open decoder-based implementation, appeared in the same week. AWS released Strands Decider 2B on October 1. OpenAI announced the Decisions API at DevDay 2026 the same week and moved it to public beta on October 6. A local runtime called Ollaya appeared shortly after, bundling seven open decision models behind a single API endpoint. The category went from nonexistent to contested in under three weeks.

This article explains what a decision model actually is, how the architecture differs from a generative LLM, what the current landscape looks like across proprietary and open options, where these models fit in an agent pipeline, and what the honest limitations are.

What a Decision Model Actually Is

A decision model takes two inputs: a block of context (a message, a document, an agent’s conversation history, a tool call’s proposed parameters) and a set of predefined possible answers. It returns one answer from that set, plus a confidence score for each option. It never generates text. The output is always one of the choices you supplied, with a probability attached.

TypeSafe calls this a “System One model” which is a deliberate reference to Kahneman’s dual-process framework, positioning decision models as the fast, automatic cognition layer that handles bounded judgments while the generative LLM handles the slow, deliberate reasoning.

The name has stuck across the field even for non-TypeSafe implementations. Three question types have become the standard interface across all implementations in this space:

Question type	Returns	Example
Choice	One option from a fixed list	Route this ticket: billing / technical / shipping / other
Score	A number between 0 and 1	Urgency score from 0 (low) to 1 (critical)
Noul (yes/no)	A probability between 0 and 1	Is this tool call grounded in what the user said?

The confidence score is not a courtesy feature. It is the mechanism that makes decision models automatable. A generative LLM asked to classify a message as billing or technical will almost always give you an answer even when the message is genuinely ambiguous because that is what text generation does. A decision model with calibrated confidence scores lets you set a threshold: auto-act above 0.85, escalate to a human or a bigger model below it. You can tune where that threshold sits based on the cost of a wrong decision in your specific context. That is not something you can do cleanly with a generative model’s output.

How the Architecture Works

The defining architectural characteristic of a decision model is that it does not have a language modeling head. A standard transformer-based LLM ends with a head that produces a probability distribution over the entire vocabulary at every generation step, and then samples from that distribution sequentially to produce text. That vocabulary projection is expensive, and sequential sampling is why text generation takes seconds.

A decision model removes the vocabulary head and replaces it with a pointer head or decision head that scores only the specific options you supplied. There is no vocabulary. There is no sampling. The model reads the context, reads the options, and produces a score for each option in a single forward pass.

This architectural difference produces a dramatic latency reduction. Jev reports 70–500ms end-to-end latency versus 3+ seconds for frontier models. OpenAI’s Decisions API demonstrated roughly 150ms versus 1,600ms for standard Luna calls at DevDay which is approximately 10x faster. AWS Strands Decider 2B running on an H100 GPU reports sub-100ms decision time.

The other consequence of removing the vocabulary head is that there is no output token cost. Decision models charge for input only. At $0.042/M input tokens for Jev, the cost comparison versus a frontier model call is not marginal. A pipeline routing 10 million decisions per month through Jev spends $420. The same decisions through Claude Haiku 4.5, which is already one of the cheaper generative options, costs orders of magnitude more once you factor in the output tokens required to express the classification.

How Each Model Trains Calibration

The confidence scores a decision model returns are only useful if they are calibrated and a 0.85 confidence actually corresponds to being right 85% of the time. Getting calibration right is harder than getting accuracy right, and it is where the different models in this category diverge technically.

The training approach that has become standard across all serious implementations is called RLCD: Reinforcement Learning for Calibrated Decisions. Rather than training on cross-entropy loss (which optimizes for accuracy but produces overconfident outputs), RLCD trains the model to maximize strictly proper scoring rules. The Brier score and logarithmic scoring rules are the most common. A model trained this way has an incentive to say 0.6 when it is genuinely 60% confident, rather than rounding up to 0.95 to collect a slightly higher accuracy reward.

Laya documents its RLCD implementation publicly: GRPO-style policy gradient with a combination of logarithmic, spherical, and ranked probability scoring rules as the reward, with no supervised cross-entropy loss at all. TypeSafe describes Jev as trained with RLCD but does not publish the details. Strands Decider 2B’s training recipe, published with the model weights under Apache 2.0, describes a similar approach.

The practical consequence of calibration quality shows up in threshold-based routing. If your decision model consistently returns 0.92 confidence on decisions where it is actually right 72% of the time, a threshold of 0.85 gives you false security. Independent benchmarks on JevBench suggest that Jev currently has the best calibration among available options, with open-source implementations closing the gap at a faster rate than expected.

The Current Landscape

The decision model space went from a single proprietary offering to a contested multi-player market in under three weeks. Here is where each option sits as of early October 2026:

Model	Architecture	Parameters	Latency	Cost	License	Deployment
Jev 1.13 (TypeSafe)	Proprietary, parallel	Undisclosed	70–500ms	$0.042/M input	Proprietary	Cloud API, OpenRouter
Strands Decider 2B (AWS)	Qwen3.5-2B + pointer head	2B (~1M pointer)	~100ms (GPU)	Free (self-host)	Apache 2.0	Local, Bedrock AgentCore
Laya (Convai)	ModernBERT-large + decision head	421M	33ms (GPU)	Free (self-host)	Apache 2.0	Local, Docker, MCP
Kev-4B (Jared Palmer)	Qwen3.5-4B + LoRA + pointer	4B	18ms (H100)	Free (self-host)	Open weights	Local server
OpenAI Decisions API	GPT-6 Luna (specialized)	Undisclosed	~150ms	Proprietary	Proprietary	Cloud API
Ollaya	Runtime (bundles Laya, Decider, Kev, others)	Various	8ms (Laya-multilingual)	Free (self-host)	Apache 2.0	Desktop, Docker, CLI

The architecture split in the open-source landscape is worth understanding, because it produces different practical trade-offs.

• Laya: uses a bidirectional encoder (ModernBERT-large, 421M parameters) with a trained decision head. Bidirectional encoders read the full context in both directions before producing output, which gives them a strong comprehension at low parameter count. The downside is that Laya’s base checkpoints (without the typed-decisions fine-tune) score only moderately on JevBench.
• Kev: uses a Qwen3.5 decoder backbone with a rank-16 LoRA adapter and a custom pointer head. The pointer head compares the hidden state at a closing <decide> token with each option’s closing token and returns calibrated probabilities. Because Kev keeps the full autoregressive decoder backbone, it carries the comprehension quality of a generative model while routing output through a pointer rather than a vocabulary distribution. Kev-4B outperforms all three Laya checkpoints on independent 2,394-item evaluations. Kev-27B requires an 80GB GPU or a Mac with 96–128GB unified memory which is closer to a small generative model deployment than a lightweight classifier.
• Strands Decider 2B: follows Kev’s architectural pattern: a Qwen3.5-2B decoder torso with the language model head removed and replaced by a ~1M parameter pointer head. AWS published the full training recipe and data inventory alongside the weights, making it the most inspectable of the implementations. Independent benchmarks put Strands Decider at 72% routing accuracy on 100-item evals which is lower than Jev’s cloud performance, but competitive with Laya’s base checkpoints. AWS’s documented design intent is for Decider to handle the cases an agent hits reliably (high-frequency, low-ambiguity routing and tool gating) and escalate to an LLM for anything with a confidence below a set threshold. In one head-to-head benchmark against Claude Haiku 4.5, a hybrid pattern — Decider for high-confidence decisions, Haiku only below 0.6 confidence — reached 90% accuracy, compared to 72% for Decider alone.
• OpenAI’s Decisions API: takes a different approach from all the open models: it routes through a specialized version of GPT-6 Luna rather than a purpose-built decision architecture. The practical implication is that it likely has higher comprehension quality than any of the open models at the cost of proprietary pricing and the constraint that inputs route through OpenAI’s infrastructure. It supports text and image inputs, which none of the open models currently supports.

Where Decision Models Fit in an Agent Pipeline

The twelve use cases that have emerged most consistently across the developer community building with these models in the first weeks of the category are worth mapping explicitly, because the right mental model is not “a decision model does classification” — it is “a decision model handles every bounded choice that currently runs through your LLM unnecessarily.”

Routing and orchestration

• Model routing: score incoming request complexity and route to the cheapest model that can handle it. A decision model evaluating complexity as low/medium/high with a confidence score can replace a set of hand-written rules or an expensive preliminary LLM call. The compounding savings on a high-volume pipeline are significant, e.g. every request that correctly routes to Deepseek V4-Flash instead of Claude Opus 4.7 saves roughly $24.72/M output tokens.
• Tool selection: given a description of available tools and the current agent state, which tool should the agent call next? This is exactly a bounded choice from a known option set.
• Orchestration forking: in a multi-agent system, which agent should handle this subtask? The decision model scores each agent against the current state and returns confidence-weighted assignments

Safety and guardrails

• Tool call gating: before executing a proposed tool call, verify that the parameters are grounded in what the user actually said. AWS’s documented Strands example catches a weather agent calling a tool before the user specified a city which the decision model flags as premature rather than letting the agent hallucinate a location. This pattern sits between the agent and every tool call and adds sub-100ms overhead.
• Destructive action classification: is this shell command / database write / file deletion safe to auto-execute, or should it escalate to a human? A decision model with calibrated confidence makes this gate tunable where you can set the threshold based on the cost of a wrong execution in your environment.
• Content policy: is this user message within scope for this application? Routing out-of-scope requests at the decision layer rather than the LLM layer saves the full LLM call cost on every rejected input.

Evaluation and quality control

• Completion verification: is the agent’s task actually done, or has it incorrectly concluded? This is one of the most persistent failure modes in agentic coding. The agent reports success when the tests fail or the implementation is incomplete. A decision model evaluating completion at each loop iteration with a confidence threshold adds a lightweight check before the agent exits.
• Output quality scoring: score the LLM’s last output on relevance, accuracy, or task alignment before passing it downstream. Cheap per-check cost makes this viable at every generation step rather than as a post-hoc review.

Memory and context management

• Memory retrieval relevance: of these ten retrieved memory chunks, which ones are actually relevant to the current query? A decision model replaces a secondary LLM call for reranking.
• Context compression gating: has enough new information arrived to warrant updating the agent’s working memory, or can the current context continue? A low-cost judgment that runs continuously without adding significant overhead.
• Priority classification: in a queue of pending tasks, score each one for urgency and route accordingly.

What these use cases show

The common thread across all of these is that the decision is bounded and repetitive. You know the option set in advance. The right answer does not require generating an explanation. Running a frontier LLM for this is like hiring a senior engineer to answer yes/no questions all day which is  technically capable, but not the right tool for the task.

The Benchmark That Emerged Alongside the Category

JevBench, a benchmark for evaluating decision models on routing and classification tasks, appeared alongside Jev and has become the de facto comparison standard across implementations. Independent results from early October 2026 put Jev (the proprietary TypeSafe model) at 63.29 (86.6% public / 36.7% sealed accuracy), with Strands Decider 2B at 0.723 (72.3% accuracy on 231 items), and Laya’s base 421M checkpoint at 30.25 on the same scale.

Two things to note about these numbers. First, the gap between Jev and the open-source implementations is real on hard items, but the base Laya checkpoint is the wrong comparison. The laya-typed-decisions fine-tuned checkpoint scores 0.766 accuracy on its own benchmark and is the one designed for production agent decision tasks. Second, all of these models are weeks old. The open-source implementations are closing the gap faster than expected; the rate of improvement visible in the Kev results (Kev-4B outperforming all Laya checkpoints on a 2,394-item eval) suggests the category will look different in six months.

The honest calibration finding from independent testing: Strands Decider 2B’s 72% standalone accuracy becomes 90% in a hybrid architecture where the LLM handles low-confidence cases. For most production use cases, the right question is not “can this model match Jev?” but “what is the accuracy-cost trade-off for routing n% of decisions through Decider and the remainder through an LLM?”

The Caveats with Decision Models

Decision models solve a real problem, and the category has momentum that is unlikely to reverse. But three limitations are worth knowing before you commit to one in production.

The gap on hard items is real

JevBench’s sealed accuracy scores reveal that open-source implementations perform significantly worse than Jev on genuinely ambiguous or complex decisions. A high-confidence prediction on a clear routing case is not the same as reliable performance on a borderline case. If your application depends on correct decisions at the tails of the distribution the current open-source options are not there yet. The hybrid architecture pattern (decision model for high-confidence cases, LLM fallback for the rest) is the engineering response to this limitation, and it is the pattern AWS explicitly recommends for Strands Decider.

Decision models do not reason

Returning a bounded choice from a predefined set is not the same as understanding why that choice is correct. A decision model that classifies a message as “billing” cannot explain its reasoning, cannot catch edge cases that fall outside its training distribution, and cannot generalize to option sets it has never seen. For applications where the decision requires understanding novel context, generating a brief explanation, or handling options that were not available at training time, a decision model is the wrong tool.

CPU deployment is slow

Strands Decider 2B on CPU takes 4+ seconds per decision which is slower than a fast LLM API call. The latency advantage of decision models is contingent on GPU inference. For local deployments on developer machines or edge environments without GPU acceleration, the latency profile changes substantially. Ollaya’s Laya-multilingual checkpoint is the fastest available at 8ms median on GPU; on CPU, expect that number to increase by 50–100x depending on hardware.

What This Means for Agent Pipeline Design

The emergence of decision models as a distinct layer in the agent stack changes the design pattern for pipelines running at scale. The previous options for agent control flow were: encode the decision in code (rules tables, if/else logic), make an LLM call and parse the output, or use structured outputs with a generative model. Each has costs: rules tables are brittle and cannot handle semantic complexity, LLM calls are expensive and slow, structured outputs still pay the full generation cost and add parsing overhead.

Decision models add a fourth option: a purpose-built judgment layer at sub-200ms latency and near-zero output cost, with calibrated confidence that enables threshold-based automation. The right architecture is not decision models instead of LLMs but it is decision models for the high-frequency bounded choices that do not require generative reasoning, with LLMs reserved for the long-horizon tasks that do.

The practical starting point for most teams is model routing. A decision model that correctly identifies 80% of requests as suitable for a cheap model (V4-Flash, Haiku) and routes the remaining 20% to a frontier model captures most of the cost savings of the category without requiring deep pipeline restructuring. At $0.042/M input tokens for Jev or self-hosted Strands Decider at zero marginal cost, the routing layer pays for itself on the first day of production traffic.

The category is three weeks old. The infrastructure around suggests that decision models are not a product announcement but a new layer in the agent infrastructure stack. The next time your agent makes a bounded choice, the question worth asking is whether that choice needs a language model at all.