The first multi-turn RL run you launch will spend most of its life doing something you would not call training. You watch GPU utilization sit under half, watch a step take eleven minutes, and go hunting for a bug in your gradient accumulation. There is no bug. The trainer is waiting on rollouts.

This is the part the best-practices literature skips. It tells you to use trajectory-level rewards, keep tasks in a learnable band, train close to production. All correct, and none of it explains why your cluster sits idle two thirds of the time.

So here is the framing that helps: a multi-turn RL run is an inference workload with a gradient step bolted onto the end. Every hour tuning GRPO clip ratios is an hour not spent on the component burning your budget. Your policy rollouts need your GPUs. Nothing else in the loop does, and that is what is crowding them out.

What Actually Breaks When You Go Multi-Turn

Single-turn RLHF has a clean shape. Sample a completion, score it, update the policy. One forward pass, one reward, no ambiguity about which tokens earned it.

Add turns and three things change at once.

Credit assignment stretches. A long-horizon trajectory that fails on turn nine may have gone wrong on turn two, when the agent picked the wrong tool and spent the rest of the episode recovering. The final reward is one scalar covering every decision in between, and the policy gradient has to smear it backwards across all of them.

Episode length becomes a random variable. Some trajectories finish in three turns, others grind into the turn cap. AWS’s SageMaker guidance recommends max_turns = ceil(N * 1.5), where N is the turn count a human needs, then watching for fewer than 10% of rollouts hitting that cap without producing an answer. The TSR paper logged average interaction turns between 3.1 and 7.7 on WebShop. A synchronous batch runs at the speed of its slowest trajectory.

Rollout volume explodes. This is the one that shows up on the invoice. AWS’s reference configuration is batch size 128 with group size 8, which is 1,024 complete agent trajectories per training step. At five turns apiece, that is 5,120 model calls before you compute a single gradient.

The effect is measurable. The ProRL Agent paper reports GPU utilization of 42% without rollout load balancing and 78% with it. More than half the fleet, idle, waiting on generation.

Why Multi-Turn RL Is Mostly an Inference Problem

Count tokens instead of steps and the real shape shows up immediately.

An agent turn is not a fresh request. Every turn replays the whole conversation: system prompt, task, every prior assistant message, every tool result. Prompt length grows linearly with turn index, so prompt tokens summed across a trajectory grow with the square of turn count. Doubling your turn budget does not double rollout cost. It roughly quadruples the prefill half of it.

Here is one training step under the AWS reference configuration, assuming a 2,000-token task setup, 400 tokens of agent output per turn, and 400 tokens of tool result each time. Substitute your own.

Eighteen million input tokens per step. A short run of 500 steps clears 9 billion before you count evaluation passes, crashed runs, or reward model traffic.

Set that against the gradient work. The same trajectories yield roughly 2 million trainable tokens, and a LoRA update at that scale is minutes of GPU time. Producing them touched ten times that many tokens, serially, gated by decode latency rather than FLOPs. That asymmetry explains your utilization graph, and it is why the rollout path deserves attention first.

Decouple the Rollout Generator From the Trainer

Most RL codebases couple the two by default. The trainer owns a vLLM instance, loads current policy weights into it, generates a batch, computes advantages, updates, reloads weights. Easy to reason about, and it guarantees your GPUs idle through generation.

The fix is structural. ProRL Agent splits rollout execution from policy updates outright, exposing generation as an HTTP service that takes task instances and returns finished trajectories with reward signals. Its asynchronous rollout pipeline (INIT, RUN, EVAL) runs independent worker pools, and the paper reports action execution falling from 0.78 to 0.42 seconds alongside that utilization gain.

Once rollout generation is an HTTP call, you choose what answers it. For the policy model mid-run there is no choice: those weights are yours and they change every step, so that traffic hits your own inference server. Everything else is fixed-weight inference. Environment simulation, tool responses backed by a model, LLM-judge scoring, baseline trajectories, and the whole pre-training evaluation sweep. Point that traffic at a hosted OpenAI-compatible API and your training environment stops competing with your trainer for GPUs.

DeepInfra speaks the OpenAI Chat Completions API, including function calling for the tools your environment exposes, so one client covers both paths. Swap base_url and model.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DEEPINFRA_API_TOKEN"],
    base_url="https://api.deepinfra.com/v1/openai",
)

def rollout(task, tools, policy_model, max_turns=8):
    """Run one trajectory. Returns messages plus per-turn token usage."""
    messages = [
        {"role": "system", "content": task.system_prompt},
        {"role": "user", "content": task.prompt},
    ]
    trace = []

    for turn in range(max_turns):
        resp = client.chat.completions.create(
            model=policy_model,
            messages=messages,
            tools=tools.schema,
            temperature=1.0,   # exploration, not production defaults
            top_p=0.95,
            max_tokens=2048,
        )
        msg = resp.choices[0].message
        # whitelist the fields: reasoning models return a trace, and replaying
        # it back inflates the prompt on every turn that follows
        messages.append(msg.model_dump(
            exclude_none=True, include={"role", "content", "tool_calls"}
        ))
        trace.append({"turn": turn, "usage": resp.usage.model_dump()})

        if not msg.tool_calls:
            return {"messages": messages, "trace": trace, "truncated": False}

        for call in msg.tool_calls:
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": tools.execute(call),
            })

    return {"messages": messages, "trace": trace, "truncated": True}
copy

Log trace from step one. Per-turn token usage tells you whether the turn cap binds and what the next run costs.

Three Model Roles, Three Different Models

Most teams point one model at everything and pay for it. A training loop has three inference roles with little in common.

The policy is the only role that must be open weights, because you cannot run a gradient update against somebody’s API. It is also the highest-leverage choice in the run: your agent’s ceiling is largely set by base model tool-call fidelity before RL touches it. Our roundup of OpenClaw models weighs tool-call accuracy the same way.

The environment simulator plays the user and fabricates tool responses. Short prompts, short completions, enormous volume, so pick purely on cost per call. GLM-4.7-Flash is a 30B mixture-of-experts activating roughly 3B parameters per token, which is why it prices where it does.

The judge scores finished trajectories, and its cost profile catches people out. It reads an entire 18,000-token trajectory and emits a score plus one justifying sentence. Input price dominates completely and output price barely registers, the reverse of how you normally shop for a model.

The baseline row is not something you train. It is the number to beat, and Kimi K3 draws the line between that row and the policy row about as hard as it can be drawn. Open weights, 2.8 trillion parameters, 104 billion of them active per token, a 1,048,576-token context window, and not a set of weights anyone is running gradient updates against on a normal budget. You call it, you score it, you have your ceiling. It posts 42.0 on SWE-Marathon, a benchmark for sustained autonomous software engineering, which is the closest published proxy for the long-horizon behavior you are about to spend a month training into something smaller. Run your eval set against K3, the earlier Kimi K2.6 release, or GLM-5.1 on agentic engineering first, and you find out whether a trained policy is a win or an expensive lateral move.

Reward Design That Survives a Multi-Turn RL Run

Dense rewards per turn are seductive. They fix credit assignment, they make the learning curve look wonderful, and they teach your agent to farm whatever proxy you rewarded. Give partial credit for calling a tool and you get an agent that calls tools.

The safer default is trajectory-level scoring with a few hard gates. Score the outcome, then subtract for specific enumerable failures: blew the turn cap, emitted malformed tool calls, stopped without an answer.

AWS makes one distinction worth stealing outright: separate completion from correctness. An agent that finishes and is wrong is a different animal from one that never finishes. Collapse both into a single low score and you destroy the signal telling you which problem you have. Their guidance also flags reward saturation. In one run the reward stalled near 3.7 against a maximum of 5.0, the shape you see when remaining headroom sits behind a gate the policy cannot reach.

The moment scoring involves judgment, your judge is a model, and that model is fixed-weight inference you can move off the training cluster.

import json

JUDGE_MODEL = "deepseek-ai/DeepSeek-V4-Flash-0731"

RUBRIC = """Score this agent trajectory.
Return JSON: {"task_solved": 0 or 1, "grounded": 0 or 1, "notes": "under 20 words"}
task_solved: did the final answer satisfy the original request?
grounded: was every factual claim supported by a tool result in the trace?"""

def render(messages):
    """Flatten a trajectory into the text the judge reads."""
    lines = []
    for m in messages:
        if m.get("tool_calls"):
            names = ", ".join(c["function"]["name"] for c in m["tool_calls"])
            lines.append(f"[{m['role']} calls] {names}")
        if m.get("content"):
            lines.append(f"[{m['role']}] {m['content']}")
    return "\n".join(lines)


def malformed_tool_calls(trajectory):
    """True if the policy ever emitted tool arguments that will not parse."""
    for m in trajectory["messages"]:
        for c in m.get("tool_calls") or []:
            try:
                json.loads(c["function"]["arguments"])
            except (json.JSONDecodeError, TypeError):
                return True
    return False


def score(trajectory):
    resp = client.chat.completions.create(
        model=JUDGE_MODEL,
        messages=[
            {"role": "system", "content": RUBRIC},
            {"role": "user", "content": render(trajectory["messages"])},
        ],
        response_format={"type": "json_object"},
        temperature=0.0,
        max_tokens=128,
        extra_body={"reasoning_effort": "none"},  # or thinking eats the 128
    )
    try:
        j = json.loads(resp.choices[0].message.content or "")
    except json.JSONDecodeError:
        return {"task_solved": 0, "grounded": 0, "penalties": 0.0,
                "total": 0.0, "judge_failed": True}

    penalties = 0.0
    if trajectory["truncated"]:
        penalties += 0.5              # hit the turn cap, never answered
    if malformed_tool_calls(trajectory):
        penalties += 0.3

    return {
        "task_solved": j["task_solved"],
        "grounded": j["grounded"],
        "penalties": penalties,
        "total": j["task_solved"] + 0.3 * j["grounded"] - penalties,
        "judge_failed": False,
    }
copy

Note the return shape. Log the components separately, never the sum. When training reward climbs while held-out reward flatlines, that breakdown is the only thing telling you where the reward hacking is. Watch judge_failed for the same reason. A judge that returned nothing parseable is not a trajectory that scored zero, and letting the two average together drags your reward curve down for reasons that have nothing to do with the policy.

The Cost Math

Take that 500-step run and price the two roles you would hand to a hosted endpoint. Editorial arithmetic, not a benchmark, and every input sits above so you can redo it. Over 500 steps you produce 512,000 judgments, each reading an 18,000-token trajectory and returning 128 tokens, plus 2.05M simulator calls at four tool results per trajectory, roughly 1,500 tokens in and 400 out.

About $1,261 for both, in opposite shapes: the judge’s bill is 98% input, the simulator’s is 64% output. Running those same 13.2B tokens in place means every H100 hour spent fabricating a tool result is an hour not spent on a rollout. The question stops being whether to use a hosted judge and becomes why you would run one on hardware you are trying to train on.

Then the lever that moves the most money. Look at the turn-by-turn table again: turn three’s prompt is turn two’s prompt plus two messages. Every turn is a strict extension of the one before it, which makes agent rollouts close to the ideal case for prefix caching. Context caching collapses that quadratic prefill term back toward linear. On DeepInfra, cached input for GLM-4.7-Flash runs $0.01 per million against $0.06 uncached, and DeepSeek-V4-Flash-0731 drops to $0.016 from $0.08.

A six-to-one ratio, and it lands on the line item you just priced. Cache the simulator’s context and its $184 input bill drops to $31. The ratio only gets better as the model gets bigger. Kimi K3 reads cached input at $0.285 against $2.85, ten to one, which is what makes a baseline sweep something you can afford to re-run every time the eval set changes rather than once at the start.

Earn the Right to Train

Training is the most expensive way to improve an agent, and plenty of teams reaching for it have not exhausted cheaper options.

The gate is whether your base model already succeeds sometimes. When it never does, there is nothing to train on: zero rewards produce zero gradients, and the parameters sit unchanged for the entire run. That same work found every algorithm it tested could learn once the base model started around a 20% success rate, and that threshold cuts both ways. Below it, rewards are too sparse for the policy to find signal. Comfortably above it, a stronger base model and a tighter scaffold often get you there for the price of an afternoon.

Two things to do first.

Build the external evaluation harness regardless. You need a held-out set scored independently of training reward whatever you decide, and building it first tells you whether you have a model problem or a prompt problem. AWS notes most training failures surface within about 30 steps, which only holds if you have the evaluation in place to see them.

Then measure the ceiling of an untrained frontier open model inside a real agent framework. LangChain Deep Agents running on Nemotron 3 Ultra is a different proposition from a bare chat loop, and tool-calling agents with real search integration close more of the gap than teams expect.

Getting Started

Start by pricing your own loop. Pull turn counts and token usage from the trace object above, run them against current rates, and see which line item dominates. It is rarely the one teams budget for.

Model pages: GLM-4.7-Flash, DeepSeek-V4-Flash-0731, Qwen3.6-35B-A3B. Full rates live on the pricing page, integration details in the docs.

If you are running RL on open weights, we want to hear what worked for you. Reach us at feedback@deepinfra.com, join the community on Discord, or find us on X at @DeepInfra.