Your monitoring dashboard says the API is fast and your invoice says the same thing in different units. Somewhere between the two, cost per completed request disappears.

This guide compares the best AI inference platform options for speed and cost on the same footing: measured latency, measured throughput, current prices, and the math that turns them into your bill. The metric that decides everything is cost per completed task, not price per million tokens.

One thing to settle before we get to numbers. Every platform here sells two products. A shared serverless endpoint pools your traffic with everyone else’s, bills per token, and scales to zero when idle. A dedicated endpoint reserves silicon for you, bills by the hour, and behaves the same at 3am as it does at peak. Public benchmarks measure the first one, because it is the only tier a third party can point a script at without signing a contract. The second one is where latency-sensitive production traffic usually ends up, and hardly anyone publishes comparable numbers for it.

The gap matters most for platforms whose serverless tier is tuned for throughput per dollar rather than per-stream speed. The batching that holds a cost floor down is the same batching that costs a benchmark its tokens per second. So read every speed figure below as a measurement of one endpoint on one day, not a verdict on a company.

Three Numbers Decide This Pick

Two of the three numbers get confused with each other constantly, and this confusion leads to picking the wrong platform.

• Time to first token (TTFT). The gap between your request leaving the client and the first streamed token arriving. Prefill handling and queue depth drive it.
• Output throughput. How fast the rest of the response streams once generation starts, in tokens per second. Decode speed and batching policy drive it. Our KPIs guide covers how the two interact under load.
• Cost per completed task. Price per million tokens is an input rate, whereas the bill is a separate calculation. The bill depends on how many tokens the task consumes, and output tokens cost four to eight times more than input tokens, by AI Superior’s pricing breakdown. Reasoning models widen the gap further, because they emit long chains of thinking tokens before answering. A cheaper model that thinks longer can cost more on the same question than a pricier model that answers directly. Inference economics at scale goes deeper into the true cost of a task.

How We Picked the Best AI Inference Platform for Speed and Cost

Most speed tables are vendor-published, single-stream, and undated. Ours states its method first. Every figure carries a source, date, and label: measured for third-party test runs, vendor-published for platform claims. Prices verified against provider pages August 3 to 15, 2026.

The criteria are all measurable:

• TTFT at realistic prompt length, not empty prompts
• Sustained throughput at concurrency, not single-stream peaks
• Price per million input and output tokens, including cached and batch rates
• Cold start behavior on serverless endpoints
• Rate limits and queueing under load
• Hardware transparency

One limit the criteria cannot fix, stated plainly because it shapes every speed column here. Each third-party measurement in this guide was taken against a public serverless endpoint, since that is the only tier a benchmark can reach without a contract. Dedicated deployments are configured per customer, so no independent source publishes a comparable number for them, and any platform that sells both will read slower here than its hardware allows. The endpoint is labeled on every figure rather than papered over.

Published numbers diverge because sources test different endpoints, quantizations, and weeks. Groq alone appears in our sources at 450 to 550 tokens per second on Mixtral (DeployBase, February 2026), around 476 on gpt-oss-120B (Artificial Analysis via Infrabase), and roughly 142 on Llama 3.3 70B (llm-benchmarks.com, June 2026). Same provider, three models, three results. DeepInfra’s catalog shows the same split: Turbo FP8 endpoints trade per-stream speed for capacity against bfloat16 ones, so attach an endpoint name to any number before trusting it.

Why One Throughput Number Is Not a Platform Verdict

• Hardware generation and memory bandwidth. Decode is memory-bound. Tokens per second track how fast the accelerator feeds weights to compute. Groq’s LPU, Cerebras’ wafer-scale engine, and SambaNova’s RDU all restructure that feed.
• The serving engine. vLLM, SGLang, and TensorRT-LLM make different tradeoffs in continuous batching, KV-cache management, and prefill scheduling, moving throughput double-digit percentages on identical hardware. Our vLLM versus SGLang comparison lays out the split.
• Batching policy, which is really a pricing decision. A serverless endpoint that batches aggressively serves more concurrent requests per GPU, and a low per-token rate is built out of exactly that. The same batching lowers tokens per second on any single stream, because your request now shares a decode step with other people’s. When a cheap platform benchmarks slow on its public endpoint, that is usually the tradeoff working as designed rather than the silicon falling short. The fix sits one tier up, on a dedicated endpoint where the batch window is yours to set.
• Defaults. FP8 quantization, speculative decoding, and multi-token prediction each buy speed at a model-dependent quality cost. Two listings of the same model can be different stacks wearing the same name. Ask which defaults are on for the endpoint you price.
• Behavior under concurrency. Every number in a vendor speed table is a single-stream measurement. Production traffic is not single-stream. At 50 or 100 concurrent requests, TTFT climbs, throughput per request drops, and some providers start queueing. The metric that matters is p95 latency under your expected concurrency. The median on an empty endpoint, however, tells you little. Load-test the endpoint yourself at realistic request rates before committing. Rate limits compound the problem. A provider that caps you at 60 requests per minute will queue your 61st regardless of how fast its silicon is.
• Rate limits. A provider that caps you at 60 requests per minute will queue your 61st regardless of how fast its silicon is, and the cap usually tracks your billing tier rather than the hardware. Groq’s free tier tops out at 30 requests and 6,000 tokens per minute depending on model, and batch and flex processing only unlock on a paid Developer account (Groq rate limits).

None of this needs a benchmark vendor. Your prompt is a better test than their prompt, and the whole measurement is a stopwatch around a stream:

import os
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.deepinfra.com/v1/openai",
    api_key=os.environ["DEEPINFRA_API_TOKEN"],
)

start = time.perf_counter()
first_delta_at = None
deltas = 0

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": your_real_prompt}],
    stream=True,
)
for chunk in stream:
    if not chunk.choices[0].delta.content:
        continue
    if first_delta_at is None:
        first_delta_at = time.perf_counter()
    deltas += 1

end = time.perf_counter()
print(f"TTFT: {first_delta_at - start:.3f}s")
print(f"Output throughput: {deltas / (end - first_delta_at):.1f} tok/s")  # ~1 token per delta
copy

Run that against two providers, at your prompt length and your concurrency, and the table above becomes a starting point rather than an answer. Run it from where your servers actually live, too: network distance to the endpoint lands in the TTFT column whether or not the provider caused it.

One Model, Seven Bills for the Same Answer

Run the same model on every platform and the comparison gets clean. Llama 3.3 70B Instruct is the most widely hosted open model in this market. The table sorts on the column that reaches an invoice, cost per 1,000 completed requests, with the measured speed columns sitting beside it rather than in front of it. One model, one source per figure, stated conditions, and every speed number taken against that provider’s shared serverless endpoint.

Throughput and TTFT: Artificial Analysis, Llama 3.3 70B, 10,000-token input prompt, 72-hour median (measured). Prices: Infrabase, verified on each provider’s own pricing page, August 3, 2026. Cells read “not tested” where that provider’s endpoint carries no published measurement on this model, which is more useful than a number borrowed from a different one.

The second column is the one that lands on an invoice. It prices 1,000 requests of the support-agent workload used later in this guide, 800 input and 300 output tokens each, so the arithmetic is 0.8 times the input rate plus 0.3 times the output rate (estimate, inputs shown).

Read the two halves against each other and they refuse to line up. Per-token rates run 10x from cheapest to dearest on identical weights, and measured throughput on these shared endpoints runs 17x from slowest to fastest, but the orderings are close to unrelated. Paying more does not reliably buy speed here, and paying less does not reliably cost it. Note also that a 10,000-token prompt is the hard case for shared serverless capacity, and it is roughly twelve times longer than the support-agent workload this guide prices. Pick the column that matches your prompt length before you pick a platform.

Sorted on the bill, the field breaks into two clusters and a straggler. DeepInfra, OpenRouter, and Novita AI sit within a nickel of each other on the floor. Groq and SambaNova charge four to five times that, which is a fair trade when TTFT is the product and a poor one when it is not, and the two custom-silicon plays sit closer to each other than either sits to anything else here: five tokens per second and 18 percent on the bill separate them. Together AI is the straggler, dearest per completed task and mid-pack on throughput, a position no workload asks for on this model. Scaleway is absent from the chart because its rate card is denominated in euros, not because it is slow. The pattern holds beyond one model: our GLM-5.1 provider comparison found blended rates from $0.74 to $1.70 per million tokens across ten providers.

Eight Platforms, Reviewed One by One

DeepInfra

DeepInfra posts the cheapest published rate on the benchmark model at $0.10 in and $0.32 out per million for Llama 3.3 70B Instruct Turbo, which works out to $0.18 per thousand support-agent requests. The speed number needs its context to mean anything. Artificial Analysis measures that shared Turbo FP8 endpoint at 18 output tokens per second with 1.95s TTFT on a 10,000-token prompt (measured, 72-hour median), near the back of the fourteen providers it tracks on this model. It’s the batched, scale-to-zero tier, and the batching is a large part of what the $0.10 input rate is made of. Change the prompt length or the tier and the figure moves. llm-benchmarks.com clocks DeepInfra’s fastest listing, Nemotron 3.5 Lightning, at 171 tokens per second with near-zero TTFT on a short prompt (measured, June 2026), and traffic that needs deterministic latency runs on dedicated GPUs instead, priced later on this page.

Quantization, prompt length, and endpoint tier move throughput more than the logo on the invoice does. The catalog runs from Nemotron 3 Nano 30B A3B at $0.05 and $0.20 through DeepSeek-V3.2 at $0.26 and $0.38 up to Kimi K2.6 at $0.75 and $3.50, on bare-metal US infrastructure with zero retention. The API is OpenAI-compatible, so switching to DeepInfra is a one line change:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.deepinfra.com/v1/openai",
    api_key=os.environ["DEEPINFRA_API_TOKEN"],
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Explain TTFT in one sentence."}],
)

print(response.choices[0].message.content)
copy

Price is not a single number here either. Every serverless model runs on three scheduling tiers: Standard at the listed rate, Priority at 1.5x for faster time to first token, and Flex at 0.8x for asynchronous work that can wait in line, with cached input billed separately and well below the standard input rate on most models (DeepInfra pricing, August 24, 2026). When serverless stops fitting, the same catalog runs on dedicated GPUs by the hour: $2.20 for an H100, $2.69 for an H200, $3.69 for a B200. Hold those three numbers. They come back when the dedicated-capacity vendors further down this list quote theirs.

Pros

• Cheapest published rate on the benchmark model, and the lowest cost per completed task in the whole table at $0.18 per thousand support-agent requests
• Widest catalog here, roughly 360 open models, from $0.05 per million on Nemotron 3 Nano through DeepSeek-V3.2 up to Kimi K3
• Three scheduling tiers on every serverless model, so identical weights bill at 0.8x, 1x, or 1.5x depending on how much first-token time is worth to you
• Dedicated GPUs at roughly a third of what Baseten charges for the same B200
• OpenAI-compatible API, bare-metal US infrastructure, zero data retention

Cons

• Artificial Analysis puts the shared serverless Turbo FP8 endpoint at 18 output tokens per second on a 10,000-token prompt, near the back of the fourteen providers it tracks on this model
• The shared serverless tier trades per-stream speed for the cost floor, so latency-critical traffic belongs on a dedicated endpoint rather than the public one benchmarks measure
• Throughput swings hard with endpoint and quantization, so the number that matters to you is rarely the one on a comparison chart

Recent Updates

• August 19, 2026. Sandboxes shipped: isolated environments for running agent-generated code on the same account as inference, which closes a gap that previously sent agent builders to a second vendor.
• August 11, 2026. Day-zero support for NVIDIA Nemotron 3.5 Lightning at $0.08 in and $0.20 out per million, currently the fastest model in the catalog at 171 tokens per second on the shared serverless endpoint.
• August 10, 2026. Kimi K3 landed at $2.85 in and $14.25 out per million with a 1 million token context (Kimi K3 on DeepInfra), which resets the top of this catalog well above the Kimi K2.6 rates quoted earlier in this section.

Groq

Groq’s LPU silicon is purpose-built for decode. The rolling table at llm-benchmarks.com puts Llama 3.3 70B at 142 tokens per second with near-zero TTFT on a short prompt (measured, June 2026), and Artificial Analysis clocks the same model at 297 tokens per second with 1.01s TTFT on a 10,000-token prompt (measured, 72-hour median). Both runs hit Groq’s public serverless endpoint, which is the only Groq tier a third party can test. Tokens cost more here: $0.59 in and $0.79 out per million on the benchmark model.

Price is not a single number here either. Every serverless model runs on three scheduling tiers: Standard at the listed rate, Priority at 1.5x for faster time to first token, and Flex at 0.8x for asynchronous work that can wait in line, with cached input billed separately and well below the standard input rate on most models (DeepInfra pricing, August 24, 2026). When serverless stops fitting, the same catalog runs on dedicated GPUs by the hour: $2.20 for an H100, $2.69 for an H200, $3.69 for a B200. Hold those three numbers. They come back when the dedicated-capacity vendors further down this list quote theirs.

Pros

• Fastest first token in the serverless table at 1.01s on a 10,000-token prompt, and near-zero TTFT on short ones
• 297 output tokens per second on the benchmark model’s serverless endpoint, second only to SambaNova and 17x the cost floor
• Batch API bills at half the synchronous rate for jobs that tolerate a 24-hour to 7-day window
• Cached tokens do not count against rate limits, which is the constraint that bites first on this platform

Cons

• $0.59 in and $0.79 out per million is roughly 4x the cost per completed task of the cheapest option here
• Groq hosts what Groq chose to host, so the model you standardize on has to be one of them
• Batch pricing and prompt caching do not stack: batch tokens bill at the batch rate whatever their cache status
• Rate limits, not silicon, are usually what caps throughput on a real account

Recent Updates

• August 24, 2026. Groq announced it is among the first to bring NVIDIA’s Groq 3 LPX and Vera Rubin NVL72 systems to market, so the pure-LPU story now runs alongside NVIDIA silicon in the same cloud.
• August 17, 2026. Closed a $350 million Series A to expand inference capacity, following a $650 million raise in June.
• April 18, 2026. The most recent dated entry on the public changelog added MiniMax M2.5 and Qwen3-VL 32B for enterprise accounts plus new Orpheus voices. Product news has been landing in the newsroom rather than the changelog since, which makes version-pinning against a documented history harder here than on Together or Baseten.

Cerebras

Cerebras serves gpt-oss-120B at around 3,000 tokens per second on its wafer-scale engine, a vendor-published April 2026 figure Infrabase calls the highest measured throughput in the category. Its pricing page claims inference 20x faster than OpenAI and Anthropic (vendor-published, undated).

Access is tiered rather than posted per token, which is the practical catch. A new account carries $5 in free credits, the self-serve Developer tier starts at $10 and lifts limits more than 10x above the free tier, and the coding plans are fixed-price: Cerebras Code Pro at $50 a month and Code Max at $200 with rate limits up to 1.5 million tokens per minute (Cerebras pricing and its pay-per-token announcement, August 24, 2026). 

The pay-per-token catalog is three models deep, Qwen3 235B, gpt-oss-120B, and Qwen3 Coder 480B, and anything outside it means reserved capacity and a sales call. Cerebras also resells through AWS Marketplace, OpenRouter, Hugging Face, and Vercel, so the cheapest way to A/B it against your current provider is usually a gateway you already pay for.

Pros

• Around 3,000 tokens per second on gpt-oss-120B, the highest throughput claimed in the category (vendor-published, public endpoint)
• Fixed-price coding plans, so a heavy agent workload has a known monthly ceiling instead of a variable bill
• $5 in free credits on a new account, and a self-serve Developer tier from $10 that lifts limits more than 10x
• Resold through AWS Marketplace, OpenRouter, Hugging Face, and Vercel, so you can A/B it through a gateway you already pay for

Cons

• The pay-per-token catalog is three models deep, and anything outside it means reserved capacity and a sales call
• No published per-token rate on the benchmark model, so cost per completed task cannot be compared directly against the rest of this table
• Access tiers, not tokens, are the pricing unit, which makes forecasting awkward for spiky production traffic

Recent Updates

• July 27, 2026. GLM 4.7 was scheduled for deprecation on August 17, 2026, with the reasoning_logprobs default change held back until after it (Cerebras changelog).
• July 22, 2026. API version 2 became the default, bringing stricter validation and schema changes. Pinned clients built against v1 need a look before they break.
• July 16, 2026. Rate limiting moved to a dual-bucket model with separate uncached and total tokens-per-minute limits, and new accounts started receiving $5 in credits once a payment method is verified.
• June 29, 2026. Gemma 4 31B and image input entered public preview, the first vision capability on the platform, later expanded to 10 images per request for Developer and Enterprise accounts.

SambaNova

SambaNova’s RDU maps the model graph onto the chip directly and tiers memory across SRAM, HBM, and high-capacity pools. The SN50 announcement claims 5x the top speed of Blackwell B200 and over 3x throughput on agentic inference including Llama 3.3 70B (vendor-published). SN50 ships in the second half of 2026, so today’s SambaCloud runs the prior SN40L generation, plus prompt caching to cut TTFT.

SambaCloud posts rates, and they are mid-field rather than premium: the benchmark model at $0.60 in and $1.20 out per million, gpt-oss-120B at $0.22 and $0.59, MiniMax-M2.7 at $0.60 and $2.40 (SambaNova Cloud pricing, August 24, 2026). That buys the highest measured throughput on the benchmark model in this table, 302 output tokens per second at 1.76s TTFT on the 10,000-token prompt against its shared serverless endpoint, though Groq reaches its first token in well under two-thirds the time. 

The agentic pitch shows up in the caching numbers. Cached tokens bill 90 percent below standard input, $0.06 against $0.60 per million on MiniMax-M2.7, and SambaNova’s own measurements put the TTFT saving at 33 percent on short contexts and 91 percent at 192k, where first-token time falls from 40.6 seconds to 3.7 (SambaNova, prompt caching, vendor-published). Caching is live on MiniMax-M2.7 only for now, so check before you budget around it.

Pros

• Highest measured output throughput on the benchmark model in this table at 302 tokens per second, on the shared serverless endpoint
• Mid-field pricing rather than premium, at $0.60 in and $1.20 out per million on Llama 3.3 70B
• Cached tokens bill 90 percent below standard input, $0.06 against $0.60 per million on MiniMax-M2.7
• Prompt caching cuts TTFT by 33 percent on short contexts and 91 percent at 192k, where first-token time falls from 40.6 seconds to 3.7

Cons

• Serverless TTFT of 1.76s is well behind Groq’s 1.01s on the same tier, so the throughput lead does not translate into a snappier first response
• Prompt caching is live on MiniMax-M2.7 only for now, so check before you budget around it
• The SN50 headline numbers are forward-looking, and today’s SambaCloud runs the prior SN40L generation
• No published hourly rate for dedicated capacity, which makes the serverless-to-dedicated crossover impossible to model from outside

Recent Updates

• August 24, 2026. MiniMax M3 went live on SambaCloud, with SambaNova claiming the fastest serving of that model anywhere.
• July 16, 2026. Prompt caching launched on MiniMax-M2.7, the feature the agentic pitch rests on.
• July 7, 2026. The SambaCloud release added prompt caching platform-side, security hardening, and streaming fixes for tool-call requests (SambaCloud release notes).
• July 1, 2026. SambaCloud added support for the Anthropic Messages API, so Claude-shaped clients can point at it without a rewrite.

Novita AI

Novita AI prices Llama 3.3 70B at $0.135 in and $0.40 out per million (Infrabase, August 3, 2026), a rounding error above the cost floor, and Artificial Analysis measures 39 output tokens per second with 1.75s TTFT on a 10,000-token prompt (measured, 72-hour median). It’s the same shared-serverless tradeoff DeepInfra’s Turbo FP8 endpoint makes, priced at roughly 28 percent more per completed task

Sustained load has somewhere to go, which the serverless-only reading of this platform misses. Dedicated endpoints run isolated replicas at $2.99 an hour for an H200, $1.99 for an H100, and $0.61 for an RTX 4090, billed per second on active replicas with scale-to-zero, min and max replica counts you set, and a scale-down delay you tune. Custom weights and LoRA adapters load there, and the tier carries a latency SLA that the shared serverless endpoints do not. Below that sit raw GPU instances from about $0.33 an hour for a 24GB RTX 4090 if you would rather run the serving stack yourself (Novita AI pricing, August 24, 2026).

Pros

• $0.23 per thousand support-agent requests, a rounding error above the cost floor
• Dedicated endpoints on the same account at $1.99 an hour for an H100 and $2.99 for an H200, with per-second billing and scale-to-zero
• Custom weights and LoRA adapters load on the dedicated tier, which also carries a latency SLA the shared endpoints do not
• Raw GPU instances from about $0.33 an hour if you would rather run the serving stack yourself

Cons

• 39 output tokens per second at 1.75s TTFT on the shared serverless endpoint puts it near the bottom of the measured field, so latency-sensitive traffic has to buy the dedicated tier to become predictable
• The catalog turns over fast, and recent retirement notices have given as little as two weeks
• Scale-to-zero means cold starts, which is the tax that pays for the per-second billing

Recent Updates

• August 11 to 14, 2026. Four models were scheduled for retirement, including three inclusionAI Ling and Ring models and kwaipilot/kat-coder-pro, the last with no replacement named. One of them had been released six days earlier. If you pin model IDs in production, subscribe to this changelog rather than checking it quarterly (Novita changelog).
• August 3, 2026. The FLUX and FLUX Kontext image models were retired in favor of Qwen-Image, along with a batch of video models, on 16 days’ notice.
• June 23, 2026. A Cost Observability Center launched with three-level cost drill-down from organization to team to member to API key, plus budget quotas per user and per key. Useful on a platform whose whole pitch is the cost floor.

Together AI

Together AI prices the same model at $1.04 per million in both directions (Infrabase, August 3, 2026), the dearest per-token rate in the table, against 76 output tokens per second at 1.46s TTFT. DeployBase’s week-long run logged Mixtral at 200ms TTFT and 800-plus tokens per second in batch (measured, February 2026), which is the figure the marketing leads with. Mind the gap between the two: one is a batch aggregate on a different model, the other is per-stream on a long prompt.

What the premium buys is the rest of the platform. Fine-tuning is priced per million training tokens, roughly $1.50 to $4.12 for models in the 17B to 69B range with a $4 minimum per job, and the tuned checkpoint serves from the same endpoint you already call. Dedicated capacity runs $5.49 an hour for an HGX H100 and $8.99 for a B200 (Together AI pricing, August 24, 2026), against the $2.20 and $3.69 DeepInfra charges for the same silicon. Run inference only and you are paying for a training platform you never open.

Pros

• Fine-tuning and serving on one bill, priced per million training tokens with a $4 minimum per job
• The tuned checkpoint serves from the same endpoint you already call, so there is no second integration
• 76 output tokens per second at 1.46s TTFT is mid-pack rather than slow
• The most actively maintained public changelog in this comparison, with dated entries landing several times a week

Cons

• $1.04 per million in both directions is the dearest per-token rate in the table, and $1.14 per thousand requests is more than 6x the cost floor
• Dedicated GPUs run $5.49 an hour for an HGX H100 against DeepInfra’s $2.20 for the same silicon
• Run inference only and you are paying for a training platform you never open
• The 800-plus tokens per second figure the marketing leads with is a batch aggregate on Mixtral, not a per-stream number on the model you are pricing

Recent Updates

• August 24, 2026. Fine-tuning quality improvements shipped for Qwen and Nemotron models (Together AI changelog).
• August 21, 2026. deepcogito/cogito-v2-1-671b was removed from the catalog, a reminder that the churn cuts both ways here too.
• August 15, 2026. DeepSeek-V4-Pro was added as a serverless model with a 1 million token context, followed by GLM-5.2 fine-tuning on August 13 and Qwen3.8-2.4T and Nemotron 3.5 Lightning on August 12.
• May 29, 2026. Pricing changed on several models including Llama-3.3-70B-Instruct-Turbo, the exact endpoint this comparison prices. Re-check the rate card before you commit a budget to it.

Scaleway

Scaleway answers a question the rest of this list cannot: where the data physically sits. Its Generative APIs are serverless, OpenAI-compatible, and served from the Paris region, and Llama 3.3 70B costs €0.90 per million tokens in both directions (Scaleway pricing, August 24, 2026). Artificial Analysis measures 83 output tokens per second at 1.37s TTFT, mid-field, and more than four times what the same benchmark records on DeepInfra’s Turbo FP8 endpoint.

The flat rate cuts both ways. Output at €0.90 undercuts Together AI’s $1.04, so verbose and reasoning-heavy work does well here. Input at €0.90 is nine times DeepInfra’s $0.10, so RAG traffic that ships a document with every call pays for the symmetry. The first million tokens each month are free, the Batches API halves the rate for jobs that can wait, and the rest of the catalog keeps the same shape: DeepSeek-V4-Flash-0731 at €0.40 and €0.80, GLM-5.2 at €1.80 and €5.50. 

When serverless is not enough, Managed Inference rents dedicated GPUs by the hour inside the same EU footprint, €0.93 for an L4, €1.72 for an L40S, €3.40 for an H100, and €30.06 for an eight-way H100 SXM node.

Pros

• Serverless open weights served from the Paris region, which is the only answer here to a strict EU data residency requirement
• Output at €0.90 per million undercuts Together AI’s $1.04, so verbose and reasoning-heavy work does well
• First million tokens each month are free, and the Batches API halves the rate for jobs that can wait
• Managed Inference rents dedicated GPUs inside the same EU footprint, from €0.93 an hour for an L4
• Generative APIs events now flow into Audit Trail, which matters if residency is why you are here in the first place

Cons

• Input at €0.90 is nine times DeepInfra’s $0.10, so RAG traffic that ships a document with every call pays for the flat rate
• The catalog is smaller than the US-hosted platforms, and the deprecation policy is enforced on schedule
• Pricing is denominated in euros, so the cost per completed task is not directly comparable to the dollar rates in the main table

Recent Updates

• August 11, 2026. DeepSeek-V4-Flash arrived on Generative APIs, described by Scaleway as the most cost-efficient model for agentic and coding tasks in most July 2026 benchmark evaluations (Scaleway changelog).
• July 8, 2026. Generative APIs events became available in Audit Trail, which is the compliance half of the residency pitch finally catching up to the inference half.
• July 1, 2026. Devstral 2, Voxtral Small, Gemma 3, Pixtral, and Qwen 3 Coder reached end of life on serverless. Requests to all but Voxtral Small now route automatically to a model with similar pricing and capabilities, so nothing breaks, but you no longer control which weights answer.
• June 26, 2026. GLM-5.2 was added, described as the best open-weight model for agentic and coding tasks in most June 2026 benchmark evaluations.

Baseten

Baseten’s Model APIs price Kimi K2.6 at $0.95 in and $4.00 out per million tokens (vendor page, August 13, 2026) against $0.75 and $3.50 on DeepInfra. The token rate is not the pitch. What Baseten sells is running your own weights on isolated hardware under a compliance story, and the rate card follows from that.

Model APIs and dedicated deployments are two products on one bill, and they price differently against the field. On the API side DeepSeek-V4-Flash runs $0.13 in and $0.26 out against DeepInfra’s $0.09 and $0.18 for the same weights. On the dedicated side, per-minute billing runs from $0.01052 for a T4 to $0.10833 for an H100 and $0.16633 for a B200, which is $9.98 an hour for the B200 against DeepInfra’s $3.69 (Baseten pricing, August 24, 2026). 

The premium buys an operational envelope rather than tokens: fast cold starts, autoscaling you configure, SOC 2 Type II and HIPAA, and Basic, Pro, and Enterprise tiers where Pro adds volume discounts and priority GPU access.

Pros

• Runs your own weights on isolated hardware with fast cold starts and autoscaling you configure
• SOC 2 Type II and HIPAA, plus Basic, Pro, and Enterprise tiers where Pro adds volume discounts and priority GPU access
• Cached input tokens bill at a discounted rate automatically on most Model APIs, $0.16 against $0.95 standard on Kimi K2.6
• Per-minute billing on dedicated deployments is finer-grained than the hourly billing most of this list uses

Cons

• Token rates run above DeepInfra on the same weights, $0.95 and $4.00 against $0.75 and $3.50 on Kimi K2.6, and $0.13 and $0.26 against $0.09 and $0.18 on DeepSeek-V4-Flash
• A dedicated B200 costs $9.98 an hour against DeepInfra’s $3.69 for the same card
• Per-minute GPU billing idles expensively under spiky traffic
• No published measurement on the benchmark model, so speed has to be taken on trust or measured yourself

Recent Updates

• August 19, 2026. Runtime OIDC shipped, letting deployments authenticate to cloud providers without long-lived credentials, which is the kind of feature the compliance pitch is built on (Baseten changelog).
• August 14, 2026. DeepSeek V4 Pro 0813 became available through Model APIs on the OpenAI-compatible endpoint.
• July 23, 2026. GLM 5.2 Fast launched as a new Fast tier, alongside API key management keys for automated team key administration.
• April 17, 2026. Discounted cached-input pricing arrived on Model APIs, with cache hits billed at the lower rate automatically rather than on request.

When Serverless Is the Wrong Tier

Every rate card above prices the shared tier, and the shared tier is the wrong product for a meaningful slice of production traffic. Once predictable latency becomes a requirement rather than a preference, you stop buying tokens and start renting silicon. The comparison changes shape with it: hours instead of tokens, a posted rate instead of a benchmark, and a duty cycle instead of a leaderboard.

Hourly rates from each platform’s pricing page, August 24, 2026, sourced in the reviews above. Baseten publishes per-minute rates, so the hourly figures here are those times 60.

Novita AI undercuts DeepInfra by 21 cents on an H100 and hands it back on an H200. The wider gaps open against the platforms that bundle an operational envelope into the hourly rate. Together AI charges roughly 2.5x for an H100 and 2.4x for a B200, and Baseten close to 3x on both, which buys SOC 2 Type II, HIPAA, fast cold starts, and autoscaling you configure rather than tokens you consume. Whether that premium earns its place is a compliance question more than a performance one.

The names missing from that table are the ones the speed argument belongs to. Groq, Cerebras, and SambaNova post no hourly rate for reserved capacity. Cerebras sells fixed-price coding plans and routes everything outside its three pay-per-token models to a sales call, and the other two do much the same past the public endpoint. It’s a reasonable way to sell custom silicon in short supply. It also means the tier where their latency advantage would become contractual is the tier you cannot price without talking to someone, which is worth knowing before a benchmark table talks you into a migration.

The Closed-Model Cost Baseline

Open weights make sense only measured against closed-API pricing. The premium anchors as of early 2026, per AI Superior’s snapshot:

DeployBase’s February 2026 run logged Anthropic’s Claude Opus 4.6 at $5.00 in and $25.00 out with 35 to 45 tokens per second (measured). Enterprise clouds compete on compliance: AWS Bedrock’s Nova Micro reaches about 118 tokens per second with roughly 380ms TTFT (llm-benchmarks.com), and Google’s Gemini stack posted 180ms TTFT in DeployBase’s tests. Those are public API endpoints too, the same tier as every open-weight measurement above. The gap between these stickers and the open-weight table is what you actually buy by picking open models. Our open versus closed price gap breakdown covers the savings.

One Workload, Five Monthly Bills

Take a support chat agent: 10,000 requests a day, 800 input tokens and 300 output tokens per request. Over a 30-day month that is 240 million input tokens and 90 million output. Rough monthly totals at each platform’s Llama 3.3 70B rate, arithmetic shown:

Same model, same tokens, a 6.5x spread across the dollar rows. Now reasoning token overhead, something most tables hide. Swap in a reasoning model and the same questions can triple the output tokens. DeepSeek-R1-0528 at $0.50 in and $2.15 out would bill roughly $120 for the input and $581 for 270 million output tokens on this workload, about $700 a month (estimate, inputs shown). Its per-token rates are five to seven times DeepInfra’s Llama 3.3 70B Turbo. Its completed task is thirteen.

Rather than trusting a table that ages (inference pricing has compressed by orders of magnitude since 2022 and continues to fall), price your own workload against the live catalog. DeepInfra publishes every rate at api.deepinfra.com/models/list, so the arithmetic above fits in a few lines:

import requests

CATALOG = "https://api.deepinfra.com/models/list"


def rates_per_million(model_name):
    """Live input and output rates, in dollars per million tokens."""
    for model in requests.get(CATALOG, timeout=30).json():
        if model["model_name"] == model_name:
            pricing = model["pricing"]
            return (
                pricing["cents_per_input_token"] * 10_000,
                pricing["cents_per_output_token"] * 10_000,
            )
    raise LookupError(f"{model_name} is not in the catalog")


def cost_per_1k_requests(model_name, input_tokens=800, output_tokens=300):
    price_in, price_out = rates_per_million(model_name)
    per_request = (input_tokens * price_in + output_tokens * price_out) / 1_000_000
    return per_request * 1_000


# A direct answer against a reasoning model that triples its output.
print(cost_per_1k_requests("meta-llama/Llama-3.3-70B-Instruct-Turbo"))
print(cost_per_1k_requests("deepseek-ai/DeepSeek-R1-0528", output_tokens=900))
copy

Run it and the two lines print 0.176 and 2.335, or $0.18 and $2.34 per thousand requests. Swap your own token split in and the ranking can reorder.

The Costs That Never Appear on the Sticker

The biggest hidden cost is output-token weighting. Output priced four to eight times input means a verbose model is a stealth price increase, and reasoning models that triple the output token count turn a cheap rate card into an expensive bill.

Four more decide the rest of the gap between the sticker and the invoice:

• Cached-input discounts cut repeated system prompts to a fraction of the input rate. Baseten lists Kimi K2.6 cached input at $0.16 against $0.95 standard, SambaNova bills cached tokens 90 percent below standard input on MiniMax-M2.7, and Groq stops counting cached tokens against your rate limits at all.
• Batch pricing cuts deeper. OpenAI and Google discount batch APIs around 50 percent, per AI Superior’s snapshot, Groq’s Batch API and Scaleway’s Batches API both halve the rate, and DeepInfra’s Flex tier bills at 0.8x. All of them want a job that tolerates hours of latency, and Groq’s batch discount does not stack with caching.
• Cold starts add latency spikes to the first request after idle on endpoints that spin down, which is the tax that pays for scale-to-zero billing.
• Minimum commitments on reserved capacity pay only above a duty cycle you measure, not guess.

Our token math and cost-per-completion guide covers the mechanics behind each item.

Which Is the Best AI Inference Platform for Speed and Cost, by Workload

• Realtime chat and voice. Inference latency is the product, and TTFT decides it. Groq or Cerebras, with DeepInfra as the budget option once you confirm TTFT at your prompt lengths.
• Bulk and batch. Throughput per dollar wins. DeepInfra or Together, plus batch APIs and flex tiers where offered.
• Agent loops. Many short calls replaying growing context. Prioritize low TTFT and cached-input pricing. SambaNova’s memory tiering and prompt caching, and DeepInfra’s flex tier, both target this setup.
• RAG. Long context at a low input rate. DeepSeek-V4-Flash serves a 1 million token context at $0.09 and $0.18 per million on DeepInfra, which beats raw speed when every call carries a document. It also rules out the flat-rate platforms, where input costs what output costs.
• EU data residency. Scaleway serves open weights from the Paris region at €0.90 per million in both directions, and rents dedicated GPUs inside the same footprint when serverless will not do.
• Offline eval. Batch discounts and nothing else.
• The serverless vs. dedicated crossover. A dedicated H100 at Baseten’s $6.50 an hour bills $4,680 a month against $1,584 for the same card at DeepInfra’s $2.20, so the crossover moves with the vendor as well as with the traffic. Sustained load whose serverless token spend would clear the hourly bill moves to dedicated. Spiky traffic, heavy for four hours and idle for twenty, stays on serverless and pays nothing for the idle hours. Measure a real week of requests first, because most teams overestimate steady-state load and pay for GPUs they leave half-idle

FAQ: Speed, Cost, and Endpoint Choice

What is the fastest LLM inference API? 

Cerebras leads sustained throughput at 3,000 tokens per second on gpt-oss-120B (vendor-published, April 2026). Groq and DeepInfra lead on first-token latency, both near-zero TTFT in llm-benchmarks.com’s rolling runs. Every one of those figures is a public serverless endpoint, which is the only tier third-party benchmarks can reach.

Why do the cheapest platforms often benchmark slowest? 

Because the benchmark measures a shared serverless endpoint, and a low per-token rate is built out of aggressive batching. Batching raises the number of requests a GPU serves at once, which is what funds the rate, and lowers tokens per second on any single stream, which is what the benchmark reports. The same platform’s dedicated endpoints behave differently, and no independent source publishes comparable numbers for those because each one is configured per customer.

What Is The Cheapest AI Inference API? 

DeepInfra and OpenRouter share the lowest published rate on Llama 3.3 70B at $0.10 in and $0.32 out per million tokens (Infrabase, August 3, 2026).

What Is The Difference Between Throughput And TTFT? 

TTFT is how long until the first token arrives. Throughput is how fast the rest follow.

Does A Cheaper Model Always Mean A Cheaper Bill? 

No. Reasoning models emit multiple times more output tokens, and output costs four to eight times more than input. Price the completed task, not the token.

Can I Use The Openai Sdk With Another Provider? 

Yes. DeepInfra, Together, Scaleway, Novita, OpenRouter, and Groq expose OpenAI-compatible endpoints. Point base_url at the provider and keep your code.

How Is TTFT Measured? 

Send a request with a realistic prompt (not an empty one) and time the gap between the request leaving the client and the first streamed token arriving. Standardized benchmarks like llm-benchmarks.com use fixed prompt lengths and rolling 7-day windows to smooth variance. TTFT depends on prompt length, queue depth, and whether the endpoint needs a cold start, so a single measurement on an empty endpoint tells you very little about production behavior.

Is Serverless Inference Cheaper Than Reserved Capacity?

It depends on your duty cycle. Serverless charges per token and scales to zero when idle, so spiky or low-volume workloads pay only for what they use. Reserved or dedicated capacity charges by the hour regardless of traffic. Once your sustained token spend exceeds the hourly GPU cost, dedicated is cheaper. Below that crossover, serverless wins.

Does FP8 Quantization Affect Model Quality? 

Slightly and model-dependently. FP8 halves the memory footprint compared to bfloat16, which lets providers fit larger models on the same hardware and serve them faster. On most instruction-following and chat tasks the quality difference is negligible. On tasks that depend on fine numerical precision or long-chain reasoning, some models show measurable degradation. Check benchmark scores for the specific FP8 variant you plan to use rather than assuming the answer is universal.

Where That Leaves the Bill

Speed and cost stop conflicting once you measure both per completed task, on the tier you will actually run. The tables above show our work, sources and dates included, and label which endpoint every number came from, so you can redo the math against your own traffic instead of inheriting ours. If a published throughput figure is about to decide a migration, price the dedicated tier before signing anything. This is the number nobody benchmarks, and the one a latency budget ends up living on.

Start with the DeepInfra model catalog and price your workload at these rates. The documentation covers the OpenAI-compatible API in detail. Questions and benchmark disagreements are welcome at feedback@deepinfra.com, in the Discord, or on X @DeepInfra.