DeepInfra raises $107M Series B to scale the inference cloud — read the announcement

Best AI Inference Providers for Enterprise in 2026: Ranked by What They Will Sign - Deep Infra

Published on 2026.11.16 by DeepInfra
Best AI Inference Providers for Enterprise in 2026: Ranked by What They Will Sign - Deep Infra

If I could put one question to an inference vendor before a demo, it wouldn’t be about tokens per second. It would be the one that sits about a third of the way down every security questionnaire: does the vendor keep prompts or completions, for any purpose, for any length of time?

The sales deck says zero retention. I went and read the documentation, and the documentation has more to say.

Google’s Vertex AI caches your inputs and outputs for up to 24 hours by default, until a project admin turns caching off. GroqCloud retains nothing by default, apart from the logs it may hold for up to 30 days for reliability and abuse monitoring, until someone enables Zero Data Retention. On Amazon Bedrock, some current Anthropic models only run in a mode that keeps prompts and completions inside AWS for up to 30 days for human review.

Every one of those sentences comes from the vendor’s own docs. Each default you can’t live with turns into a ticket, an approval form or a contract amendment, and those take weeks.

So here is how I’d choose: pick the provider whose defaults already match the contract you need to sign. Speed and per-token price still matter, and I covered them in our speed and cost comparison. But the enterprise deals I’d worry about don’t stall on latency. They stall on a retention exception somebody has to request, a BAA that excludes the tier you planned to use, or a capacity reservation that turns out to reserve a price and nothing else.

One disclosure before the list. You’re reading this on DeepInfra’s blog, and DeepInfra is ranked first. I’ve tried to earn that by writing down where it loses, and for some of you those gaps will rule it out.

The Short List

ProviderWhere I’d use it
DeepInfraOpen-weight models in production, from serverless API to dedicated GPUs to multi-year clusters, on hardware the vendor owns
AWS BedrockAWS-committed orgs, regulated data, closed frontier models next to open ones
Azure AI FoundryMicrosoft shops, OpenAI models under Azure terms
Google Vertex AIGCP-committed orgs, Gemini plus partner models
BasetenCustom and fine-tuned models, VPC or self-hosted deployment, HIPAA
Nebius Token FactoryEU data residency with a contractual SLA
GroqLatency-critical products
Hugging Face Inference EndpointsTeams already managing private models on the Hub
Novita AICost-led dedicated GPUs for non-regulated workloads

Each of these has a catch, and the catches are what the rest of this piece is about.

The Eight Questions I’d Score Every Vendor On

Enterprise checklists run to sixty rows. I’d cut to eight, because these are the ones that decide who makes the shortlist and how long the paperwork takes. (Our guide to what enterprise buyers should look for in AI inference has the long version, including how to read a SOC 2 report.)

1. What does the vendor keep by default, and who can change it? I no longer bother asking “do you train on my data.” Almost nobody does. I ask what gets logged for abuse monitoring, what gets cached, which features persist state (batch, fine-tuning, stored responses), and whether turning those off is a toggle I control or a form someone else approves.

2. Is the compliance evidence in scope for the tier I’ll use? A SOC 2 Type II report, ISO 27001 and a BAA are the entry fee. What I look for is the exclusions. Groq’s BAA excludes services “provided for free or at no additional charge” and anything in preview. Baseten’s BAA excludes one named training product. I read the exclusions before I look at the badges.

3. Where is the request processed? Stored and processed are different promises. Azure’s Global and Data Zone deployment types may process prompts in any geography where the model is deployed, or anywhere inside the US or EU zone, while the stored data stays put. If your DPA says “processed in the EU,” the deployment type matters more than the region dropdown.

4. How isolated is the deployment? I think of it as a ladder: shared serverless, then a dedicated endpoint on reserved GPUs, then a single-tenant cluster or a deployment inside your own VPC, then renting the GPUs and running the stack yourself. Every rung costs more and answers more of the security questionnaire, so settle which one your data classification needs before you compare prices.

5. What is guaranteed about capacity? Rate limits on a shared tier, a contractual SLA and reserved throughput are three separate things, and vendors blur them. Microsoft, to its credit, doesn’t: its docs say PTU reservations “don’t guarantee capacity”. They get you a discounted rate once a deployment exists. And an uptime percentage on a marketing page is not an SLA until it’s in your contract.

6. What shape is the bill? Per-token, per-GPU-hour, or a committed unit (Bedrock Model Units, Azure PTUs) with a term. A per-token bill follows usage. A GPU-hour bill follows the hours a deployment exists, whether anyone calls it or not. A commitment buys a discount and takes away your exit. I still compare on cost per completed task; the shape tells me how far to trust the forecast.

7. Which models, and for how long? Are the models on your roadmap there? Closed frontier models live mostly on the hyperscalers, and open-weight breadth lives on the specialists. And what happens when a model ID is retired? I wrote a whole piece on building for model deprecation. For procurement, the ask is a notice period in writing.

8. What does leaving cost, and will the vendor still be around? OpenAI-compatible APIs and open weights keep the exit cheap, since moving is a base URL and a model ID. Viability is the other half: funding, whether the vendor owns its hardware or rents it, and any corporate event that changes who runs the company.

I left performance under load off this list on purpose. It varies by model, tier and day, and a benchmark of a shared endpoint says little about the dedicated deployment you’ll end up buying. Measure it on your own traffic during the pilot.

Nine Providers, Same Eight Questions

Retention defaultCompliance evidenceResidencyIsolationCapacity guaranteeBill shape
DeepInfraNot stored (Google/Anthropic models excepted)SOC 2, ISO 270018 US data centersServerless, dedicated GPUs, GPU instances, clusters99.982% SLA on DeepCluster; 200 concurrent requests/model on serverlessPer-token, per-GPU-minute, cluster terms
AWS BedrockPer-Region mode; model default until setHIPAA-eligible; reports in AWS ArtifactAWS RegionsPrivateLink, Provisioned ThroughputModel Units: 0, 1 or 6 monthsPer-token or hourly MUs
Azure AI FoundryHuman-review storage unless approved to modifyAzure compliance offeringsRegional, Data Zone or GlobalProvisioned deploymentsPTUs; reservations don’t guarantee capacityPer-token, PTU/hour, reservations
Google Vertex AICaching on (up to 24h)Not covered hereRegional or globalProject-level controlsNot covered herePer-token
BasetenZDR by contractSOC 2 Type II (NDA), HIPAA with BAA, GDPR DPAMulti-cloud, region-lockingCloud, single-tenant, self-hosted, hybridNot publishedDedicated and per-token
Nebius Token FactoryZero-retention modeSOC 2 Type II, HIPAA, ISO 27001Finland, France, USShared or dedicated99.9% SLA on EnterprisePer-token
GroqUp to 30-day logs until ZDR is onBAA (paid, GA services)USShared cloudNot publishedPer-token
Hugging Face EndpointsPayloads not stored; logs 30 daysSOC 2 Type 2; DPA via Enterprise HubRegion you deploy inDedicated, PrivateLinkNot publishedPer-instance
Novita AINot publishedNo public SOC 2 or HIPAAUS and EUDedicated GPU, IP allowlist99.5% SLA on dedicatedPer-token, per-GPU-hour

That table will get you to a shortlist of two or three. The reviews below cover what the cells can’t hold.

What Independent Price Data Shows

Contract terms get you to a shortlist. Price breaks most of the ties, so I pulled per-token prices from Artificial Analysis, which tracks the same open-weight model across hosts. It covers eight of the nine providers here. Hugging Face Inference Endpoints isn’t tracked as a provider.

I picked two models. gpt-oss-120b is the one most of this list serves. GLM-5.2 is the model in my cost example further down, and none of the three hyperscalers appears in its comparison.

gpt-oss-120b, from Artificial Analysis’s provider comparison:

ProviderInput ($/1M tokens)Output ($/1M tokens)Output vs DeepInfra
DeepInfra$0.037$0.17—
Novita$0.05$0.251.5×
Google Vertex$0.09$0.362.1×
Baseten$0.10$0.502.9×
Groq$0.15$0.603.5×
Azure$0.15$0.603.5×
Nebius$0.15$0.603.5×
Amazon Bedrock$0.15$0.603.5×

GLM-5.2, from its provider comparison:

ProviderInput ($/1M tokens)Output ($/1M tokens)Output vs DeepInfra
DeepInfra$0.562$1.80—
Nebius$1.40$4.402.4×
Novita$1.40$4.402.4×
Baseten$1.40$4.402.4×
Baseten (FAST)$2.10$6.603.7×

Prices are list prices per 1M tokens as shown by Artificial Analysis on 2026-11-16. DeepInfra’s GLM-5.2 price includes a 25% promotion; its list price is $0.75 input and $2.40 output.

Of the providers on this list, DeepInfra has the lowest input and output price for both models. On gpt-oss-120b, Groq, Nebius, Azure and Amazon Bedrock all charge about four times as much for input and 3.5 times as much for output. On GLM-5.2 the gap is 2.4 times with the promotion, and at DeepInfra’s list price the output rate is still about 45% lower than the next provider’s.

Faster endpoints exist. Groq’s endpoint generates output about ten times faster on gpt-oss-120b in the same Artificial Analysis data, so if response speed is your product, pay for it. For batch work, agents and back-office workloads, the per-token gap compounds.

The Providers, One by One

1. DeepInfra: The Open-Weight Specialist That Owns Its Stack

I’d point two kinds of teams here. The first has already decided that open-weight models (DeepSeek, Qwen, GLM, Kimi, Llama, Nemotron) are good enough for most of its traffic. The second is a platform team that wants one vendor from the first API call to a dedicated cluster and watches the bill as closely as the security review.

DeepInfra owns and operates its GPU infrastructure across eight U.S. data centers and serves 150+ open models behind OpenAI-compatible APIs. Its data privacy docs say inputs are not written to disk during inference, outputs are not stored, and customer data is neither used for training nor shared with third parties. The pricing page carries SOC 2 and ISO 27001 certification, and the audit detail sits in the trust center at trust.deepinfra.com.

What puts it first for me is that the whole isolation ladder sits in one account:

RungWhat you getPrice
Serverless, FlexAsync and non-production work; slower responses, occasional unavailability0.8x base
Serverless, StandardDefault schedulingBase per-token price
Serverless, PriorityScheduled ahead of standard traffic, set per request with service_tier1.5x base
Private deploymentsAny Hugging Face LLM or LoRA on dedicated GPUs, autoscaling to zero$1.49 (A100) to $5.89 (B300) per GPU-hour, billed per minute
GPU instancesSSH into B200 containers, no egress fees$4.89 per GPU-hour
DeepCluster256 to 5,000 B300 GPUs, 99.982% uptime SLA$2.99 (3-year) or $1.98 (5-year) per GPU-hour

On viability, DeepInfra raised a $107M Series B in May 2026, co-led by 500 Global and Georges Harik, with NVIDIA, Felicis, Samsung Next and Supermicro participating.

Now the fine print. The zero-retention statement has an exception written into it: Google and Anthropic models are handled differently. If you plan to route closed models through the same account, read that section before you tell your security team “zero retention.” Serverless also defaults to 200 concurrent requests per public model. You can ask for more. Ask before launch week. On price, the Artificial Analysis tables above show DeepInfra with the lowest per-token rates on this list for both gpt-oss-120b and GLM-5.2.

Where DeepInfra loses. Every data center is in the US, so if your contract says EU, go to Nebius or a hyperscaler. DeepInfra’s privacy policy describes technical and organizational measures for GDPR and HIPAA compliance. DeepInfra doesn’t publish a BAA, so a PHI workload starts with a sales conversation. Outside DeepCluster, the pricing page publishes no uptime SLA for serverless or dedicated endpoints. And Baseten will run inside your own VPC, which DeepInfra doesn’t advertise.

What I’d ask. Is a BAA available, and for which tiers? What’s the SLA for a dedicated endpoint, in writing? How much notice before a model is retired? Can the concurrency limit go into the contract?

My verdict: the best fit on this list for open-weight production work when US hosting is fine. If you need EU processing or a BAA on day one, start further down the list.

2. AWS Bedrock: The Default When the Data Already Lives in AWS

If you have committed AWS spend, a signed AWS BAA and a security team that thinks in IAM policies, I’d be surprised if Bedrock weren’t already on your shortlist. It’s also the practical answer when you want closed frontier models and open-weight models under one contract.

The compliance story is the strongest here. Bedrock is on AWS’s HIPAA Eligible Services list, and the audit reports are a download away in AWS Artifact. Model providers have no access to the accounts that serve their models, so they never see your prompts or Bedrock’s logs. Private connectivity runs through VPC endpoints and PrivateLink.

Retention is where I’d slow down. Bedrock now treats it as a mode you set per Region, at account or project level:

ModeWhat happens to your prompts
noneZero retention. Nothing written to durable storage.
defaultThe model’s own policy applies. AWS may retain data for abuse prevention.
aws_reviewAWS retains inputs and outputs for human review, inside the AWS boundary.

New accounts start with no setting and fall back to each model’s default. I like the design: one Service Control Policy can force none across an entire organization. The cost is that somebody has to own the setting in every Region you use, because it doesn’t propagate.

Here’s the part that will surprise people. Some current Anthropic models require aws_review, which means prompts and completions are kept for up to 30 days. Set none, and those models show as unavailable. Zero-retention exceptions are negotiated per account and per model. Turn on cross-region inference, and retained data is stored in the destination Region.

The bill has its own catch. If you fine-tune, you must buy Provisioned Throughput to serve the result, billed hourly per Model Unit, with no-commitment, one-month or six-month terms. Your fine-tuned model is an hourly line item from the day it ships.

What I’d ask. Which of the models on my list require aws_review? What throughput does one Model Unit deliver for each? (AWS shares that through your account manager; it isn’t in the docs.) Does my fine-tuned model’s traffic justify a six-month commitment?

My verdict: the right default for AWS-native, regulated organizations, provided someone is named as the owner of that retention setting.

3. Azure AI Foundry: OpenAI Models on Microsoft’s Paper

This is where I’d send a Microsoft-committed enterprise that wants OpenAI models under Azure’s data terms, with Entra identity and Azure Cost Management already wired up.

Microsoft’s commitments are clear. Prompts, completions and fine-tuning data for Models sold by Azure are not available to OpenAI or other model providers and aren’t used to train foundation models without your permission. Regional deployments process in your chosen geography, Data Zone deployments anywhere within the US or EU zone, and Global deployments wherever the model runs. Dedicated capacity comes as provisioned throughput units, billed per PTU per hour or discounted with one-month or one-year reservations.

Two things in those same docs would make me pause, and both are in Microsoft’s own words.

The first is abuse monitoring. When the system flags a pattern, a sample of your prompts and completions is stored in a per-geography data store for human review. Only “managed customers” can apply to modify that, and automated review continues even after approval.

The second is capacity. “Having PTU quota doesn’t guarantee that capacity is available,” and reservations “don’t guarantee capacity.” Scale a deployment down, and the capacity may not be there when you scale back up. There is a spillover feature that sends overflow to a standard deployment, but it doesn’t cover the non-OpenAI models such as Azure DeepSeek or Meta Llama.

What I’d ask. Do we qualify as a managed customer for modified abuse monitoring? Is capacity available for my model and region today, before I buy a reservation? Do the open-weight models I need support provisioned deployments at all?

My verdict: strong for OpenAI-model workloads on Microsoft paper. I wouldn’t pick it as an open-weight platform.

4. Google Vertex AI: Good Guarantees, Homework Attached

For GCP-committed organizations building on Gemini, with partner models through Model Garden. (Google’s docs now file this under the Gemini Enterprise Agent Platform name.)

The headline commitment is solid: Google will not use your data to train or fine-tune models without permission, for GA and pre-GA models alike. Prompt logging for abuse monitoring only applies to customers on the standard Google Cloud Platform Terms of Service.

Getting to zero retention is another matter. I counted five separate things on that one page that keep data unless you act:

FeatureWhat it keepsHow you stop it
Implicit cachingInputs and outputs, up to 24 hours; a durable tier with no customer-managed keysDisable per project (admin role)
Grounding with Google SearchDerived queries, up to 3 daysYou can’t; use Web Grounding for Enterprise
Interactions APIConversation stateSet store=false on every call
OpenAI models on VertexPrompts and responses, up to 30 days, US multi-regionApproval that runs through OpenAI
Global endpointCached data may be served from any regionUse a regional endpoint

None of this is unreasonable, but missing any single row breaks a “zero retention” answer on your questionnaire.

What I’d ask. A written list of every feature in my design that persists data, mapped against that page. Whether my contract puts us outside abuse-monitoring logging.

My verdict: a solid platform if your security team will own that table. Give them the time to do it.

5. Baseten: Your Model, Your VPC, a Signed BAA

If the model is your own fine-tune and the data is regulated, this is where I’d look first. Baseten’s customer list includes OpenEvidence, which serves medical information to clinicians.

Baseten’s security practices page is the most specific document I read for this piece, and it’s incorporated into the terms, so it binds them. SOC 2 Type II, with the report available under NDA. HIPAA-eligible configurations, with BAAs for qualifying customers. A GDPR DPA. Zero data retention defined in contract language for Model APIs and Dedicated Inference: no logging, recording or persistent storage of inputs or outputs. Breach notification within 48 hours, and fifteen days’ notice before a new subprocessor.

Its standard gpt-oss-120b endpoint also has the shortest time to first token of the providers on this list in Artificial Analysis’s gpt-oss-120b comparison, 0.32 seconds. The deployment options are what set it apart:

OptionWhere it runs
Baseten CloudBaseten’s multi-cloud infrastructure, with region-locking for residency
Single-tenantA dedicated cluster for you
Self-HostedYour own cloud; “no Customer Content leaves the customer’s environment”
HybridSelf-hosted, with spillover to Baseten Cloud

I have three cautions. If you opt in to asynchronous inference, model inputs can sit in a queue for up to 72 hours. The enterprise page advertises 99.99% uptime, and I’d treat that as marketing until the same number appears in your order form. And self-hosting moves work onto your side: you skip part of the security review and gain a cluster to keep running.

What I’d ask. The SLA for the deployment option I’m buying. Which cloud and region a dedicated deployment lands in. The BAA’s exclusions.

My verdict: my pick when the model is yours and the data is regulated.

6. Nebius Token Factory: EU Residency With an SLA Attached

For European enterprises, or anyone whose DPA requires EU processing, running open-weight models. This is the gap in DeepInfra’s offer, and Nebius fills it.

Token Factory’s enterprise page lists SOC 2 Type II, HIPAA and ISO 27001, a zero-retention mode, and data centers in Finland, France and the US. Dedicated endpoints carry a 99.9% SLA with reserved capacity and failover. One detail I like: custom fine-tuned models are priced per token, the same as standard endpoints, so serving your own weights doesn’t mean paying for idle GPUs. One number I’d test in a pilot: its GLM-5.2 endpoint shows a 9.65-second time to first token in Artificial Analysis’s GLM-5.2 comparison.

Read the plan table before you get excited, though.

StarterEnterprise
AvailabilityBest-effort 99%99.9% SLA, reserved capacity, failover
Limits400K TPM / 600 RPM baselineUncapped, autoscaling
Data termsZero-retention inferenceZero retention plus custom DPA
SupportDocs and 24-hour emailDedicated Slack, solution architect

Everything a procurement team wants is in the right-hand column. The page also says audit timelines are available under NDA, so ask for the report dates.

What I’d ask. Which certifications cover which regions. Whether zero retention is the default or something I configure. The SLA credit schedule.

My verdict: the first call I’d make for EU residency on open-weight models.

7. Groq: Buy It for Latency, Then Do the Diligence

When time-to-first-token is the product (voice agents, real-time copilots, anything a user watches load), Groq belongs on the list. On gpt-oss-120b it is the fastest provider on this list in Artificial Analysis’s comparison, at 393 tokens per second.

On data, GroqCloud does not retain inference data by default. It may log inputs and outputs for up to 30 days to troubleshoot reliability problems or investigate abuse, and batch and fine-tuning retain data while they run. Any customer can switch on Zero Data Retention in Data Controls, which also disables those features. Customer data that is retained sits in GCP buckets in the United States. Groq also publishes a Business Associate Addendum, effective October 15, 2025, that you accept online.

I’d flag three things. The BAA covers generally available, paid services only, so beta, preview, demo, trial and free services are out. ZDR is opt-in, which means an org admin has to turn it on before the first production request, and I’d want a screenshot in the audit folder. The third is the one your vendor-risk team will raise without my help. Under a non-exclusive licensing agreement with NVIDIA, Groq’s founder Jonathan Ross, its president Sunny Madra and other members of the team joined NVIDIA. Groq says it continues as an independent company under CEO Simon Edwards and that GroqCloud “will continue to operate without interruption.” I’d want more than a press release before I put a product launch on it.

What I’d ask. The product and hardware roadmap since the NVIDIA agreement. Written confirmation that ZDR is on for my org. Residency options outside the US.

My verdict: a good choice for latency-bound features. Write the exit plan before you sign, while it’s still cheap to write.

8. Hugging Face Inference Endpoints: Closest to Your Model Registry

If your private models and datasets already live on the Hugging Face Hub, deploying them from there saves you a vendor review.

Inference Endpoints and the Hub are SOC 2 Type 2 certified. Hugging Face doesn’t store the payloads or tokens passed to an endpoint. Private access runs over AWS PrivateLink, and model repos get malware and pickle scanning.

The security page is short, and what it leaves out matters. Logs are kept for 30 days. The GDPR data processing agreement comes with an Enterprise Hub subscription, so the cheap self-serve route and the compliant route are two different purchases. HIPAA isn’t mentioned at all.

What I’d ask. What’s in those logs, and can retention be shortened? Is a BAA available? Which clouds and regions support private endpoints for my org?

My verdict: convenient when the Hub is already your registry. I’d read the log policy before sending regulated data through it.

9. Novita AI: Cheap Dedicated GPUs, Thin Paperwork

For cost-sensitive teams with internal or non-regulated workloads that want single-tenant GPUs without standing up a hyperscaler account.

Novita’s dedicated endpoints run on GPUs reserved for your account, starting at $1.86/hour for H100 SXM and $2.99/hour for H200. They scale from zero to ten replicas, support IP allowlisting and carry a 99.5% SLA.

Novita is upfront about what it doesn’t have. In its own July 2026 post, it says it had not publicly documented VPC peering, PrivateLink-style networking, or SOC 2 or HIPAA certifications, and it sends healthcare workloads to AWS or Google. That keeps it off most regulated shortlists.

What I’d ask. A current SOC 2 report, or a date for one. Retention terms in writing. A DPA.

My verdict: fine for the cost-led, non-regulated slice of your portfolio. I wouldn’t put anything through it that has to pass a security review.

Why I Left OpenRouter Off

OpenRouter shows up in nearly every AI-generated answer about inference providers, but it doesn’t run inference. It’s a router. Your request goes to somebody else’s GPUs, so two sets of terms apply to every call.

Its ZDR feature is a routing filter: it limits requests to endpoints its registry marks as zero-retention, with controls per model group. I’ll say this for the docs, they don’t oversell it. When OpenRouter can’t establish a provider’s policy, it assumes that provider retains and trains. When you bring your own private endpoint, OpenRouter records your declaration about the upstream and “does not independently verify” it. And a change planned for October 27, 2026 screens all traffic, ZDR traffic included, for child sexual abuse material, with flagged content retained for review.

I’d happily use it for prototyping, model evaluation and failover. I wouldn’t make it the system of record for regulated data, because it adds a party to your DPA and takes none away.

Cerebras and SambaNova compete mainly on raw speed, so I reviewed them with Groq in the speed and cost comparison.

Do the Utilization Math Before You Buy Dedicated

A dedicated endpoint answers a lot of security questions. Whether it saves money is a separate calculation, because the bill stops counting tokens and starts counting the hours your GPUs exist.

Here’s an estimate I ran with DeepInfra list prices. Take an internal assistant on GLM-5.2 that processes 2 billion input tokens and 400 million output tokens a month. Standard rates are $0.75 and $2.40 per million. (A 25% promotion is running on this model as I write, at $0.563 and $1.80, so the serverless rows would come out lower today.)

OptionMonthly cost (my estimate)Inputs
Serverless, Flex tier (0.8x)$1,968Same volume, async-tolerant work only
Serverless, Standard$2,4602,000M × $0.75 + 400M × $2.40
Serverless, Priority (1.5x)$3,690Same volume
Dedicated, 4× H200, always on$10,1914 × $3.49 × 730 hours
Dedicated, 8× B300, always on$34,3988 × $5.89 × 730 hours

Pricing as of 2026-11-16.

On those numbers, the token bill has to grow about four times before an always-on 4× H200 deployment costs the same. Treat that as a sanity check and size the real thing in a pilot. You buy dedicated for isolation, custom weights and steadier latency. It only wins on cost once the GPUs stay busy. Autoscaling to zero helps a great deal, and a deployment left running over a long weekend undoes most of it.

At the top of the ladder, a 256-GPU DeepCluster on a three-year term comes to roughly $559,000 a month (256 × $2.99 × 730). That goes in the same planning meeting as your training roadmap.

If the plan is to get there by moving workloads off a closed model, Niklas’s AI spend audit covers which ones to move first.

Keep the Exit Cheap

DeepInfra’s API is OpenAI-compatible, and so are most of the open-weight specialists’. That makes the provider a config value, which is easy to change when a vendor’s terms do.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.deepinfra.com/v1/openai",
    api_key=os.environ["DEEPINFRA_API_KEY"],
)

response = client.chat.completions.create(
    model="zai-org/GLM-5.2",
    messages=[{"role": "user", "content": "Summarize the data retention clause in two sentences."}],
    service_tier="priority",  # 1.5x base price, scheduled ahead of standard traffic
)
print(response.choices[0].message.content)
copy

Put the base URL, model ID and tier in config, and keep them out of your call sites. Do that with open weights whose license you’ve checked, and changing providers takes a deploy.

Ten Questions I’d Send Before the First Call

These are the questions that separated the nine providers above. I’d email them ahead of the demo, since the answers take longer to get than the demo takes to sit through. For the full 25-question version, organized by where your data goes, see the enterprise AI security checklist.

  1. Retention: What do you retain by default, for how long, and why? Which features change that? Is turning it off a setting or an approval?
  2. Evidence: Can you send the current SOC 2 Type II report, with its audit period and the tiers it covers?
  3. BAA scope: Is a BAA available, and what does it exclude?
  4. Processing location: Where is a request processed, as opposed to stored? Can any deployment type route it out of region?
  5. Isolation: Is the GPU reserved for my account? Is there private networking, or an IP allowlist only?
  6. Capacity: What SLA goes in the contract for my tier? Does a reservation guarantee capacity, or only a price?
  7. Limits: What are the default rate and concurrency limits, and how are increases granted?
  8. Model lifecycle: How much notice before a model ID is retired, in writing?
  9. Subprocessors: Who are they, and how much notice before you add one?
  10. Continuity: Who owns the hardware, and what happens to my deployment if the company is acquired?

Which One I’d Pick, by Situation

You’re in a regulated industry and need a BAA now. Baseten for custom models, AWS Bedrock if you’re already on AWS, Nebius if the data has to stay in the EU. If your requirement is SOC 2, ISO 27001 and zero retention rather than a BAA, DeepInfra covers that by default on open-weight models, and a BAA is a question for its sales team.

Your spend is committed to a hyperscaler. Use that hyperscaler for closed models, and move open-weight volume to a specialist like DeepInfra, where the per-token math is better. Spending down a commitment on the wrong workload is still overpaying.

Your product is latency-critical. Groq, with ZDR on and a continuity answer in writing.

You run high-volume batch or offline jobs. DeepInfra’s Flex tier at 0.8x base price was built for this, as long as the work can wait and can retry.

You serve your own fine-tuned models. Baseten if it has to run inside your VPC. DeepInfra dedicated deployments if you want per-GPU-minute pricing on hardware the vendor owns. Nebius if you’d rather keep paying per token for custom weights.

Data must stay in the EU. Nebius Token Factory first, Azure Data Zone deployments second. DeepInfra can’t do this today. I went through the business case for residency in data sovereignty as a business decision.

You’re moving workloads off a closed model to cut cost. DeepInfra. Open-weight models the hyperscalers also resell are served per token, with Flex for the work that can wait, and the API is OpenAI-compatible, so the switch is a base URL and a model ID. Niklas’s AI spend audit covers which workloads to move first.

You want open-weight breadth from API to cluster with one vendor. DeepInfra. Serverless tokens, dedicated endpoints, SSH GPU instances and multi-year B300 clusters all come from one account and its own data centers.

FAQ

What makes an inference provider “enterprise-ready”? I’d want four documents: a current SOC 2 Type II report that covers the tier you’re buying, written data retention terms, an SLA in the contract, and isolation options that match your data classification. Add a BAA if you handle PHI.

Is zero data retention the default anywhere? It’s close on some. DeepInfra says inputs and outputs aren’t stored, with exceptions for Google and Anthropic models. Baseten writes ZDR into its terms for Model APIs and Dedicated Inference. On the hyperscalers and Groq, it’s something you configure or apply for.

Are hyperscalers safer than specialist inference providers? They have deeper compliance programs and more mature identity and networking controls. They also have more retention defaults to configure, and their capacity reservations come with terms the specialists don’t impose. I’d call them safer only if someone on your team owns that configuration.

Where This Leaves You

Send the ten questions, cross off every provider whose defaults don’t fit your contract, and only then benchmark speed and price on your own traffic. If open-weight models are the plan, start with DeepInfra’s model catalog and the pricing page, read the docs on private deployments, or talk to sales about dedicated capacity and the gaps I listed above. And if I’ve misread a provider’s terms, I’d like to hear it: email, Discord or X all work.

Related articles
The Enterprise AI Security Checklist: What to Verify Before Your Data Reaches an Inference Provider - Deep InfraThe Enterprise AI Security Checklist: What to Verify Before Your Data Reaches an Inference Provider - Deep Infra<p>Most enterprise AI security reviews do not fail on the headline items. Nobody is surprised that a vendor needs encryption or a SOC 2 report. They fail on questions nobody thought to ask until the contract was nearly signed: who owns the servers, which companies sit between you and the GPU, and where a prompt [&hellip;]</p>
How Open Source AI Is Closing the GapHow Open Source AI Is Closing the Gap<p>At the end of 2023, the gap between open-weight and closed-source AI models was real and easy to describe. If you wanted the best performance on reasoning, language understanding, or multi-step problem solving, you paid for a proprietary API. Open models were useful, capable for many tasks, and dramatically cheaper to run but they were [&hellip;]</p>
Best AI Inference Platforms for Speed & Cost in 2026Best AI Inference Platforms for Speed & Cost in 2026<p>TL;DR: Best Inference Models and Platforms by Workload The best inference models for most production work are open-weight, and once you have picked one, the platform serving it moves your bill more than the model does. Llama 3.3 70B costs anywhere from $0.18 to $1.14 per 1,000 identical requests across seven providers. Here is the [&hellip;]</p>