DeepInfra raises $107M Series B to scale the inference cloud — read the announcement

What Enterprise Buyers Should Look for in AI Inference - Deep Infra

Published on 2026.11.16 by DeepInfra
What Enterprise Buyers Should Look for in AI Inference - Deep Infra

Pick any popular open-weight model and you will find it served by many different providers. The weights are identical. The benchmark scores are identical. What differs is everything your security team asks about once a proof of concept turns into a procurement request: where the prompts go, who can read them, how long they live, and what happens when traffic doubles on a Tuesday afternoon.

That second set of questions is about inference, not about the model. When enterprise buyers say they are choosing an AI provider, they often mean the company that built the model. Inference is the layer underneath. It is the infrastructure that receives your data, runs the model and returns the answer, and it is the part that your auditors, your legal team and your on-call engineers will deal with every day.

Four things decide whether an inference service can clear an enterprise review: its SOC 2 posture, its reliability, its data retention terms and the private endpoint options it offers. This article takes each in turn, explains what good looks like, and ends with a checklist of the evidence to ask for.

Model, inference and contract are three separate decisions

A strong model on a weak inference service is a problem. So is the reverse, although it is easier to fix, because a good inference service can run many models and you can change the model later. Treating the two as one decision is how teams end up with a leaderboard winner that their security team will not approve.

DecisionWhat it determinesWhere the evidence lives
The modelCapability, cost per token, calibration, license termsBenchmarks and your own evaluations
The inference serviceWhere your data goes, who can see it, uptime, latency under load, isolationAudit reports, contract language, load tests
The commercial termsPrice structure, support, liability, exit optionsOrder form, data processing agreement, SLA

Keeping the scorecards separate also protects you from churn. Models turn over every few months. The requirements on the inference layer, such as audit evidence, retention terms and isolation, change far more slowly, so the work you put into evaluating them keeps paying off.

SOC 2: read the report, not the badge

SOC 2 is an attestation by an independent auditor that a service provider operates controls over security and, optionally, availability, confidentiality, processing integrity and privacy. A badge on a website tells you that a report exists. The report tells you what was actually tested, and that is the document your reviewers care about.

Type II and the audit period. A Type I report describes controls at a single point in time. A Type II report covers a window of operation, usually six to twelve months, and tests whether the controls worked throughout. Enterprise procurement policies almost always ask for Type II. Check the dates, because a report that ended eighteen months ago says little about today.

Scope. This is the gap that catches the most buyers. A report can cover a company’s corporate IT and internal processes without covering the production platform that handles your requests. Read the system description and confirm that the inference service itself is inside the boundary.

Trust criteria. Security is always included. For inference, confidentiality and availability matter most, so confirm that both are in scope rather than assuming they are.

Exceptions and user controls. The auditor’s opinion lists any exceptions found during testing, and a section on complementary user entity controls lists what you are expected to do on your side. Both are more informative than the headline result.

ISO 27001 is a useful companion. It certifies an information security management system rather than a point-in-time control test, and many enterprise questionnaires ask for both.

DeepInfra is SOC 2 and ISO 27001 certified, and its documentation covers the practical items reviewers ask about next, including Okta SSO, data privacy and a subprocessor list. Subprocessors deserve a read of their own, because your data flows through every one of them.

Reliability: Look at the tail, not the average

Uptime percentages can create a false sense of precision. While they look reassuring on a provider’s website, they reveal relatively little about what users will actually experience. A 99.5 percent uptime commitment translates to roughly 44 hours of downtime per year. At 99.9 percent, that falls to around nine hours, while 99.982 percent allows for just over an hour and a half. The differences may appear marginal on paper, but their operational impact is anything but.

The first thing to establish is exactly what an uptime commitment covers. An SLA for one product or infrastructure tier does not automatically extend to another. DeepInfra, for example, publishes a 99.982 percent uptime SLA for its dedicated B300 GPU clusters. When assessing a provider, make sure the commitment applies to the specific service you intend to use. It is equally important to understand what compensation or remedy applies if the SLA is breached and whether the provider publishes a status page with a meaningful incident history.

Bar chart of allowed downtime per year at each availability level, from 87.6 hours at 99.0% to 1.6 hours at 99.982%, the SLA DeepInfra publishes for dedicated B300 GPU clusters

Uptime, however, is only part of the picture. Latency is where averages become particularly misleading. Queueing can cause response times to fluctuate significantly under load, so average latency is a poor indicator of what users will experience during busy periods. Ask providers for P95 and P99 measurements for both time to first token and inter-token latency at your expected concurrency. Consider a service with an average latency of 400 milliseconds but a P99 of six seconds. Most requests might appear fast, yet one in a hundred takes long enough to disrupt the experience. In an agent workflow involving 50 calls per task, these delays can accumulate quickly.

It is also worth looking beyond advertised performance to understand how much traffic the service can actually handle. Check the default concurrency and rate limits, and find out how quickly they can be increased. DeepInfra documents its rate limits and applies a default concurrency limit per model, with higher limits available on request. The distinction matters: a provider that can raise your limits within a day offers a very different level of flexibility from one that requires a lengthy sales process.

Scaling introduces another potential bottleneck. Even when using dedicated infrastructure, additional GPU capacity may not be immediately available. DeepInfra explicitly notes this in its documentation. For workloads where latency matters, maintaining a minimum replica count is a safer approach. Rather than waiting for additional capacity to become available when traffic spikes, you already have the infrastructure running to handle the demand.

Finally, consider what happens when a provider becomes unreliable or no longer meets your requirements. Most major inference APIs are OpenAI compatible, which makes switching between providers relatively straightforward. Keep the provider’s base URL in configuration rather than hardcoding it into your application, and, more importantly, test your fallback before you actually need it. Portability only matters if the alternative works in practice.

Data retention: Three questions and one easily overlooked risk

Data retention is often where security reviews become particularly important. The underlying questions are straightforward: Does the provider store the content of requests and responses? If so, for how long, and who can access it, including employees and subprocessors? Finally, is any of that data used for model training or evaluation?

A zero data retention policy is intended to address all three concerns. DeepInfra, for instance, states that it does not retain inputs, outputs or user data after returning a response. This does not necessarily mean that absolutely no information is recorded. Providers may still collect operational metadata, including request identifiers, token counts and timing information, for purposes such as billing and debugging. That is normal. The important distinction is that this metadata should not contain the actual content of prompts or responses.

There is, however, a difference between a published policy and a contractual commitment. A statement on a provider’s website describes its intended practices; a clause in a data processing agreement gives you something more concrete to rely on. If you handle regulated or otherwise sensitive information, make sure the retention terms are explicitly documented in your agreement. Request the current subprocessor list as well, so you understand which other parties may be involved in processing your data.

Another detail worth investigating is prompt caching. Caching can reduce inference costs and improve response times, but it also means that some prompt state may be retained temporarily. Ask how cached data is isolated between accounts, how long it remains available and whether it is covered by the provider’s standard retention commitments. A zero-retention policy is only meaningful if its scope is clearly defined.

Geography matters, too. The location of the infrastructure determines which legal frameworks may apply to your data. DeepInfra operates hardware that it owns and manages in data centres in the United States. As we discussed in our article on Chinese open-weight models, the origin of a model’s weights does not determine where your data is processed or stored. That depends on the inference provider and the infrastructure behind its service.

Diagram showing a prompt going from your application to the provider's API gateway and GPU, where retention terms apply, and also into your own application logs, tracing platform and error reporter, which the provider does not cover

Perhaps the most easily overlooked risk, however, sits outside the provider’s infrastructure altogether: your own logging. Zero data retention on the inference side offers little protection if your application stores the same information elsewhere. Many frameworks log complete request and response objects by default, potentially sending prompts and sensitive user data into application logs, tracing platforms and error-reporting services.

This is why a provider’s retention policy should never be the end of your review. Audit your own observability stack just as carefully as you assess the vendor. Otherwise, you may have a perfectly sound data retention agreement with your inference provider while continuing to expose the very same information through your application.

Private endpoints: three meanings, one question

Vendors use the phrase private endpoint for at least three different things, and the differences matter for both security and cost. Ask which one is on offer before comparing anything else.

What the vendor means by “private”What you actually getWhat to ask
Dedicated computeGPUs reserved for your traffic and not shared with other tenants. Predictable latency and no shared router limits.Is it genuinely single tenant? What is the minimum number of replicas? How is it billed?
A private network pathTraffic between your systems and the service never crosses the public internet.Which network options exist? In which regions? Is there an extra charge?
A private modelYour own fine-tuned or proprietary weights, served for you alone.Where are the weights stored? Who can access them? Can you export them?

DeepInfra offers private deployments on dedicated A100, H100, H200, B200 and B300 GPUs, with autoscaling and an OpenAI-compatible endpoint. The deployment is isolated to your account, you can serve your own fine-tuned or custom weights, and requests are addressed to a model identifier that belongs to you. That covers dedicated compute and private models. Network-level isolation is a separate requirement, so confirm it explicitly with any vendor, including us.

When it pays off. Shared endpoints bill per token and are the cheaper option for low or spiky volume. A private deployment bills per GPU hour whether or not traffic arrives. The break-even point is simple arithmetic: divide the hourly cost of the GPUs by the per-token price you would otherwise pay. As an illustration, if a deployment costs 4 dollars per hour and you would otherwise pay 1 dollar per million tokens, the deployment wins once you sustain more than 4 million tokens an hour, provided the hardware can actually serve that rate. Many buyers also choose private deployments below that point, because the isolation is the requirement and the cost is secondary.

Operational notes. DeepInfra’s documentation lists a default limit of four GPUs per user, with more available on request, and billing for private deployments is per GPU hour on a weekly invoice. As noted above, GPU availability during scale-up is not guaranteed, which is another reason to hold a minimum replica count for production traffic.

The wider spectrum. Private endpoints are one step on a ladder. At the first rung you use a shared API with zero retention. Next comes a dedicated endpoint on isolated GPUs. Beyond that you run the weights inside your own cloud account, and at the top you run on premises or fully air-gapped. Each step gives you more control and costs more in money and operational effort. Most enterprise workloads need only the first or second rung, and the right answer depends on the sensitivity of the data rather than on the size of the company.

The evidence checklist

Use the table below to turn the four areas into requests you can send to a vendor on day one. For a deeper, 25-question version that follows a prompt hop by hop, see our enterprise AI security checklist. Every row names the evidence to ask for and the red flag that should slow the process down.

RequirementEvidence to requestRed flag
SOC 2Current Type II report under NDA, with the system description and any noted exceptionsA badge on the website only, a report older than twelve months, or a scope that excludes the inference platform
ISO 27001Certificate with scope statement and expiry dateA certificate that covers a parent company but not the service you will use
ReliabilityWritten SLA with scope and remedy, status page history, P95 latency data, rate limit documentationA single uptime percentage with no scope, no remedy and no history
Data retentionContract language on non-retention, subprocessor list, prompt cache policyA policy statement on a web page and nothing in the data processing agreement
Private endpointsA written definition of the isolation provided, minimum commitment, scale-up behaviorThe word “dedicated” used for capacity that is shared
Access controlSSO support, scoped API keys, audit logsOne shared API key for the whole organization

Putting it together

The sequence that saves the most time is also the cheapest. Screen on paper first: SOC 2 report, retention language and subprocessor list can be reviewed in a day, and they eliminate most candidates. Then test what paper cannot show: run your own traffic for a week and measure the tail. Then decide on private endpoints based on volume and on how sensitive your data is, not on what sounds most secure.

If you want to see how specific providers measure up against these criteria, our roundup of the best AI inference providers for enterprise scores nine of them side by side.

Finally, keep your options open. Hold the base URL in configuration, keep one alternative provider tested and avoid building application logic that depends on a single vendor’s quirks. The model you choose will change more than once this year. The requirements on the inference layer will not.

Related articles
GLM-4.6 vs DeepSeek-V3.2: Performance, Benchmarks & DeepInfra ResultsGLM-4.6 vs DeepSeek-V3.2: Performance, Benchmarks & DeepInfra Results<p>The open-source LLM ecosystem has evolved rapidly, and two models stand out as leaders in capability, efficiency, and practical usability: GLM-4.6, Zhipu AI’s high-capacity reasoning model with a 200k-token context window, and DeepSeek-V3.2, a sparsely activated Mixture-of-Experts architecture engineered for exceptional performance per dollar. Both models are powerful. Both are versatile. Both are widely adopted [&hellip;]</p>
From Recordings to Records: NVIDIA Nemotron 3.5 Transcribe on DeepInfraFrom Recordings to Records: NVIDIA Nemotron 3.5 Transcribe on DeepInfraNVIDIA Nemotron 3.5 Transcribe is live on DeepInfra from day zero: multilingual transcription with word timestamps and speech translation into English, in one OpenAI-compatible API.
AI Model Distillation: Teacher vs. Student ModelsAI Model Distillation: Teacher vs. Student Models<p>Your ticket classifier runs on a 1.6 trillion parameter model. It reads a support message, decides whether it is a billing question or a bug report, and returns one word. You&#8217;re paying frontier prices for a job a model a hundredth the size could finish in half the time. AI model distillation is the standard [&hellip;]</p>