We use essential cookies to make our site work. With your consent, we may also use non-essential cookies to improve user experience and analyze website traffic…
DeepInfra raises $107M Series B to scale the inference cloud — read the announcement
Starting from langchain v0.0.322 you can make efficient async generation and streaming tokens with deepinfra.
Async generation
The deepinfra wrapper now supports native async calls, so you can expect more performance (no more threads per invocation) from your async pipelines.
from langchain.llms.deepinfra import DeepInfra
async def async_predict():
llm = DeepInfra(model_id="meta-llama/Llama-2-7b-chat-hf")
output = await llm.apredict("What is 2 + 2?")
print(output)
Response streaming
Streaming lets you receive each token of the response as it gets generated. This is indispensable in user-facing applications.
def streaming():
llm = DeepInfra(model_id="meta-llama/Llama-2-7b-chat-hf")
for chunk in llm.stream("[INST] Hello [/INST] "):
print(chunk, end='', flush=True)
print()
You can also use the asynchronous streaming API, natively implemented underneath.
async def async_streaming():
llm = DeepInfra(model_id="meta-llama/Llama-2-7b-chat-hf")
async for chunk in llm.astream("[INST] Hello [/INST] "):
print(chunk, end='', flush=True)
print()
Related articles
Reliable JSON-Only Responses with DeepInfra LLMs<p>When large language models are used inside real applications, their role changes fundamentally. Instead of chatting with users, they become infrastructure components: extracting information, transforming text, driving workflows, or powering APIs. In these scenarios, natural language is no longer the desired output. What applications need is structured data — and very often, that structure is […]</p>
Introducing the Batch API: Run Large Inference Jobs 20% CheaperDeepInfra's new Batch API lets you submit large volumes of completions, chat, and embedding requests as a single asynchronous job—processed within 24 hours at 20% off real-time pricing. It's fully OpenAI-compatible, so if you've used OpenAI's Batch API, you already know how it works.
Top 6 GLM-5.2 Max API Providers Compared<p>Deploying the GLM-5.2 (max) Mixture-of-Experts model — 753B total parameters with roughly 40B active per token and a 1M context window — requires infrastructure that separates production-grade API providers from the rest. This guide breaks down the top providers by throughput, latency, pricing, and quantization architecture. GLM-5.2 (max) API Review Summary (2026-06-27) TL;DR: Best Providers […]</p>
Reliable JSON-Only Responses with DeepInfra LLMs<p>When large language models are used inside real applications, their role changes fundamentally. Instead of chatting with users, they become infrastructure components: extracting information, transforming text, driving workflows, or powering APIs. In these scenarios, natural language is no longer the desired output. What applications need is structured data — and very often, that structure is […]</p>
Introducing the Batch API: Run Large Inference Jobs 20% CheaperDeepInfra's new Batch API lets you submit large volumes of completions, chat, and embedding requests as a single asynchronous job—processed within 24 hours at 20% off real-time pricing. It's fully OpenAI-compatible, so if you've used OpenAI's Batch API, you already know how it works.Top 6 GLM-5.2 Max API Providers Compared<p>Deploying the GLM-5.2 (max) Mixture-of-Experts model — 753B total parameters with roughly 40B active per token and a 1M context window — requires infrastructure that separates production-grade API providers from the rest. This guide breaks down the top providers by throughput, latency, pricing, and quantization architecture. GLM-5.2 (max) API Review Summary (2026-06-27) TL;DR: Best Providers […]</p>
© 2026 DeepInfra. All rights reserved.