Function Calling

Function Calling allows models to call external functions provided by the user, and use the results to provide a comprehensive response to the user query. To learn more, read our blog.

We provide OpenAI compatible API.

Currently supported for:


Let's go through some simple example of requesting a weather.

This is how you set up our endpoint

import openai
import json

client = openai.OpenAI(

This is the function that we will execute whenever the model asks us to do so

# Example dummy function hard coded to return the same weather
# In production, this could be your backend API or an external API
def get_current_weather(location):
    """Get the current weather in a given location"""
    print("Calling get_current_weather client side.")
    if "tokyo" in location.lower():
        return json.dumps({
            "location": "Tokyo",
            "temperature": "75"
    elif "san francisco" in location.lower():
        return json.dumps({
            "location": "San Francisco",
            "temperature": "60"
    elif "paris" in location.lower():
        return json.dumps({
            "location": "Paris",
            "temperature": "70"
        return json.dumps({"location": location, "temperature": "unknown"})

Let's now call our DeepInfra endpoint with tools and a user request

# here is the definition of our function
tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                        "The city and state, e.g. San Francisco, CA"
            "required": ["location"]

# here is the user request
messages = [
        "role": "user",
        "content": "What is the weather in San Francisco?"

# let's send the request and print the response
response =
tool_calls = response.choices[0].message.tool_calls
for tool_call in tool_calls:


{'id': 'call_X0xYqdnoUonPJpQ6HEadxLHE', 'function': {'arguments': '{"location": "San Francisco"}', 'name': 'get_current_weather'}, 'type': 'function'}

Now let's respond back with a function call response and see the results

# extend conversation with assistant's reply

for tool_call in tool_calls:
  function_name =
  if function_name == "get_current_weather":
      function_args = json.loads(tool_call.function.arguments)
      function_response = get_current_weather(

  # extend conversation with function response
      "role": "tool",
      "content": function_response,

# get a new response from the model where it can see the function responses
second_response =



The current temperature in San Francisco, CA is 60 degrees.

Tips on using function calling

Here are some tips to get the most out of function calling:

  • Make sure the descriptions of the functions are well written, it will make models perform better.
  • Make sure to use lower temperatures < 1.0, this ensures the model won't plug in some random stuff to the parameters
  • Try not to use system messages
  • Models function calling quality degrades with the number of functions supplied.
  • Try to keep top_p and top_k values on the default


There is additional usage for prompting when using function calling + your function definitions will also be counted toward usage.


  • single calls
  • parallel calls (though quality might be lower, it's under active development)
  • tool_choice with only auto or none

Not supported:

  • nested calls (not supported)
  • streaming mode

© 2023 Deep Infra. All rights reserved.

Discord Logo