Returning Log probabilities

In some cases you might want to get the log probabilities for each token generated by our LLM streaming API. By default, our streaming API returns the generated tokens on by one and the log probabilities are attached to each token.

Example

Here is quick example.

curl -X POST \   
    -d '{"input": "I have this dream", "stream": true}'  \
    -H "Authorization: bearer $(deepctl auth token)"  \
    -H 'Content-Type: application/json'  \
    'https://api.deepinfra.com/v1/inference/meta-llama/Llama-2-7b-chat-hf'
data: {"token": {"id": 29892, "text": ",", "logprob": -2.65625, "special": false}, "generated_text": null, "details": null}
data: {"token": {"id": 988, "text": " where", "logprob": -0.39575195, "special": false}, "generated_text": null, "details": null}
data: {"token": {"id": 1432, "text": " every", "logprob": -3.15625, "special": false}, "generated_text": null, "details": null}
data: {"token": {"id": 931, "text": " time", "logprob": -0.1385498, "special": false}, "generated_text": null, "details": null}

The logprob field is the log probability of the token generated.

Log probabilities are currently not returned for the non-streaming API, or out OpenAPI compatible API.