Token confidence
Token confidence from the API
When a provider returns token log probabilities, counterfix keeps a summary on each
record (record.metadata["logprobs"]: token count, mean and lowest log probability, the
least likely tokens). Ask for them with logprobs=True:
counterfix eval qa.jsonl --model openai:gpt-4o-mini --param logprobs=true --metric exact_match --metric confidence
On a guarded call, read it from the response:
import counterfix
from counterfix.detect import confidence
from openai import OpenAI
client = counterfix.guard(OpenAI())
r = client.chat.completions.create(model="gpt-4o-mini", messages=messages, logprobs=True)
confidence.mean_probability(r) # 0.87, or None when the provider sent no logprobs
confidence.describe(r) # 'token confidence 0.87 over 12 tokens; least likely: ...'
- The
confidencemetric is the answer's geometric-mean token probability, exp(mean logprob), from 0 to 1 (confidence:0.5fails answers below 0.5; with the optionuse="min"it is the least likely token's probability). - When a judge confirms invented facts (F4), the token confidence is added to the evidence ("token confidence 0.41 over 23 tokens; least likely: '1889' 0.03").
Low confidence is a signal, not a verdict: models are often unsure and right, and sure and wrong. counterfix never reports a failure from confidence alone.
| Provider | Logprobs |
|---|---|
| OpenAI Chat Completions | yes |
| Ollama 0.12.11 or later | yes (checked with qwen2.5) |
| vLLM, most OpenAI-compatible servers | usually |
| Groq | no (HTTP 400; counterfix drops the field and the metric is "not scored") |
| Gemini through its OpenAI-compatible endpoint | no (HTTP 400; Vertex AI's own API has them) |
| OpenAI Responses API, Anthropic | no |
counterfix setup --test asks each configured provider once (a 5-token request) and
prints "available" or "not available", so you know before a run.