Auto-generated from source code docstrings.
Full path: llmcomp.question.question.FreeForm
Question type for free-form text generation.
Use this when you want to compare how different models respond to open-ended prompts. The model generates text freely up to max_tokens.
__init__(self, *, temperature: 'float' = 1, max_tokens: 'int' = 1024, judges: 'dict[str, str | dict | FreeFormJudge | RatingJudge] | None' = None, **kwargs)
Initialize a FreeForm question.
Arguments:
temperature: Sampling temperature. Default: 1.max_tokens: Maximum number of tokens in the response. Default: 1024.judges: Optional dict mapping judge names to judge definitions. Each judge evaluates the (question, answer) pairs. Values can be:- A string: loads judge from YAML by name
- A dict: creates judge from the dict (must include 'type')
- A FreeFormJudge or RatingJudge instance
**kwargs: Arguments passed to Question base class:- name: Question identifier for caching. Default: "__unnamed".
- paraphrases: List of prompt variations to test.
- system: System message prepended to each paraphrase.
- messages: Alternative to paraphrases - [{'role': ..., 'content': ...}, {'role': ..., 'content': ...}, ...]
- samples_per_paraphrase: Number of samples per prompt. Default: 1.
- logit_bias: Token bias dict {token_id: bias}.
Execute question and return results as a DataFrame.
Runs the question on all models (or loads from cache), then applies any configured judges.
Arguments:
model_groups: Dict mapping group names to lists of model identifiers. Example: {"gpt4": ["gpt-4o", "gpt-4-turbo"], "claude": ["claude-3-opus"]}
Returns:
DataFrame with columns:
- model: Model identifier
- group: Group name from model_groups
- answer: Model's response text
- question: The prompt that was sent
- api_kwargs: Full API parameters sent to model (including messages, temperature, etc.)
- paraphrase_ix: Index of the paraphrase used
- {judge_name}: Score/response from each configured judge
- {judge_name}_question: The prompt sent to the judge
Full path: llmcomp.question.question.NextToken
Question type for analyzing next-token probability distributions.
Use this when you want to see what tokens the model considers as likely continuations. Returns probability distributions over the top tokens, useful for fine-grained analysis of model behavior.
__init__(self, *, top_logprobs: 'int' = 20, convert_to_probs: 'bool' = True, num_samples: 'int' = 1, **kwargs)
Initialize a NextToken question.
Arguments:
top_logprobs: Number of top tokens to return probabilities for. Default: 20. Maximum depends on API (OpenAI allows up to 20).convert_to_probs: If True, convert logprobs to probabilities (0-1 range). If False, returns raw log probabilities. Default: True.num_samples: Number of samples to average. Useful when logprobs are non-deterministic. Default: 1.**kwargs: Arguments passed to Question base class:- name: Question identifier for caching. Default: "__unnamed".
- paraphrases: List of prompt variations to test.
- system: System message prepended to each paraphrase.
- messages: Alternative to paraphrases - [{'role': ..., 'content': ...}, {'role': ..., 'content': ...}, ...]
- samples_per_paraphrase: Number of samples per prompt. Default: 1.
- logit_bias: Token bias dict {token_id: bias}.
Full path: llmcomp.question.question.Rating
Question type for numeric rating responses.
Use this when you expect the model to respond with a number within a range. Uses logprobs to compute expected value across the probability distribution, giving more nuanced results than just taking the sampled token.
__init__(self, *, min_rating: 'int' = 0, max_rating: 'int' = 100, refusal_threshold: 'float' = 0.75, top_logprobs: 'int' = 20, **kwargs)
Initialize a Rating question.
Arguments:
min_rating: Minimum valid rating value (inclusive). Default: 0.max_rating: Maximum valid rating value (inclusive). Default: 100.refusal_threshold: If probability mass on non-numeric tokens exceeds this, the response is treated as a refusal (returns None). Default: 0.75.top_logprobs: Number of top tokens to request. Default: 20.**kwargs: Arguments passed to Question base class:- name: Question identifier for caching. Default: "__unnamed".
- paraphrases: List of prompt variations to test.
- system: System message prepended to each paraphrase.
- messages: Alternative to paraphrases - [{'role': ..., 'content': ...}, {'role': ..., 'content': ...}, ...]
- samples_per_paraphrase: Number of samples per prompt. Default: 1.
- logit_bias: Token bias dict {token_id: bias}.
Execute question and return results as a DataFrame.
Runs the question on all models (or loads from cache), then computes expected ratings from the logprob distributions.
Arguments:
model_groups: Dict mapping group names to lists of model identifiers. Example: {"gpt4": ["gpt-4o", "gpt-4-turbo"], "claude": ["claude-3-opus"]}
Returns:
DataFrame with columns:
- model: Model identifier
- group: Group name from model_groups
- answer: Mean rating (float), or None if model refused
- raw_answer: Original logprobs dict {token: probability}
- probs: Normalized probabilities dict {int_rating: probability}
- question: The prompt that was sent
- api_kwargs: Full API parameters sent to model (including messages, temperature, etc.)
- paraphrase_ix: Index of the paraphrase used
Full path: llmcomp.question.judge.FreeFormJudge
Judge that evaluates answers using free-form text responses.
Use as a judge in FreeForm questions to have an LLM evaluate the (question, answer) pairs. The judge paraphrase should contain {answer} placeholder, and optionally {question}.
Initialize a FreeFormJudge.
Arguments:
model: Required. Model identifier to use for judging (e.g., "gpt-4o").temperature: Sampling temperature. Default: 0.**kwargs: Arguments passed to FreeForm base class. Must include:- paraphrases: Single-element list with the judge template. Template must contain {answer}, optionally {question}. Example: ["Is this answer correct? {answer}"]
Return all cached judge evaluations as a DataFrame.
Useful for inspecting what the judge has evaluated so far.
Returns:
DataFrame with columns:
- question: Original question (None if judge doesn't use {question})
- answer: Original answer that was judged
- judge_question: The formatted prompt sent to the judge
- judge_answer: The judge's response text
Full path: llmcomp.question.judge.RatingJudge
Judge that evaluates answers using numeric ratings.
Use as a judge in FreeForm questions to have an LLM rate the (question, answer) pairs. Returns mean rating computed from logprobs. The judge template should contain {answer} placeholder, and optionally {question}.
Initialize a RatingJudge.
Arguments:
model: Model identifier to use for judging (e.g., "gpt-4o").**kwargs: Arguments passed to Rating base class. Must include:- paraphrases: Single-element list with the judge template. Template must contain {answer}, optionally {question}. Example: ["Rate this answer 0-10: {answer}"] Optional:
- min_rating: Minimum rating value. Default: 0.
- max_rating: Maximum rating value. Default: 100.
Return all cached judge evaluations as a DataFrame.
Useful for inspecting what the judge has evaluated so far.
Returns:
DataFrame with columns:
- question: Original question (None if judge doesn't use {question})
- answer: Original answer that was judged
- judge_question: The formatted prompt sent to the judge
- judge_answer: Expected rating (float) computed from logprobs
- judge_raw_answer: Raw logprobs dict {token: probability}
Full path: llmcomp.config.Config
Global configuration for llmcomp.
Modify class attributes directly to change configuration. Changes take effect immediately for subsequent operations.
| Attribute | Default | Description |
|---|---|---|
timeout |
60 |
API request timeout in seconds |
reasoning_effort |
'none' |
|
max_workers |
100 |
Max concurrent API requests (total across all models) |
cache_dir |
'llmcomp_cache' |
Directory for caching question and judge results |
yaml_dir |
'questions' |
Directory for loading questions from YAML files |
verbose |
False |
Print verbose messages (e.g., API client discovery) |
URL-key pairs for client creation.
Auto-discovered from environment variables on first access. Users can modify this list (add/remove pairs).
Returns list of (base_url, api_key, env_var_name) tuples.
Get or create an OpenAI client for the given model.
Clients are cached in client_cache. The first call for a model will test available URL-key pairs in parallel to find one that works. Thread-safe: only one thread will attempt to create a client per model. Failures are also cached to avoid repeated attempts.
Temporarily set config values, restoring originals on block exit.
Example:
with Config.override(timeout=5, max_workers=10):
runner.get_many(...)
# timeout and max_workers are back to their previous values here
Only keys in Config._defaults are accepted (typos raise ValueError).
Nesting is supported: each call saves the value at the moment it is entered and restores it on exit, so inner blocks compose cleanly with outer ones.
On exit, the overridden keys are checked: if any of them was changed
from inside the block (e.g. Config.timeout = 99 between enter and
exit, or a concurrent thread mutating the same key), a RuntimeError
is raised after the originals are restored. Mutating an overridden
key inside its own block is not supported.
Note: this is process-global, just like Config.X = y. It is NOT
thread-local: if two threads enter overlapping overrides for the
same key, they will clobber each other's saved values. Use only
when you control concurrent access to the relevant keys.
Reset all configuration values to their defaults.
Full path: llmcomp.runner.model_adapter.ModelAdapter
Adapts API request params for specific models.
Handlers can be registered to transform params for specific models. All matching handlers are applied in registration order.
Register a handler for model-specific param transformation.
Arguments:
model_selector: Callable[[str], bool] - returns True if this handler should be applied for the given model name.prepare_function: Callable[[dict, str], dict] - transforms params. Receives (params, model) and returns transformed params.
Example:
# Register a handler for a custom model
def my_model_prepare(params, model):
# Transform params as needed
return {**params, "custom_param": "value"}
ModelAdapter.register(
lambda model: model == "my-model",
my_model_prepare
)
Prepare params for the API call.
Applies all registered handlers whose model_selector returns True. Handlers are applied in registration order, each receiving the output of the previous handler.
Arguments:
params: The params to transform.model: The model name.
Returns:
Transformed params ready for the API call.
Full path: llmcomp.question.question.Question
Create a Question instance from a type string and keyword arguments.
Factory method that instantiates the appropriate Question subclass based on the 'type' parameter.
Arguments:
**kwargs: Must include 'type' key with one of:- "free_form": Creates FreeForm question
- "rating": Creates Rating question
- "next_token": Creates NextToken question
- "free_form_judge": Creates FreeFormJudge
- "rating_judge": Creates RatingJudge Other kwargs are passed to the constructor.
Returns:
Question subclass instance.
Raises:
ValueError: If 'type' is missing or invalid.
Example:
>>> q = Question.create(
... type="free_form",
... name="my_question",
... paraphrases=["What is 2+2?"]
... )
Load question configuration as a dictionary from YAML files.
Searches all YAML files in Config.yaml_dir for a question with matching name.
Arguments:
name: The question name to look up.
Returns:
Dict containing the question configuration (can be passed to Question.create).
Raises:
ValueError: If question with given name is not found.
Example:
>>> config = Question.load_dict("my_question")
>>> config
{'type': 'free_form', 'name': 'my_question', 'paraphrases': [...]}
Load and instantiate a Question from YAML configuration.
Convenience method combining load_dict() and create().
Arguments:
name: The question name to look up in YAML files.
Returns:
Question subclass instance.
Raises:
ValueError: If question not found or has invalid type.
Example:
>>> q = Question.from_yaml("my_question")
view(self, df: 'pd.DataFrame', *, sort_by: 'str | None' = '__random__', sort_ascending: 'bool' = True, open_browser: 'bool' = True, port: 'int' = 8501) -> 'None'
View a DataFrame directly (class method usage).
Arguments:
sort_by: Column to sort by. Default "random" shuffles rows randomly (new seed on each browser refresh). Use None for original order.
plot(self, df: 'pd.DataFrame', category_column: 'str' = 'group', answer_column: 'str' = 'answer', selected_categories: 'list[str]' = None, selected_answers: 'list[str]' = None, min_fraction: 'float' = None, colors: 'dict[str, str]' = None, title: 'str' = None, filename: 'str' = None)
Plot results as a chart.
Can be called as: - Question.plot(df) - plot a DataFrame directly - question.plot(model_groups) - run df() on models, then plot - question.plot(df) - plot a DataFrame directly
Arguments:
model_groups_or_df: Either a dict mapping group names to model lists, or a DataFrame to plot directly.category_column: Column to group by on x-axis. Default: "group".answer_column: Column containing answers to plot. Default: "answer" (or "probs" for Rating questions).selected_categories: List of categories to include (in order). Others excluded.selected_answers: List of answers to show in stacked bar. Others grouped as "[OTHER]".min_fraction: Minimum fraction threshold for stacked bar. Answers below grouped as "[OTHER]".colors: Dict mapping answer values to colors for stacked bar.title: Plot title. Auto-generated from question if not provided.filename: If provided, saves the plot to this file path.
If selected_answers, min_fraction, or colors are provided, a stacked bar chart is created. Otherwise, llmcomp will try to create the best plot for the data.
Clear cached results for this question and model.
Arguments:
model: The model whose cache should be cleared.
Returns:
True if cache was found and removed, False otherwise.
Example:
>>> question = Question.create(type="free_form", paraphrases=["test"])
>>> question.df({"group": ["gpt-4"]}) # Creates cache
>>> question.clear_cache("gpt-4") # Clear cache
True
>>> question.clear_cache("gpt-4") # Already cleared
False