G-NLL

Reference implementation and reproduction code for “Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure”, published at ICLR 2026 (paper).

Lukas Aichberger, Kajetan Schweighofer, and Sepp Hochreiter

Method

G-NLL estimates predictive uncertainty from the negative log-likelihood of a single greedily decoded response. A larger G-NLL indicates that the model assigned a lower joint probability to its response and therefore represents higher uncertainty.

G-NLL requires one generated response and the conditional log-probability of each selected token. It does not require repeated sampling, semantic clustering, or an auxiliary inference model.

Installation

The reference environment uses Python 3.11. The numerical implementation of G-NLL can be installed without PyTorch, Transformers, model weights, or network access:

python -m pip install -e .

import math

from gnll.measures import g_nll

uncertainty = g_nll([-0.10, -0.35, -0.05])
assert math.isclose(uncertainty, 0.50)

The input consists of natural logarithms of the probabilities assigned to the generated tokens. The function returns their negative sum without length normalization.

Install the additional dependencies for model generation and reproduction of the experiments with:

python -m pip install -e ".[experiments]"
cp .env.example .env

Set HF_TOKEN in .env if access to a gated Hugging Face model requires authentication. Model and dataset files are downloaded when they are first used.

Tests

Install the development dependencies and run all unit tests with:

python -m pip install -e ".[dev]"
python -m pytest

Using python -m pytest runs pytest with the same Python interpreter in which the package was installed. Pytest reads its configuration from pyproject.toml and automatically discovers the tests in tests/. The unit tests require no model downloads or GPU.

Minimal generation example

The following commands calculate G-NLL for ten TriviaQA examples using the configured Llama 3.1 8B model:

gnll-generate \
  --config configs/main.yaml \
  --model llama-3.1-8b \
  --dataset trivia_qa \
  --task short \
  --num-samples 0 \
  --limit 10 \
  --output outputs/example/generations.jsonl

gnll-compute \
  --input outputs/example/generations.jsonl \
  --g-nll-only \
  --output outputs/example/measures.jsonl

This example still downloads the selected model and dataset. A smaller compatible causal language model can be supplied with --model-id for local testing, but reproducing the reported results requires the model revisions specified in configs/main.yaml.

Reproducing the paper

The configuration files record the models, datasets, decoding parameters, random seeds, and evaluation settings used for each reported result:

Result	Configuration
Tables 1–3	configs/main.yaml
Figure 2	configs/ablation_decoding.yaml
Table 4	G-NLL and LN-G-NLL from configs/main.yaml
Table 5	configs/ablation_beam.yaml
Figures 1 and 3–5	configs/synthetic.yaml

To reproduce Tables 1–3, first generate and inspect the shell script containing all required commands:

gnll-plan \
  --config configs/main.yaml \
  --output-root outputs/main \
  --output run-main.sh

less run-main.sh
bash run-main.sh

The generated script performs response generation, uncertainty calculation, correctness assessment, and statistical evaluation. These commands may be run serially or submitted individually to a workload manager while preserving their dependency order.

After all computations have completed, verify that the expected results are present and generate the paper tables:

gnll-audit --config configs/main.yaml --output-root outputs/main
gnll-tables --config configs/main.yaml --input-root outputs/main

The reproducibility guide gives the commands for all ablations and synthetic experiments, together with the exact numerical conventions and computational requirements. The output-format reference describes the generated records and their validation.

Experimental design

The empirical evaluation comprises six models from the Llama 3.1 and Falcon Mamba families, TriviaQA, SVAMP, and NQ-Open, and both short-answer and single-sentence response formats. Correctness is evaluated with SQuAD F1 where applicable and with the reported Llama 3.1 70B judges. AUROC is the primary evaluation metric.

The implementation follows these methodological choices:

• gnll.measures.g_nll computes the unnormalized negative sum of the selected token log-probabilities and is used throughout the experimental code.
• The response used for G-NLL is generated by greedy decoding. LN-G-NLL divides the same quantity by response length and is reported only as an ablation.
• PE, LN-PE, SE, LN-SE, and D-SE are calculated from ten unrestricted multinomial samples, following the paper.
• Semantic clusters are obtained by strict bidirectional entailment with the specified DeBERTa model, conditioned on the question.
• Each output records its configuration, code, model and dataset revisions, package versions, and file hashes.
• Statistical analyses read the locally stored JSON and JSONL results. Neither Weights & Biases nor a particular cluster scheduler is required.

The complete evaluation requires access to gated models and substantial GPU resources. The experiments reported in the paper used a node with eight NVIDIA A100 GPUs.

Repository scope and provenance

This repository focuses on the method, baselines, and ablations reported in the final paper. It derives in part from jlko/semantic_uncertainty at commit a8d9aa8cecd5f3bec09b19ae38ab13552e0846f4.

Exploratory uncertainty measures and experiments that were not retained in the paper are intentionally excluded. Datasets and model weights are downloaded from their respective providers and are not redistributed by this repository; their individual licenses and terms of use continue to apply.

Citation

@inproceedings{aichberger2026rethinking,
  title     = {Rethinking Uncertainty Estimation in LLMs:
               A Principled Single-Sequence Measure},
  author    = {Aichberger, Lukas and Schweighofer, Kajetan and Hochreiter, Sepp},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026},
  url       = {https://proceedings.iclr.cc/paper_files/paper/2026/file/bcc9a3b15a5bf2aa5003da6919e992da-Paper-Conference.pdf}
}

License

This repository is distributed under the BSD 3-Clause Clear License, matching the license of the source repository from which parts of the implementation were derived. Using the same permissive license for the complete codebase preserves the upstream requirements without adding further restrictions or a second license.

Model weights and datasets are obtained separately from their respective providers and are not distributed under the repository's software license.