Scry what model is really behind that endpoint.
An open signature format and a self-service tool to verify that an LLM endpoint serves the model and the precision it declares, with an optional community registry for model identification.
An inference endpoint declares "this is Llama 3.1 70B Instruct at fp16". Nothing in the API response proves it. A provider, a gateway, or a misconfigured deployment can serve a smaller model or a more aggressive quantization behind the same name, and the consumer has no independent way to notice. This project provides that way.
A model served at a given precision on a given engine has a measurable behavioral signature: the distributions of token log-probabilities it produces on a standard set of probe prompts at temperature zero. The signature of a public model is reproducible by anyone from the published weights. Verification is a statistical comparison between the signature you generate yourself and the evidence collected from the endpoint under examination.
flowchart LR
PS[Probe set<br/>versioned, part of the spec] --> RUN[Probe runner]
EP[Subject endpoint<br/>OpenAI-compatible API] <--> RUN
RUN --> EB[Evidence bundle]
REF[Reference signature<br/>self-generated or from registry] --> ME[Matching engine]
EB --> ME
TH[Thresholds<br/>from published calibration] --> ME
ME --> V[Verdict<br/>confirmed, refuted, inconclusive, unknown]
The primary workflow, verify, requires no catalog and no third party: you generate the reference signature from public weights on any endpoint you control, then compare. The community registry accelerates verification and enables the secondary workflow, identify, but the project is fully functional with an empty registry.
pip install modelscry
# 1. Generate the reference signature from a deployment you control
modelscry fingerprint \
--endpoint http://localhost:8080/v1 \
--model llama-3.1-8b-instruct \
--claim "llama-3.1/8b-instruct@q4_k_m+llama.cpp" \
--probe-set probes/probe-set-0.1.0.json \
--output my-reference.json
# 2. Verify a third-party endpoint against it
modelscry verify \
--endpoint https://provider.example/v1 \
--model llama-3.1-8b-instruct \
--reference my-reference.json \
--probe-set probes/probe-set-0.1.0.json \
--thresholds calibration/thresholds-0.1.0.example.jsonExit codes: 0 confirmed, 1 refuted, 2 inconclusive. The verdict is a reproducible JSON document; anyone with the same probe set, signature, and thresholds artifact obtains the same outcome.
- L1 (log-probabilities). Available when the endpoint returns
top_logprobs. Supports both model discrimination and precision discrimination. - L0 (greedy text only). Fallback when log-probabilities are unavailable. Supports model discrimination with reduced power; it does not support precision verdicts. The verdict always declares the level it used.
Honesty about scope is a design requirement (spec section 11):
- It certifies an endpoint on a sampling basis at probing time; it does not certify individual responses or continuous behavior between sessions.
- It certifies cooperative or unaware endpoints. A provider that recognizes the public probe set and routes probes to the declared model while serving cheaper responses to regular traffic defeats the protocol by design.
- Thresholds are valid only when backed by the published calibration dataset. The values shipped in
calibration/*.example.jsonare development placeholders and produce nonconforming verdicts, as the CLI warns. - Passive attribution of already-generated text, without query access, is out of scope.
| Path | Content |
|---|---|
spec/ |
The signature specification and the generated JSON Schemas |
probes/ |
Versioned probe sets (part of the specification) |
registry/ |
Community signature registry (optional infrastructure) |
calibration/ |
Calibration datasets and derived thresholds artifacts |
src/modelscry/ |
Reference implementation (Python 3.11+) |
scripts/ |
Schema export and probe set hashing utilities |
Version 0.1.0 is a draft: the specification, the data models, the distance functions, and the matching engine are implemented and tested offline; the initial probe set is a 6-probe illustrative draft, to be replaced by the constructed 48-probe set described in spec Appendix A; the calibration campaign has not run yet, so no conforming thresholds artifact exists. Contributions are welcome, starting from the open questions in spec Appendix B.
This project builds on a well-established research line and differs from it deliberately. LLMmap (Pasquini, Kornaropoulos, and Ateniese, USENIX Security 2025) demonstrated active fingerprinting with a trained classifier over a closed set of model versions. TOPLOC (Ong et al., 2025) provides stronger per-inference guarantees when the serving engine cooperates by committing to activations. This project targets the space between: no trained classifier (registry matching scales by contribution, not retraining), no prover cooperation required, an open signature format, and precision-level discrimination as a first-class goal.
Code is released under the MIT License. Registry signatures and calibration datasets are released under CC-BY 4.0.