Skip to content

fix(pydantic-ai): preserve usage in structured results - #31

Closed
mikemikimike wants to merge 1 commit into
evaluation-context-protocol:mainfrom
mikemikimike:contrib/issue-28
Closed

mikemikimike wants to merge 1 commit into
evaluation-context-protocol:mainfrom
mikemikimike:contrib/issue-28

Conversation

@mikemikimike

Copy link
Copy Markdown

Related issue

Closes #28

Background

The PydanticAI adapter read real token counts but appended them to evaluation_context as text. This left the structured audit usage empty and polluted evaluator-visible text with accounting metadata.

Changes

  • Return PydanticAI input and output token counts through Result.usage.
  • Keep token accounting out of evaluation_context.
  • Regenerate the PydanticAI seeded conformance fixture and add direct adapter assertions.

Implementation and compatibility

The adapter now uses the existing ECP Result.usage contract, which the runtime audit layer aggregates across steps. No protocol shape or public adapter constructor changes; only the previously empty usage field and evaluation-context content change for PydanticAI results that report usage.

Tests

  • python scripts/seed_adapter_fixtures.py — regenerated fixture; usage is {"input_tokens": 61, "output_tokens": 14}.
  • PYTHONPATH=sdk/python/src;runtime/python/src python -m unittest sdk/python/tests/test_adapters.py sdk/python/tests/test_adapter_conformance.py — 11 passed.
  • PYTHONPATH=sdk/python/src;runtime/python/src python -m unittest discover sdk/python/tests -p 'test_*.py' — 37 passed.
  • PYTHONPATH=sdk/python/src;runtime/python/src python -m unittest runtime/python/tests/test_audit.py — 14 passed.
  • ruff check sdk/python/src/ecp/adaptors/pydantic_ai.py sdk/python/tests/test_adapters.py scripts/seed_adapter_fixtures.py — passed.

Checks not run or environment limitations

  • ruff format --check reports pre-existing formatting differences in the touched adapter/test files and the seed script; no unrelated formatting was included.
  • Framework-specific live PydanticAI tests were not run because the adapter is intentionally tested with seeded replay fixtures and no framework/API credentials are available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PydanticAI adapter stringifies token usage instead of using the usage field

1 participant