Skip to content

PydanticAI adapter stringifies token usage instead of using the usage field #28

Description

@aniketwattamwar

Problem

sdk/python/src/ecp/adaptors/pydantic_ai.py already reads real token counts, then formats them into a string and appends it to evaluation_context:

usage_str = f"Usage: {usage.input_tokens} input, {usage.output_tokens} output tokens ({usage.requests} requests)"
self.captured_thoughts.append(usage_str)

This predates the structured usage field. Two consequences: the audit record's aggregated token totals stay empty, and graded text has token counts glued onto it — which can trip a does_not_contain grader for no reason.

Proposal

python Result(..., usage={"input_tokens": usage.input_tokens, "output_tokens": usage.output_tokens})

Acceptance criteria

  • Usage reported via the structured field, not appended text
  • Aggregated into the audit record's usage totals
  • Fixture regenerated
  • Audit other adapters for frameworks that expose usage

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    adaptersFramework adaptersbugSomething isn't workinggood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions