Document the agentdataflow trace format and add the first trace - #67
Open
astrogilda wants to merge 1 commit into
Open
Document the agentdataflow trace format and add the first trace#67astrogilda wants to merge 1 commit into
astrogilda wants to merge 1 commit into
Conversation
The dataset README marked the data format as a TODO while
data_validation/schemas/agentdataflow_trace.schema.json already defined
one. Replace the TODO with the schema's required and optional fields, and
add a proposed shape for the objects inside trace_data, which the schema
leaves unconstrained.
Add TRACE-0001, the directory's first trace. It is adversarial: a
tool-call scope pulled an out-of-scope document into the working context
and a later send carried the whole context, and the agent's own account
of the run is accurate about the total and silent about the attachment.
The per-step observed_by key is what makes that divergence readable, so
the example exercises it rather than assuming the agent's record is
reliable.
Validated against the schema with:
python -m jsonschema -i \
datasets/agentdataflow_toolexchange_traces/TRACE-0001.json \
data_validation/schemas/agentdataflow_trace.schema.json
Author
|
The README this PR adds asks the pull-request body to name which DSGAI entries a trace is evidence for, and what a defence would observe. It names 0 entries. TRACE-0001 maps to DSGAI06, Tool, Plugin & Agent Data Exchange Risks, at step 2, and DSGAI15, Over-Broad Context Windows & Prompt Over-Sharing, at step 4. A defence would have had to read two records the agent does not write: the gateway log and the mail relay's count of 48210 bytes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README says the trace schema is a TODO and
data_validation/schemas/agentdataflow_trace.schema.jsonalready defines one. This makes the README match the schema and adds the first trace, so the directory has something to validate against.Three things I could not settle from the repository, each written so any answer is a small edit. trace_data is an array of unconstrained objects, so the first file committed sets the de facto step shape; the example proposes one and the README marks it a proposal, so tightening it later need not undo anything. trace_id is a four-digit pattern, which caps the set at ten thousand and makes the id sequential. And the example is adversarial on purpose: a benign trace never exercises the case where the record of what happened is itself the unreliable part.
Tell me which of the three you want fixed in this PR and I will push the change here.