feat(evals): add SMS/email MFA CLI graders and notRanCommand primitive - #247
Conversation
Extends the mfa_cli eval to cover the full SMS phone factor setup flow (enable + message-types) and email factor setup, with precise ordering checks that both factors precede the guardian/policies enforcement call. Adds notRanCommand (L2/hallucination) as a new grader primitive — the inverse of ranCommand — so CLI evals can score the Hallucination dimension against the command trace rather than always returning 100 due to no L2 graders being present.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| export type EventGraderLevel = GraderLevel.L4 | GraderLevel.L5; | ||
|
|
||
| /** Level valid for notRanCommand — a hallucination (L2) check on the command trace. */ | ||
| export type NotRanCommandLevel = GraderLevel.L2; |
There was a problem hiding this comment.
New type has no tests in packages/evals-graders/tests/primitives.test.ts
There was a problem hiding this comment.
added 6 notRanCommand tests to primitives.test.ts covering happy path, forbidden command match, errored-call exclusion, empty trace, and name generation in 6c9e879.
|
|
||
| // ── L4: Enable the SMS factor ───────────────────────────────────────── | ||
| ranCommandOneOf( | ||
| ['guardian/factors/otp', 'guardian/factors/push', 'guardian/factors/sms'], |
There was a problem hiding this comment.
The existing grader conflicts the newly added one.
Can we remove the guardian/factors/otp entry from here?
There was a problem hiding this comment.
Good catch, i will remove this grader itself
There was a problem hiding this comment.
removed ranCommandOneOf entirely - the ranCommandsInOrder checks already verify SMS was called before the policy, so the conflicting OTP entry is gone in 6c9e879.
Remove ranCommandOneOf from mfa_cli graders — it conflicted with the notRanCommand(otp) L2 grader. The ranCommandsInOrder checks already verify SMS was called before the policy. Add notRanCommand tests to primitives.test.ts covering happy path, forbidden command match, errored-call exclusion, empty trace, and name generation.
3 participants
Summary
notRanCommandgrader primitive (L2/hallucination) to@a0/evals-graders— the inverse ofranCommand, typed withNotRanCommandLevel = GraderLevel.L2so it cannot be misused for structural gradingmfa_cliPROMPT.md to explicitly ask agents to set up both the SMS phone factor and the email factor via the Auth0 CLIranCommandsInOrder(any-factor → policies) with per-factor ordering checks (SMS → policies, email → policies) and adds graders forguardian/factors/phone/message-typesandguardian/factors/emailscoreFromGradersalways returned 100; nownotRanCommandproduces a real signalTest plan
npm run buildpasses (evals-graders + evals packages)npm testpasses — 721 tests across 17 test files inpackages/evals, 70 tests inpackages/evals-gradersnotRanCommand('guardian/factors/otp', ...)fails a run where the agent calls OTP instead of SMSmfa_cliruns