feat(ccaf): align to exam guide v1.0 — 30 task statements, per-objective diagnostics (single-answer items) - #52
Open
Dinesh-Girbide wants to merge 4 commits into
Open
Conversation
…sk statements, prep guide The revised CCAF exam guide (v1.0, effective July 2026, exam code CCAR-F) changes the item format and publishes a full objective index. Aligns the plugin to both. Multiple-response items. The guide specifies multiple-choice *and* multiple-response items, each stating how many responses to select, so a mock is now 45 `select: 1` + 11 `select: 2` + 4 `select: 3` (25% multiple-response, enforced exactly; choose-three capped at 5). Items carry a `select:` line that survives into the key-free questions file, because the administering skill needs the count without ever seeing the key. `init` rejects any item whose `select:` disagrees with its `answer_key`, repeats a letter, or lists letters out of A-D order — and holds a pre-filled `user_answer` to the same shape, since scoring is exact string comparison and an unsorted "DB" would otherwise score a correct "BD" as wrong. `record` accepts a letter set in any case or order and normalizes it, so scoring is exact set equality: all-or-nothing, no partial credit. Multiple-response items render with `multiSelect: true`; a wrong response count is re-asked once. The key-spread guard now covers the single-answer items only, with a band proportional to their count. Known divergence, disclosed in the README rather than hidden: every item has four options (A-D) because AskUserQuestion renders at most four. That makes a choose-three item easier than a five-option one would be, which is why choose-three stays a minority. The 30 task statements are now real. The blueprint formalizes D1.1-D5.6 with self-authored descriptions naming the exact identifiers items use (Task / allowedTools, AgentDefinition, fork_session, PostToolUse, context: fork, allowed-tools, argument-hint, Explore subagent, @import, ~/.claude.json, --json-schema, Pydantic, state manifests, the interview pattern, the Edit->Read+Write fallback). This closes a dangling reference: ccaf-tutor and ccaf-check-author already cited these codes against a blueprint that never defined them. Scenario facts corrected. Case briefs re-authored (still our own prose) around the guide's actual identifiers get_customer / lookup_order / process_refund / escalate_to_human, the 80%+ first-contact-resolution target, and four research subagents including a separate report generator. The previously invented names are gone. Bank grown 12 -> 24. Twelve new self-authored anchors covering D1.2, D1.4, D1.5, D2.1, D2.4, D3.3, D3.4, D4.5, D4.6, D5.1, D5.2, D5.3 — five of them multiple-response so that format has anchors too. All 24 tagged with `task:` and `select:`. Six task statements still lack an anchor (D1.7, D2.5, D3.5, D4.2, D4.4, D5.4); flagged in the bank, not hidden. Reporting. `score` emits `pct=` per domain, matching the real score report's percent-correct-by-domain, presented as diagnostic-only since pass/fail is the total scaled score. New data/ccaf-prep-guide.md. Study routes, four hands-on exercises mapped to task statements, multiple-response answering strategy, and certification logistics (fee, Pearson VUE delivery, 14/30/90-day retake ladder, 12-month validity, free renewal assessment). The tutor can assign exercises at domain boundaries. Everything shipped stays self-authored: the guide's published facts are encoded, none of its prose is, and its own sample questions were rewritten as original items rather than quoted. The guide PDF is gitignored so it can never ship with the plugin. Fork-cost discipline: process creation is slow on some machines (~640ms/fork measured on Windows + AV), so normalize_answer is pure bash and each validation check is a single awk pass. Net effect is that `score` got faster than before this change despite the added checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test suite could not finish on Windows. It died partway through with Cygwin's `fork: Resource temporarily unavailable` (`dofork: child died unexpectedly ... 0xC000026B`) — not an assertion failure, but the shell running out of fork capacity. Process creation costs ~640ms here (20 sequential greps = 12.8s), and check_composition_questions spent ~15 subprocesses per call while running on every init and every score. Folds check_response_mix into check_composition_questions and rewrites the whole thing as a single awk pass covering domain quotas, the item-format mix, the scenario count, per-scenario case blocks, unknown scenarios, and case-section layout. Structural problems are recorded rather than reported on sight, so failures still surface in the documented order (quotas, mix, scenarios, then layout) and every existing error message is preserved verbatim. Measured on a valid 60-item exam: init 2.8s, audit 2.6s, score 1.8s. The full suite now completes — 109 passed, 0 failed — where it previously aborted around 37. Verified every rejection branch individually (bad quota, all-single-select, three scenarios, missing case block, question under the wrong case block, up-front case blocks, duplicate case block, unlisted scenario) still produces its own specific message. Records the constraint in CLAUDE.md so the grep-per-check shape does not come back: it is what made the suite unrunnable, not a micro-optimisation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erage
Three follow-ups, plus two administration bugs the smoke test caught.
Per-task-statement diagnostics. Every item now carries a validated `task:` tag
(D1.1-D5.6) and `score` aggregates misses by task statement, so a result names
the objectives to revisit ("D5.2 Escalation and ambiguity resolution 0/2")
rather than only the weak domain. The real score report stops at domains; this
is the most actionable thing a mock can add, because it turns a weak domain
into a short reading list. `score` reads both files to join question to task
rather than widening the answers-file line format. The tag is validated at
write time — it must be well-formed, exist, and belong to the item's own domain
(D1 publishes 7 task statements, D2 5, D3-D5 6 each) — because a mistagged item
would send a candidate to study the wrong objective, which is worse than no
diagnostic at all.
Bank 24 -> 30, one anchor per task statement. Adds the six that were missing:
D1.7 (stale session state vs resume), D2.5 (Grep/Glob/Edit selection and the
Read+Write fallback), D3.5 (the interview pattern), D4.2 (few-shot for
structural variety), D4.4 (which validation failures a retry can actually fix),
D5.4 (context degradation and scratchpads). Two are multiple-response, so the
mix is now 23 single / 6 choose-two / 1 choose-three. Every task statement has
exactly one anchor, so a generator always has an in-objective example to match.
Two administration bugs, found by driving a real screen. Both were unspecified
in the skills and would have hit the first candidate:
* AskUserQuestion option `label` must be the bare letter and the option text
must go in `description`. Labels are expected to be a few words; a
full-sentence label renders badly and truncates.
* A multiSelect answer arrives comma-separated ("A, B") and must be joined
("AB") before recording. The helper rejects the raw string rather than
guessing, so this fails loudly instead of mis-recording — but it stops the
screen, so the skill has to strip the separators itself.
Also gitignores .vscode/ and records that the suite can flake on Windows when a
fork fails inside a fixture (re-run before investigating).
Verified: 119 passed, 0 failed. Separately, assembled a real blueprint-
conformant 60-item exam that init accepted first try (composition=OK, domains
16/11/12/12/9, mix 45/11/4, key spread 12/11/11/11, zero keys in the questions
file), drove one four-item screen through AskUserQuestion with a genuine
multiSelect item, recorded the round-trip, confirmed the per-objective
breakdown, and cleared the state. That exercise used placeholder stems for the
56 items it did not display, so it verifies the schema and the administration
path — not the quality of 60 generated items, which only the per-item verifier
subagents can judge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…port
Dinesh's call: keep single-option questions, and otherwise follow the guide.
This is a known DIVERGENCE from exam guide v1.0, not alignment with it. The
guide gives the item format as "multiple-choice and multiple-response items;
each item states how many responses to select". Serving single-answer only is a
deliberate decision for this plugin, and it is recorded as such in five places
rather than papered over: the blueprint's "Deliberate divergence" section, the
README fidelity table (now an explicit ❌ row), CLAUDE.md conventions, the prep
guide, and the tutor — which tells learners the format exists, that it is scored
all-or-nothing, and that they will not get to rehearse it here.
The consequence is stated wherever the divergence appears: single-answer items
are the easier format, so a score here is if anything OPTIMISTIC. 720 on a mock
is the floor of readiness, not a comfortable margin.
Removed rather than left dormant, per the Clean Code pack's dead-code rule —
version control is the archive: the `select:` payload field, multi-letter keys
and answer-set normalization, the 45/11/4 mix enforcement,
check_select_alignment, the select histogram in `audit`, multiSelect rendering,
the response-count re-ask, and the comma-join step. `normalize_answer` shrank to
a case-insensitive single letter, which still earns its place — a candidate can
type `c` into the free-text field instead of picking the option.
The seven multiple-response bank anchors were REWRITTEN as single-answer
questions on the same task statements, not deleted, so all 30 task statements
keep exactly one anchor.
One side effect worth naming. Rewriting those seven left the bank's key letters
skewed toward A, and every explanation names option letters — including English
articles ("A bare numeric code (A) assumes...") — so an automated remap would
have corrupted the prose. Rather than massage the examples, the property is now
enforced: check_key_spread tightened from a sixth-to-a-half band to a
sixth-to-a-third, and the bank carries an explicit "do not imitate these answer
positions" note. Enforcing beats hoping 30 examples teach it.
Everything else from guide v1.0 stays: the 30 task statements, the corrected
scenario identifiers, percent-per-domain plus per-objective reporting, the prep
guide, and the certification logistics.
Verified: 101 passed, 0 failed. A real 60-item assembly was accepted first try —
composition=OK, domains 16/11/12/12/9, key spread 15/15/15/15, zero keys in the
questions file — and a lowercase answer recorded and scored correctly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dinesh-Girbide
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Aligns the CCAF plugin to the revised exam guide (v1.0, effective July 2026, exam code
CCAR-F), which publishes a full 30-objective index and corrects several scenario details.Everything shipped stays self-authored: the guide's published facts are encoded, none of its prose is, and its own sample questions were rewritten as original items rather than quoted. The guide PDF is gitignored so it can never ship with the plugin.
The guide gives the item format as multiple-choice and multiple-response, each item stating how many responses to select. This plugin serves single-answer items only — four options, exactly one correct. That is a decision, not an oversight, and it has a consequence reviewers should hold onto:
Multiple-response support was built, then removed at the owner's direction. It is recorded as a divergence in five places rather than papered over: the blueprint's Deliberate divergence section, the README fidelity table (an explicit ❌ row),
CLAUDE.md, the prep guide, and the tutor — which tells learners the format exists and that they will not practise it here. The machinery is in this branch's history if the decision is ever revisited.What the guide changed, and what we did
get_customer/lookup_order/process_refund/escalate_to_human, the 80%+ FCR target, four research subagentsscoreemitspct=, labelled diagnostic-onlydata/ccaf-prep-guide.mdWeights unchanged (27/18/20/20/15 → 16/11/12/12/9).
Beyond what the real report gives you
task:tag andscoreaggregates misses by task statement, so a result saysD5.2 Escalation and ambiguity resolution — 0/2rather than only "D5 is weak". The tag is validated at write time (well-formed, exists, belongs to its own domain) because a mistagged item would send a candidate to study the wrong objective — worse than no diagnostic.check_key_spreadrefuses a skewed mock. This matters because the reference bank's own keys lean toward A, and every anchor explanation names option letters — including English articles like "A bare numeric code (A) assumes…" — so remapping them automatically would have corrupted the prose. Enforcing the property beats hoping 30 examples teach it; the bank now also carries an explicit "do not imitate these answer positions" note.Verification
initfirst try —composition=OK, domains 16/11/12/12/9, key spread 15/15/15/15, zero keys in the questions file — with a recorded round-trip (including a lowercase answer) and a confirmed per-objective breakdown.labelmust be the bare letter with the text indescription, and (then) that multi answers arrive comma-separated. The first fix survives in this branch.Performance note
Process creation is slow on some machines (~640 ms/fork measured on Windows + AV). An early grep-per-check version of the validation made the suite exhaust Cygwin's fork table mid-run, so it could not finish at all. Each validation check is now a single
awkpass andnormalize_answeris pure bash.CLAUDE.mdrecords the constraint, and notes that the suite can still flake on Windows — re-run before investigating.Still open
No full 60-item human run yet. The schema, composition enforcement, and a driven screen are verified; nobody has sat a complete mock end to end.
🤖 Generated with Claude Code