Skip to content

test(skill): add a portable behavioral benchmark #10

Description

@nstranquist

Finding

The portable skill passes ndev skills quality --strict with a 10/10 score and no findings. The command ndev skills benchmark wip-commit has no registered runner.

The product already has extensive unit, race, hosted-platform, and real capture evidence. This issue is for a stable skill-level regression baseline, not a replacement for those tests.

Acceptance

  • keep a deterministic runner with the portable skill, under skills/wip-commit/benchmark
  • make NDev resolve a runner from the current external repository before its internal fallback, without duplicating the skill package
  • use disposable Git repositories and invoke shipped entrypoints
  • cover shared and linked-worktree setup, disjoint parallel lanes, reviewed split defaults, explicit single-commit opt-out, foreign-index preservation, dry-run evidence, failed verification, and reconciliation
  • include a negative control for each safety guarantee
  • write a bounded, versioned JSON report with a score, case results, elapsed time, and zero-AI usage fields
  • support a reviewed baseline and write a proposal when the score regresses
  • run without network access, credentials, private repository data, or a user-specific path
  • keep the fast benchmark distinct from the complete unit, race, and hosted Windows suites

Done evidence

  • ndev --json skills benchmark wip-commit succeeds from this repository
  • a deliberately weakened candidate fails at least one case and cannot replace the baseline
  • Linux, macOS, and Windows runs produce the same semantic case results

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions