Highly skeptical review skills for machine learning and data engineering.
| Skill | Asks |
|---|---|
is-this-real |
What does this actually do, and is its name real outside this repo? |
In testing: is-this-needed, what-goes-where, make-it-make-sense, review.
You take a look at your agent's output after its autonomous session. Or worse,
you take a look at a PR you're supposed to review. Somewhere inside some
enormous diff, you pick out a few names: a feature-lineage-conformance-contract-v4,
a MATERIALIZE_OR_HYDRATE_POLICY enum, and a
three-tier degraded-build fallback ladder.
None of this feels entirely real, even though everything looks plausible. There are no syntax errors, there are type hints everywhere, test coverage is confidently high, docstrings look beautiful, and there are no linters nagging.
From the first look you can't immediately tell that all of these changes sit on top of an algorithm that is not actually the algorithm its name claims, a subsystem nobody needs, a data flow nobody can trace, and vocabulary and concepts that exist nowhere outside your own repository.
You launch some default code review workflow of an AI coding agent of your
choice, and, in the meantime, to get a better understanding of what these code
changes could be referring to, you take a look at an accompanying markdown file,
thirteenth of its kind slapped into docs/ by now.
The document, describing the design behind the changes, talks of
delta-append build resolution for historical catch-up replay artifacts,
generated on a cadence that is itself resolved by some
resolve_train_cadence_decision method of a TrainCadenceDecision class, whose
digest has to match the provenance_token sitting in a locally-saved json
file emitted alongside snapshot_fidelity_contract generated by a
build_snapshot_fidelity_fingerprint_factory, whose hashed payload includes,
among eleven other fields, the aforementioned feature-lineage-conformance-contract-v4.
The document makes sure to warn you:
This
feature-lineage-conformance-contract-v4is load-bearing, with two honest caveats.
If you're lucky, there's a mermaid diagram somewhere after the 300 line mark of the doc with 16 nodes that you can't quite zoom into.
Meanwhile, your review agent returns with findings of impossible edge cases to
snapshot_fidelity_contract validation at read-time and possible failure modes
of the three-tier degraded-build fallback ladder, proposing an extra, fourth
tier.
At some point you begin to wonder if we've lost the plot.
If any of that sounded vaguely reminiscent of what you've dealt with, you are not alone. The skills in this repo are what I use in my day-to-day to keep my work grounded in reality, one adversarial question at a time, reading what the code does before accepting what it is called, and treating its accompanying documentation as a claim rather than a source.
/plugin marketplace add anasashb/grounded
/plugin install grounded@grounded
codex plugin marketplace add anasashb/grounded
codex plugin add grounded@groundedInstall Grounded directly with Vercel Labs' skills CLI:
npx skills@latest add anasashb/grounded --agent kiro-cli --globalKiro makes installed skills available automatically and as slash commands, such
as /is-this-real.
In the Kiro IDE:
- Open Powers and select Add Custom Power.
- Select Import power from GitHub.
- Enter
https://github.com/anasashb/groundedand select Install.
Kiro CLI v3 automatically detects Powers installed through the IDE.
You are an ML Engineer, Data Scientist, or Data Engineer.
You look at a diff showing an OutputUncertaintyCalibrator, with three private
methods and ten try-excepts wrapping a np.clip. Somewhere else in a diff
there's an EnsembleStrategy class holding one model. A +427 diff on a
markdown file talks of a "load-bearing"
DailyFrequencyModelTrainingOrchestrator class, which, upon inspection, seems
to be a for loop wrapped with a try-except.
Or, you're reviewing a branch called feature/outcome-maturity-reconciliation
and its new code and documentation to find out that what they mean by this is
just plain model evaluation with delayed labels. Maybe, somewhere in there there's
a LossAdapterResolutionPolicy doing god-knows-what. An accompanying YAML
file defines config-driven steps, fifth of which is called
entity_resolution_stage, in reality doing a simple join between two
dataframes.
Are any of these concepts in code, config or docs real?
Probability calibration is a real thing that exists in ML literature, but if
the agent implements a np.clip below 1.0 and calls it
OutputUncertaintyCalibrator, then it's wrong. It goes the other way too. You
look at an ArtifactRetentionGovernancePolicy class, convinced that the AI made
it up, and it turns out to wrap a plain TTL method under a confusing name. And
when your agent implements a custom, in-house business rule for filtering data
or outputs, it will "package it" as something so formal that one would think
this is a widely-adopted method straight from some data engineering textbook.
The is-this-real skill tries to address these issues by making the AI agent review
the code by reading the implementation first, and challenging existing naming,
methods and documentation without trusting what the repository claims it does
blindly.
It looks into three issues (and conducts comprehensive web searches):
- Where did this concept come from? Is it established in the field, adapted from something established, or local to this repository?
- Does the implementation fit the concept? Does the code have the concept's defining properties, only some of them, or does the name promise behavior that the code does not provide?
- What about the claims written around the concept - e.g. that a threshold is tuned, that an approach is standard practice, or that some benchmark was met. Does the evidence support the claims made by the code and/or its documentation?
A simplified example of what is-this-real judges:
Local names, or in-house logic and concepts are not a problem. The skill will not call a choice wrong only for being local, but will push back on it if it uses a name that is established for something else (even if similar).
This skill relies heavily on external sources and web search and does not answer from memory. If it cannot find authoritative evidence, it will report this or that claim as unverified.