Add consulting plugin: UBML discovery pipeline

[david-hudec-networg]

Consulting plugin: UBML discovery pipeline

Four skills that take a transcript, interview or document and end with a
validated business model.

	skill	does
1	extract-insights	register the source, mine every claim, mark them proposed
2	validate-insights	walk them with a stakeholder, one at a time
3	promote-to-model	validated insights become actors, entities, processes
4	validate-model	CLI check, gate every commit on zero errors

The chain the plugin exists to protect: source → insight → model. Any field
in the resulting model walks back to the sentence someone said.

The insights are the record; the model is a reading aid over them. This is
the correction that shaped the plugin most. A model built as the destination
quietly discards reasoning, because reasoning rarely lands on a model element —
and the reasoning is the part a business twin exists to hold. The model is built
last, from whatever it can usefully consume, and it is expected to be smaller
than the evidence.

Requires ubml 1.4

sources and insights are 1.4 document types and the pipeline is built on
both, so each skill pins the major version: npx ubml@1.4 <command>.

Worth pinning rather than trusting whatever is installed. In 1.3, validate on a
directory ran only the id cross-reference pass — it accepted illegal enum values,
invented properties, malformed ids and missing required fields without a word, so
a clean bill of health from it meant only that the YAML parsed.

Two rules worth reading before the diff

confidence and status are different fields with different owners.
confidence is the author's read of the evidence. status is the reviewer's
read of the extraction. Corroboration between two sources raises the first and
leaves the second alone. Conflating them produces a workspace that looks
reviewed and is not.

A reviewer's answer validates a claim; it never becomes one. New information
from a reviewer needs a source registered before it enters, even if that source
is a corridor conversation with a named person on a given date.

Run twice on the same engagement

The pipeline was run end to end on a live engagement, then the workspace was
reset and run again from the same raw sources. Both runs are on record.

	first run	rebuild
sources	4	5
insights extracted	88	185
walked with the stakeholder	19	180
model	mostly empty scaffolds	41 elements, 100/100

The gap is not effort. The first run scaffolded all fourteen document types up
front, named files after the capability, judged extraction by whether a claim
would land on a model element, and kept conclusions while cutting the reasons
out of the very quotes it preserved. Every rule below is the fix for something
in that column.

What the runs changed

A derived source is not a second witness. The commercial proposal in that
engagement was written from the client's own outline and said so. Nineteen of
its twenty-six insights restated that outline. Treating agreement as convergence
would have raised confidence on claims with a single origin, so the skill now
asks where the second source got the claim before counting it as corroboration.

Record the restatement anyway. It adds no confidence, but it adds a second
dated instance of someone saying it — which is what settles a claim that is
later disputed. A de-duplicated workspace has thrown that away.

A risk raised and not taken up is still an insight. Extract what was said,
not only what was agreed. The dated record that someone put a risk on the table
is worth most precisely when nobody acted on it.

decision and constraint are the pair most often confused. A decision
records that a choice was taken; a constraint records a limit the model must
respect. Something agreed whose operative content is a boundary — out of scope,
never shown to X — is a constraint.

Files are named for the document type, never for the subject. A PR#####
already carries its own name and description. Putting the capability in the
filename encodes a project boundary that survives until the second capability,
and then means a second place to look.

The rule needed correcting once the run reached the later document types: three
of them (glossary, hypotheses, strategy) require a prefix segment and
will not be recognised without one. So the rule is which word goes in it -
business-terms.glossary.ubml.yaml, not order-and-collect.glossary.ubml.yaml

• rather than whether one belongs there at all.

One process per outcome, not per path to it. startsWith is an array, and
that is the schema saying multiple entry points into one process is the expected
case. The pull toward splitting is that the branches are what you can see and
the shared tail is further down. Count before splitting: in this engagement
three "obviously different" paths shared ten of eighteen steps.

Read back the validated insights the model does not cite. Deriving each
element type separately is what makes the work honest, and it is the same thing
that leaves a claim belonging to no type unowned. This check earned itself
twice: the first time it recovered a missing delivery-note step, the second it
caught that nothing in the model recorded the capability shipping as a second
application — three validated insights that no single type had claimed.

Show a correction back before moving on. A correction is where the author is
most likely to introduce a new error — just told they misread something,
rewriting under that pressure, reviewer already moving on. They can only catch a
bad correction if they see it.

Do not model a capability the sources only mention. A source about one
capability will mention its neighbours. Those mentions are evidence about the
capability being discussed, not about the neighbour. Extract them, exclude the
neighbour from the model, and put the reason in the workspace description so the
gap does not read as a decision.

The three lenses fill up with the wrong thing. A glossary becomes a
second-rate copy of the entity list unless it is aimed at words that carry
several meanings — in this engagement "order" meant three documents running in
two directions, and "price" meant three numbers, only one of them in the system
being designed. A hypothesis tree becomes a to-do list unless pending decisions
are kept out of it and left as insights with kind assumption.

Check what nextid hands you. It prefers a cached id count over reading the
files, and the cache survives a branch switch. It will report Highest: AC00008
and Next ID: AC01020 in the same breath, and say it again one higher next
time.

invalidated is not a status value. The enum is proposed, validated,
disputed, retired. validate-insights previously told a reviewer rejecting
a claim to write invalidated, which the validator refuses.

Merge TALXIS/ubml#46 first

The skills call ubml walk, rely on ubml nextid being trustworthy, and put
derivedFrom on strategy elements. None of that exists in 1.4.1. Merged in the
wrong order, this plugin ships instructions that produce an invalid workspace.

On length

CONTRIBUTING targets ~70 lines. These are 82-122, and validate.mjs passes
(bodies 900-1500 tokens against a 5000 budget). Here is the accounting rather
than leaving it to be found.

674 lines down to 411, no rule lost. Three passes:

1. Cut paragraphs that restated the one before them.
2. Applied this repo's own principle - every workaround a skill describes is a
   defect in the tools. Five were, filed as U1-U5 in TOOLING-BACKLOG, all fixed
   in fix: seven CLI defects a real end-to-end run exposed ubml#46, and the prose deleted rather than reworded.
3. Moved the ordering, presentation-format and per-source map out of
   validate-insights into ubml walk, where they are assertions in the CLI's
   own tests instead of descriptions here.

I tested it rather than asserting it. 55 candidate rules were mined mechanically
from the original 674-line version and checked against the current plugin: 54
carried, and the one that failed was real - Model the business twin only had
kept the rule and dropped the examples, so an agent no longer knew what counted
as delivery-side. Restored. The same check caught three rules I had assumed
walk covered and it does not: the CLI sorts on a source's date and cannot
know a note was circulated later than written, and folding a wholesale-restating
source into its parent is judgement. Both back in prose.

What is left is one rule plus one failure mode each. The next cut removes failure
modes, which is a trade rather than a tidy-up: the implement skills are ~115
tokens because they route to txc for the how, while these carry a method, and
an agent that cannot see what breaks will optimise a counter-intuitive rule away.
That is what the first run of this pipeline did.

Overrule me if the target is hard and they come down to ~70 with the failure
modes stripped.

Notes for review

Registering a source is a step, not a skill. 1.4 makes sources a document
type with nextid coverage, so what would have been a separate capture skill is
one entry in sources.ubml.yaml and an invariant in extract-insights: register
before interpreting.

The skills stay thin on purpose. They route to the ubml CLI for shapes and
ids rather than restating a schema that will drift. "Ask the CLI first, never
write a UBML file from memory" is an invariant, not advice.

That invariant has one documented exception now, and the skills say so:
VS##### and CP##### have no derivedFrom in 1.4, so provenance in a strategy
document is prose the validator cannot check.

One limitation worth knowing. supersedes takes a single reference, so one
later decision cannot formally close two earlier open questions. It came up
twice in one walk; both times the second link had to be a related plus a note.

Upstream

The rebuild surfaced defects in the ubml CLI itself, filed separately as
TALXIS/ubml#46: ubml add scaffolded a document that ubml validate rejects,
for all thirteen types it can create. That is why this plugin's invariant is
"ask the CLI for the shape" and not "copy the scaffold".

[david-hudec-networg]

Smoke-tested end to end against ubml@1.4.1 on a scratch workspace: init → add sources → register an SR##### → emit insights citing it → promote to actors and a process with derivedFrom → validate → show. Final state 5 files, 0 errors, health 100/100.

Both traceability guarantees are enforced by the validator, not just by convention. A derivedFrom: [IN09999] on an actor and a source: SR09999 on an insight each fail with Reference to undefined ID, so a promoted element cannot outlive the insight it came from and an insight cannot cite a source that was never registered.

Two corrections folded in from the run:

• invalidated is not a status value. The enum is proposed, validated, disputed, retired. validate-insights told a reviewer rejecting a claim to write invalidated, which the validator refuses — a rejected claim is disputed. Fixed, and the enum is now stated in the skill so the next author does not invent a word that reads right.
• Steps carry RACI, not actor, and flow lives in links, not next. Writing the process from memory produced nine errors; ubml syntax step gives the real shape. This is the case for the skills' "ask the CLI first, never write a UBML file from memory of the schema" rule, so it stays as an invariant rather than being papered over with a template.

Two upstream wrinkles worth knowing, neither blocking:

• ubml syntax hypothesis answers Unknown element type: hypothesis while listing hypothesis among the available types.
• ubml add hypotheses scaffolds a shape its own bundled schema rejects — an id property and array children where the schema wants HY-keyed objects. Five errors on an otherwise-clean init, so do not copy that scaffold.

[david-hudec-networg]

Superseded by #9, which contains every commit from this PR plus the paired extract-and-propose change.

Closing rather than leaving both open: this is a strict subset, so reviewing it separately would mean reading the same four skills twice. The reasoning that lived here - the two-run comparison, the rules each run forced, the length accounting - is carried forward in #9's description.