feat(mdcode): guard the demo agent on rules stated in words - #423
Merged
Merged
Conversation
libei
force-pushed
the
judge-demo
branch
3 times, most recently
from
September 14, 2026 18:19
bf7ccb1 to
54533aa
Compare
The commerce demo declared three constraints as arithmetic and guarded on none of them, because nothing evaluates an expression yet. It now declares three more as sentences a judge settles, guards on those, and reaches all three of `on_violation`'s outcomes against a live store: a credit over the self-service ceiling is held, a credit whose memo names no service failure is written with a warning, and a credit the memo admits is one piece of a larger amount is refused. Every transcript the README pastes is a capture of that run. The three expressions stay declared and unreferenced. Two of them compare the call against stored rows and are waiting on the evaluator, for two different reasons. `CreditWithinOrderTotal` is out of reach of the judge this demo hires, which makes one model call with no tools -- but `Judge` is an interface, and an implementation over a store connection would settle it, which is how reviewing agents elsewhere check that class of rule. What argues for leaving it an expression is cost and repeatability, plus the fact that a judge reads outside the transaction. `OrderTotalMatchesLineItems` is out of reach of any judge however much it can read, because it constrains the state the write leaves behind and guards are settled before the transaction opens. The third has a judged twin, marked temporary: once arithmetic can be settled by a query, settling it by a model call is paying for nondeterminism. That twin is named for the expression it duplicates, with how it is settled on the end -- `CreditUnderReviewThreshold` and `CreditUnderReviewThresholdWithJudge`. The two are one rule down to the description and differ only in who settles it, so the suffix says which of the pair goes away when the evaluator lands. It is on that pair alone: the other two judgments have no expression counterpart and need no qualifier. The split-credit rule needed help from the other side. The action's `ai_context` now tells callers to disclose a split, because a fact left out of the memo is a fact no rule can weigh. That makes it a check on honest mistakes rather than a control: it does not stop a caller who has decided to get around it, and the version that would hold regardless is an expression over what is already stored. Both documents lead with the failure mode a judged guard has and an expression does not. The judge returns `holds: true` when it cannot assess a rule, so an unassessable guard is held rather than reported. The rule never fires, the write goes through, and no warning, log line or loader diagnostic says anything. The only way to know a judged guard works is to run a case it should refuse. They also stop short of saying a judge cannot see stored rows. The interface says what the request carries, not what an implementation may hold, and the durable limit is a matter of timing: a guard is settled before the transaction opens, so no judge can be shown the state a write would produce. The actions guide keeps `Name fields model-qualified` as the habit, since resolving `Entity.field` tokens buys rename safety that naming an argument does not, and states the caution beneath it. Also: - `modelTools` takes a `judge`, so an action guarded by a judgment is offerable to an agent rather than withheld. The judge goes to the derivation rather than to each call: the same object has to answer `runnable` and answer the call, or a tool is advertised as callable and refused mid-call. - `kcmd agent tools --judge` lists what such an agent is offered. No model is called -- a judge settles a rule when an action runs, and a listing runs none. - `kcmd action list` names `--judge` in its `run:` hint when a guard is judged, so the suggested command is not one certain to be refused. - The AlloyDB walkthrough becomes step 8, since the outcomes it used to follow are now step 7, and it runs the credit that lands rather than the Labor Day one. That request is refused at a guard before a transaction opens, so against the second store it would have exercised nothing. Its `action run` example takes `--judge` and credits an amount under the ceiling for the same reason, and both it and the `psql` output beneath it are recaptured from the same live cluster.
The judge is instructed that arguments which do not contain enough to tell mean the rule does not hold, and that the reason says what is missing, so the intended answer to missing evidence is a refusal naming the missing value rather than a silent pass. The previous commit described the opposite as a property of the runtime, in the demo README and in the actions guide. It is an instruction given to a model, and the goodwill memo run is a case where the model did not follow it: a rule it had no evidence for came back held, and a guard that passes prints nothing. The observation and the advice it produced are unchanged. Word a guard in terms of the call's own arguments, and test every judged guard against a case it should refuse.
libei
marked this pull request as ready for review
September 14, 2026 20:04
libei
added a commit
that referenced
this pull request
Sep 15, 2026
…rd (#426) Follows #423, now merged as `e6d34fd`. This branch is rebased onto it and carries the reading-judge feature plus a rewrite of the actions guide. #423 left `CreditWithinOrderTotal` as an expression because a judge handed only the attempted call has no evidence for it: the order's total is on record and the call never states it. This gives the judge the model's own tables, so that rule gets a judged twin too. ```console $ ../../../dist/kcmd action run IssueCredit --judge --judge-reads-store --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late" Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_agent_demo... rules stated in words go to gemini-2.5-flash (us-central1) it may read commerce's tables to settle them the judge reads: SELECT total FROM Orders WHERE order_id = 12346 order: '12346' -> Order 12346 Committed at 2026-09-14T19:12:04.996343Z. ``` Nobody wrote that SQL. The rule names the order's total in words, and the judge was told which tables hold this model's data. Ask for more than the order is worth and the same guard refuses, comparing $20.00 against the $15.00 then on record — under the $25 ceiling, so no threshold rule catches it. ## What the judge is allowed to do **It is shown the model rather than the database.** The entities, tables, columns and dialect in its system instruction come from the selected binding profile, the same source the lookup tools come from, so a column the model does not bind is a column the judge is not told exists. One sentence in `commerce.yaml` produces `SELECT total FROM Orders` under the Spanner profile and `SELECT order_total FROM purchase_order` under the AlloyDB one. **Every statement is checked.** A single read beginning with `SELECT` or `WITH`, checked after comments and string literals are blanked out so that a `;` inside a quoted memo cannot split one statement into two. **Every statement is wrapped**, and the wrap is the part carrying the weight rather than the keyword check. `WITH gone AS (DELETE ... RETURNING *) SELECT * FROM gone` begins with `WITH` and deletes rows. What runs is `SELECT * FROM (...) AS judge_read LIMIT 21`, and PostgreSQL refuses a data-modifying CTE inside a subquery: ```console $ psql "host=$PGHOST dbname=semantic_agent_demo" -c "SELECT * FROM ( WITH gone AS (DELETE FROM order_line WHERE order_id = 12346 RETURNING *) SELECT * FROM gone ) AS judge_read LIMIT 21" ERROR: WITH clause containing a data-modifying statement must be at the top level ``` That ran against the live cluster, and the rows it named were still there afterwards. It matters most on AlloyDB, where the store's query path runs in an implicit transaction and a write reaching it would commit; on Spanner the read path is read-only and could not write whatever was sent. **Every read is printed and every result is capped.** At most 20 rows, each cell clipped, and the judge is told when its answer was cut short. A judge that went and looked did something on the caller's behalf that the transcript has to show. ## What it costs Gemini refuses `tools` and `responseSchema` in one request, so the exchange is two-phase: a read loop with function declarations and no schema, then a verdict call with the schema and no tools. A guard that reads nothing costs two model calls rather than one, and each round of reading adds another, up to four reads. The demo's committed run put four guards to Gemini, one of which read once, for nine calls. ## Verified against real instances Every transcript the README pastes is a capture. On Spanner: the $3 credit above, the $20 refusal, the same call unsettled when the judge cannot read, and the whole action refused when there is no judge at all. On AlloyDB: both outcomes again in the other dialect against the other tables, plus the wrapped CTE refusal above. AlloyDB returns rows without column names, so the judge is handed values and told nothing about which column each came from — it settled the rule anyway, because the statement it wrote selected one column and it knew which one it asked for. ## What this does not fix A reading judge inherits the timing every guard has. Guards settle before the transaction opens, so it sees committed state, and two credits racing each other can each read a total that neither will leave behind. That is the strongest argument for keeping the arithmetic twin, which can eventually run inside the write. `OrderTotalMatchesLineItems` stays an expression for the same reason and a stronger one: it constrains the state the write leaves behind, so no judge settles it however much it can read. A rule worded to read a memo column would be reading text a customer or an agent wrote, inside the prompt that decides whether a write proceeds. The judge is told to compose every statement itself and never to treat a value it read back as an instruction, results are capped and clipped, and every statement is printed. None of that is a guarantee, and the README says so. ## Also - `--judge-reads-store` says what a judge may do rather than hiring one, so it needs `--judge` and is an error alone. - The actions guide gains `A guard that reads a row`; the reference gains the flag. Both drop the stale claim that the only store an action runs against is Spanner. - The three expressions stay declared and unreferenced. ## The actions guide, rewritten Nine commits at the end of the branch rewrite `docs/semantic-model/actions.md`. The guide runs to 1,553 lines and had settled into a whitepaper register: third person throughout, mechanics stated before motive, and headings that named objects rather than the job a reader came to do. All 44 prose passages are rewritten. Your model rather than a model, the problem before the YAML that solves it, sentence-case headings under an 85-character cap on H2 and 50 on H3, captions on all eight visuals, and no nested lists. Three headings changed and the file's only H4 became an H3; nothing in the repo links to any of them. Four sentence-level sweeps ran over it, each fixing an instance and then the class: garden-path openings, counts announced without their members, sentences whose subject is a reduced relative, and paragraphs that grant a permission before naming the situation that makes a reader want it. The last of those is worth calling out because it survived the first pass. "You can also write no executor anywhere" tells you what the format allows and not why anybody would want it, so the option lands as trivia. Three paragraphs opened that way and each now opens on the case instead. The rewrite is prose-only by construction: the file is split into alternating prose and fenced segments and only the prose is replaced, so all 564 lines of its 43 fenced blocks pass through byte-identical. Three exceptions are deliberate, all inside hand-drawn illustrations, where "exactly one" became "one kind only" and "a single row" with the column alignment preserved. Those keep the constraint they state. The two remaining uses sit inside captured `kcmd agent tools` output and are left verbatim, because captured output is a recording. No prose line exceeds 80 columns. `npm test` is unchanged at 5 suites, 0 fail, 1106 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto
main, which now carries #422 (the judge), #424 (AlloyDB as a second store) and #425 (the AlloyDB leg run for real). One commit,28f69cb; the diff is this change alone.The commerce demo declared three constraints as arithmetic and guarded on none of them, because nothing evaluates an expression yet. It now declares three more as sentences a judge settles, guards on those, and reaches all three of
on_violation's outcomes against a live store.What the demo shows
CreditUnderReviewThresholdWithJudgeescalate— refused, and the refusal says an approver could allow itCreditMemoNamesAServiceFailurewarn— written, with the warning in the same answerCreditIsNotSplitToAvoidReviewreject— refused outrightThe three expressions stay declared and unreferenced, as asked. Two of them compare the call against stored rows and stay expressions, for two different reasons --
CreditWithinOrderTotalbecause the judge this demo hires makes one model call with no tools, andOrderTotalMatchesLineItemsbecause it constrains the state after the write, which nothing settled before the transaction can see. The third has a judged twin, marked temporary: once arithmetic can be settled by a query, settling it by a model call is paying for nondeterminism. That twin is named for the expression it duplicates with how it is settled on the end,CreditUnderReviewThresholdandCreditUnderReviewThresholdWithJudge, since the two are one rule down to the description and the suffix says which of the pair goes away.Code
modelToolstakes ajudge, so an action guarded by a judgment is offerable rather than withheld. The judge goes to the derivation rather than to each call — the same object has to answerrunnableand answer the call, or a tool is advertised as callable and refused mid-call.kcmd agent tools --judgelists what such an agent is offered. No model is called: a judge settles a rule when an action runs, and a listing runs none.kcmd action listnames--judgein itsrun:hint when a guard is judged.The caution the docs now carry
The judge is instructed that arguments which do not contain enough to tell mean the rule does not hold, and that the reason says what is missing, so missing evidence is meant to refuse a call rather than pass it. That is an instruction to a model rather than a property of the runtime. An earlier wording of the memo rule on this branch is a case where the model did not follow it: a rule it had no evidence for came back held, the write went through, and no warning, log line or loader diagnostic said anything. The only way to know a judged guard works is to run a case it should refuse.
A second overclaim of the same shape was caught in review and is also gone: the docs said a rule comparing the call against stored rows has no judgeable wording at all. That is true of
GeminiJudge, which makes a single model call with no tools, and not ofJudge, which is an interface an implementation can satisfy by querying the store -- a reviewing agent that looks before it answers is how other systems settle exactly that rule. What survives is narrower and is about timing rather than visibility: a guard is settled before the transaction opens, so no judge, however much it can read, can be shown the state the write would produce.An earlier draft of this branch claimed something stronger — that a judgment naming a stored field cannot fire. The actions guide's own worked example disproves it: a
LineItem.memojudgment fails a call correctly there. Wording style is a risk factor rather than a law, and the documents say so at that strength.Also in here
The AlloyDB walkthrough from #424 becomes step 8, since the outcomes it used to follow are now step 7, and it runs the credit that lands rather than the Labor Day request. That request is refused at a guard before a transaction opens, so against the second store it would have exercised nothing. Step 8's
action runexample needed the same treatment for the same reason: it now passes--judgeand credits $20 rather than $30, and it and thepsqloutput beneath it are recaptured from the AlloyDB cluster #425 used.Verified
npm run compileclean,npm test1067 pass / 0 fail,demo/semantic-model/agent && npm run checkclean.Against a real Spanner instance, on this commit: all four agent scenarios, both
kcmd action runpaths (with--judgeit commits, without it refuses and names the two non-advisory guards), and everykcmdlisting on the demo page diffed against its live run. The store was reset to seed state afterwards.Two things worth knowing, both observed rather than inferred:
@google/adkdoes not install against the internal registry on the machine with cluster access.kcmd action runcovers everything under it, and the page says so.