diff --git a/toolbox/mdcode/demo/semantic-model/agent/README.md b/toolbox/mdcode/demo/semantic-model/agent/README.md index e6d3376a..ce41cef4 100644 --- a/toolbox/mdcode/demo/semantic-model/agent/README.md +++ b/toolbox/mdcode/demo/semantic-model/agent/README.md @@ -1,16 +1,17 @@ # Build an agent that acts on a semantic model This is a recipe. Follow it and you get an agent that takes a natural language -request and changes a row in an operational store. The point of the recipe is -how little of that turns out to be agent work. +request, checks it against the policy the model states, and changes a row in an +operational store — or declines to, and says which rule stopped it. The point of +the recipe is how little of that turns out to be agent work. -The whole agent is one file, `agent.ts`, 59 lines of code. Not one of them -mentions credits, orders, customers, tables or SQL. The file does four things: -it creates the runtime, derives the tools, adapts them to the framework, and -runs. Everything that knows what business this is lives in the model, and -outlives the agent. +The whole agent is one file, `agent.ts`, 66 lines of code. Not one of them +mentions credits, orders, customers, tables, SQL or a dollar threshold. The file +does five things: it creates the runtime, hires a judge, derives the tools, +adapts them to the framework, and runs. Everything that knows what business this +is lives in the model, and outlives the agent. -They do not mention a database either. [Step 7](#7-run-the-same-agent-against-alloydb) +They do not mention a database either. [Step 8](#8-run-the-same-agent-against-alloydb) runs the identical file against AlloyDB instead of Spanner -- different tables, a different column name, a different SQL dialect -- by setting one environment variable. @@ -22,23 +23,28 @@ actions, `gcloud` for the store, ADK for the agent. The business is a small ecommerce operation: customers, their orders, and the lines that make up an order. There is one thing you can do to it: credit a -customer against an order. Three policy rules say when that is allowed. +customer against an order. Policy says when that is allowed, and the model +states it as six constraints — three as arithmetic a query settles, three as +sentences a language model settles. The action names the second three in +`guards`, which is what puts them between the agent and the write. [Step 6](#6-run-it) runs the request a support desk gets every day. A customer was charged for shipping that was supposed to be free, and someone inside the company has to find the order and put the money back. The request names no identifier: it gives a customer's name, a holiday, and a description of what -went wrong. +went wrong. It also happens to be over the self-service ceiling, so the interest +is in what the agent does when it is told no. That is the shape worth testing. A request that already speaks in order ids and line types proves only that the tools can be called. ## Before you start -You need a cloud project and application-default credentials, which serve both -the store and Gemini. Steps 1 to 6 bind the model to Spanner, so the project -needs a Spanner instance. Step 7 does the same against AlloyDB and says what it -needs there; you can stop after step 6 and have a working agent. +You need a cloud project and application-default credentials, which serve the +store, the agent's model, and the judge. Steps 1 to 7 bind the model to Spanner, +so the project needs a Spanner instance. Step 8 does the same against AlloyDB +and says what it needs there; you can stop after step 7 and have a working +agent. ```bash gcloud auth application-default login @@ -59,9 +65,9 @@ The model is two files, split so that nothing physical appears in the logical one. `catalog/EntryGroups/commerce_demo/commerce.yaml` says what the business is: -three entities, the relationships between them, one action, three rules. It -names no table, no column and no SQL. The same file would serve if the orders -lived in another store. +three entities, the relationships between them, one action, six rules. It names +no table, no column and no SQL. The same file would serve if the orders lived in +another store. `catalog/EntryGroups/commerce_demo/commerce.profiles/spanner.yaml` says where the business lives: a table per entity, a column per field, and the two @@ -83,7 +89,7 @@ write there, and the agent bills Gemini to the same project unless `GOOGLE_CLOUD_PROJECT` is already set. There is nothing else to keep in step. There is a second profile next to it, `alloydb.yaml`, pointing the same model -at a PostgreSQL database instead. It is not used until [step 7](#7-run-the-same-agent-against-alloydb); +at a PostgreSQL database instead. It is not used until [step 8](#8-run-the-same-agent-against-alloydb); it is mentioned here because it is the reason the split is drawn where it is. The two profiles disagree about tables, about one column name and about SQL dialect, and `commerce.yaml` is the same bytes under both. @@ -92,6 +98,99 @@ The files sit in a `kcmd` workspace (`catalog.yaml` scopes it and names `spanner` as the default profile), so the CLI and the agent read the same files rather than copies that can drift. +### Write the policy down twice + +A constraint carries one of two bodies. An `expression` is arithmetic a query +settles; a `judgment` is a sentence a language model settles. `commerce.yaml` +states the credit policy both ways, and the difference is not stylistic — it is +about what each kind can see. + +```yaml + - name: CreditUnderReviewThreshold + expression: amount <= 25 + on_violation: escalate + + - name: CreditUnderReviewThresholdWithJudge + judgment: >- + The credit amount requested must not exceed 25 dollars, which is the + self-service ceiling for this desk. Read the amount as dollars. + on_violation: escalate +``` + +Those two say the same thing. Only one of them runs today, because this runtime +has a judge and does not yet have an expression evaluator — so the action guards +on the judged one, and the expression sits there declared and referenced by +nothing. When the evaluator lands, the judged twin is the one to delete: +arithmetic a query settles costs no model call and cannot answer two identical +calls differently. + +The other two expressions have no twin, and they have no twin for two different +reasons. + +`CreditWithinOrderTotal` compares the requested amount against `Order.total`, +which is a stored value. The judge this demo hires cannot settle that, because +`GeminiJudge` is handed the attempted call and makes one model call with no +tools attached. That is a property of this implementation rather than of judges: +`Judge` is an interface, and one built over a store connection could read the +order and answer. What argues against writing that one here is cost and +repeatability — a query settles the comparison for nothing and returns the same +answer twice — plus a subtlety about when it would read, covered under +[what is not wired up yet](#what-is-not-wired-up-yet). + +`OrderTotalMatchesLineItems` is the harder case. It holds that an order's total +equals the sum of its lines, which is a statement about the state a write leaves +behind. Guards settle before the transaction opens, so nothing evaluated there +can settle it — not a judge with a database connection, and not an expression +either. It belongs inside the transaction or in the schema. + +Two rules exist *only* because a judge exists, and both are about what the caller +said rather than about what is stored: + +```yaml + - name: CreditMemoNamesAServiceFailure + judgment: >- + The memo argument of this call must name a specific thing that went + wrong on the order: a late delivery, a damaged item, a shipping + charge applied in error. A memo saying only that the customer asked, + or that the credit is goodwill, or giving no reason at all, names no + failure and does not satisfy this rule. + on_violation: warn + + - name: CreditIsNotSplitToAvoidReview + judgment: >- + The credit requested must be the whole of what this order is owed, + not one piece of a larger amount divided to stay under the 25-dollar + self-service ceiling. ... + on_violation: reject +``` + +No expression states either one. "Names a specific service failure" is not a +comparison, and neither is "is not a piece of a larger amount". Between the three +judged rules the demo covers all three consequences `on_violation` can carry: +`escalate` holds the write for a person, `warn` lets it through and reports, and +`reject` refuses outright. + +One wording change is worth reporting, because the failure it fixed is silent. +An earlier draft of the memo rule opened `LineItem.memo must name a specific +thing that went wrong`, and the goodwill memo below committed with no warning at +all: the judge held the rule. Changing those four words to `The memo argument of +this call` made it fire on the first retry, and it has fired on every run since. + +The likely reason is that `LineItem.memo` is a stored field and the judge reads +no stored data, so the rule asks about evidence the judge does not have. That is +a reason to suspect such a wording rather than a rule about it — the same field +name settles a judgment correctly in +[the actions guide](../../../docs/semantic-model/actions.md#a-guard-settled-in-words). + +What generalises is the failure mode: a judged guard can pass a call it should +have refused, and nothing says so. The judge is told that a rule it has not been +given enough to tell about does not hold, so missing evidence is meant to come +back as a refusal naming what is missing. That is an instruction to a model +rather than a property of the runtime. The goodwill memo is a case where the +model did not follow it: the rule came back held, and a guard that passes prints +nothing. Word a guard in terms of the call's own arguments, and test it against +a case it should refuse. + ### Put the persona in the model too `commerce.yaml` carries a model-level `ai_context`: @@ -116,6 +215,19 @@ Only what is specific to *this business* belongs there. How to use a lookup, and what to do when a write is refused, belong to the derived tools rather than to commerce, so the derivation supplies them and the model does not repeat them. +The action carries its own `ai_context` for what a caller has to supply, and one +clause in it is there because of a judged rule: + +```yaml + ... and, if this credit is part of a larger amount owed, say that + and give the total. A rule reads the memo, so a fact left out of it + is a fact the rule cannot weigh. +``` + +A judged rule is settled from the call's own arguments, so a fact the caller +leaves out of the memo is a fact no rule can weigh. Telling the caller which +facts a rule reads is therefore part of describing the action. + The persona also settles which of two agents this is. An internal one reads and writes every customer's orders; a customer-facing one would see only its own caller's. Only the internal one is built here. @@ -190,13 +302,16 @@ Model 'commerce' (commerce_demo), profile 'spanner': IssueCredit: Credit a customer against one order -- a late delivery, a coupon, a shipping charge applied in error. The credit is added as a negative line and the order total is recomputed from the lines. parameters: order (Order, reference), amount (Decimal), memo (String) executor: sql + guards: CreditUnderReviewThresholdWithJudge, CreditMemoNamesAServiceFailure, CreditIsNotSplitToAvoidReview affects: LineItem (create), Order (modify) - run: kcmd action run IssueCredit --arg order= --arg amount= --arg memo= + run: kcmd action run IssueCredit --judge --arg order= --arg amount= --arg memo= ``` -Two warnings print above this, one for `CreditWithinOrderTotal` and one for -`CreditUnderReviewThreshold`. They are the model reporting on itself, and they -differ only in the constraint they name: +The `run:` line names `--judge` because the guards are judged, and a suggested +command certain to be refused is worse than no suggestion. + +Three warnings print above this. Two of them name an expression that reads an +argument of `IssueCredit` and report that nothing references it: ``` Warning: [commerce] model 'commerce': constraint 'CreditWithinOrderTotal' reads 'amount', @@ -204,18 +319,50 @@ a parameter of action 'IssueCredit', but 'IssueCredit' does not list 'CreditWith in guards. A constraint over an action's parameters is checked only as a guard of that action. ``` -That is accurate and deliberate — see [What is not wired up -yet](#what-is-not-wired-up-yet). +That is accurate and deliberate: nothing evaluates an expression yet, so naming +one in `guards` would make the action unrunnable rather than checked. Only +constraints that read an action's parameters get this warning, which is why +`OrderTotalMatchesLineItems` is silent — it reads no argument at all. + +The third is about what *is* referenced: + +``` +Warning: [commerce] model 'commerce': every constraint action 'IssueCredit' names in guards +is judged, so the action has no deterministic gate. Every gate costs a model call, and none +can lower to a store-level check. +``` + +That is the honest price of this configuration, and the demo pays it on purpose. +Three model calls per attempted write is what gating entirely on judgment costs, +and once an expression evaluator exists the threshold rule stops costing one. The action runs from the command line before any agent exists: ```console -$ ../../../dist/kcmd action run IssueCredit --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late" +$ ../../../dist/kcmd action run IssueCredit --judge --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late" Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_agent_demo... + rules stated in words go to gemini-2.5-flash (us-central1) order: '12346' -> Order 12346 -Committed at 2026-09-12T20:46:28.033336Z. +Committed at 2026-09-14T16:10:09.128896Z. ``` +Three judgments were put to Gemini and all three held, so the write went +through. Drop the flag and the same call is refused, because the runtime will not +apply a write the model says must be checked when it has nothing to check with: + +```console +$ ../../../dist/kcmd action run IssueCredit --arg order=12346 --arg amount=3.00 --arg memo="Coupon applied late" +Running 'IssueCredit' on projects/my-project/instances/my-instance/databases/semantic_agent_demo... +Error: Action 'IssueCredit' is guarded by 'CreditUnderReviewThresholdWithJudge' and +'CreditIsNotSplitToAvoidReview', which are settled by judgment rather than by an expression, +and this runtime was given no judge to ask. Running it would apply a write the model says +must be checked first, so it is refused rather than run unchecked. +``` + +The advisory rule is absent from that list, and belongs absent: a `warn` rule +never stops a call, so a missing judge cannot make it stop one. Refusing on its +behalf would make a model that states advisory rules permanently unrunnable. + `order=12346` was text; the runtime resolved it to a row and says which one. The total moved from $18.00 to $15.00 with nobody doing arithmetic, because the second statement recomputes it from the lines. However the action is called, the @@ -235,27 +382,72 @@ order_id placed_on total ## 4. Look at the tools before writing the agent `kcmd agent tools` prints exactly what an agent will be handed, before there is -an API key, a language model, or a line of agent code: +an API key, a language model, or a line of agent code. Ask it without a judge +first, because what an agent is offered depends on what it is holding: ```console $ ../../../dist/kcmd agent tools Model 'commerce' (commerce_demo), profile 'spanner': store: my-project/my-instance/semantic_agent_demo - action issue_credit (IssueCredit) + action issue_credit (IssueCredit) [NOT RUNNABLE] Credit a customer against one order -- a late delivery, a coupon, a shipping charge applied in error. The credit is added as a negative line and the order total is recomputed from the lines. Give the order as its number, the amount in dollars, and a memo saying why. Look the order up first if you were given a customer name rather - than a number: an Order is identified by its key alone. + than a number: an Order is identified by its key alone. Say in the memo + what actually went wrong on the order -- a late delivery, a damaged item, + a shipping charge applied in error -- and, if this credit is part of a + larger amount owed, say that and give the total. A rule reads the memo, + so a fact left out of it is a fact the rule cannot weigh. + + This call is gated by CreditUnderReviewThresholdWithJudge and + CreditIsNotSplitToAvoidReview. + + Calling this will not work: Action 'IssueCredit' is guarded by + 'CreditUnderReviewThresholdWithJudge' and + 'CreditIsNotSplitToAvoidReview', which are settled by judgment rather + than by an expression, and this runtime was given no judge to ask. + Running it would apply a write the model says must be checked first, so + it is refused rather than run unchecked. Report that rather than + retrying. order: string -- Which Order this applies to. Give its key, or text that identifies exactly one; the call fails when nothing matches or more than one does. amount: number -- The amount, as a decimal number. memo: string -- The memo, as text. +``` +Pass `--judge` and the same action is offerable, with the same description and +the same parameters, minus the refusal: + +```console +$ ../../../dist/kcmd agent tools --judge +Rules stated in words go to gemini-2.5-flash (us-central1). +Model 'commerce' (commerce_demo), profile 'spanner': + store: my-project/my-instance/semantic_agent_demo + + action issue_credit (IssueCredit) + ... + This call is gated by CreditUnderReviewThresholdWithJudge and + CreditIsNotSplitToAvoidReview. + order: string -- Which Order this applies to. Give its key, or text that + identifies exactly one; the call fails when nothing matches or more + than one does. + amount: number -- The amount, as a decimal number. + memo: string -- The memo, as text. +``` + +Neither run calls a model. The derivation reports what the runtime *would* do +with the judge it holds, and finding that out costs nothing: a judge settles a +rule when an action runs rather than when a listing is printed. If it were +otherwise, reading this listing would bill you per guarded action. + +The rest of the listing is the same either way: + +```console lookup find_customer (Customer) Look up Customer records. @@ -307,14 +499,24 @@ Model 'commerce' (commerce_demo), profile 'spanner': compute a total or a balance yourself; the tools do that. When a tool reports that a write did not happen, read the reason it gives and repeat it plainly; if it says a person has to decide, say so and stop, because - you cannot approve it yourself. Finish by saying what you changed. + you cannot approve it yourself. When a write did happen and the tool + returns warnings, the change landed and a rule still went unmet or + unchecked: report both, because nobody else will. Finish by saying what + you changed. ``` Every line of that came out of the model. The action's description and its `ai_context.instructions` became the tool description; its typed parameters became typed tool parameters; each entity's description and bound fields became -a lookup and its filters. The instruction is the model's persona followed by the -tool contract the derivation itself defines. +a lookup and its filters; the guards it names became the line that says what +gates the call. The instruction is the model's persona followed by the tool +contract the derivation itself defines. + +Two sentences of that contract exist because a guard can now fire. "If it says a +person has to decide, say so and stop" is what `escalate` needs from a caller. +The sentence about warnings is what `warn` needs: a write that landed while a +rule went unmet is the one case where reporting the outcome alone would be a lie +by omission. Look at `type: string -- item, tax, fee, or credit.` in particular. That line is the field's description in `commerce.yaml`, and it is the only place a caller @@ -331,7 +533,7 @@ enough, write a query rather than widening the tool. ## 5. Write the agent -Here is the whole agent: the four steps, with the framework's own API doing the +Here is the whole agent: the five steps, with the framework's own API doing the work: ```ts @@ -342,14 +544,22 @@ if ('error' in runtimes) throw new Error(runtimes.error); const [runtime] = runtimes; if (!runtime.store) throw new Error(runtime.storeError); -// 2. Derive what the model offers, and keep what this binding can serve -- -// what `kcmd agent tools` just printed. -const {callable, withheld, instruction} = callableTools(modelTools({runtime})); +// 2. Hire something that can settle a rule stated in words. +const judge = geminiJudge({ + project: process.env.GOOGLE_CLOUD_PROJECT, + location: process.env.GOOGLE_CLOUD_LOCATION, + model: process.env.DEMO_JUDGE_MODEL, +}); + +// 3. Derive what the model offers, and keep what this binding can serve -- +// what `kcmd agent tools --judge` just printed. +const {callable, withheld, instruction} = + callableTools(modelTools({runtime, judge})); for (const tool of withheld) { console.error(`(withheld) ${tool.name}: ${tool.unavailable}`); } -// 3. Adapt each one to ADK. +// 4. Adapt each one to ADK. const tools = callable.map( tool => new FunctionTool({ name: tool.name, @@ -364,7 +574,7 @@ const tools = callable.map( execute: (args: unknown) => tool.invoke(args as Record), })); -// 4. Run. +// 5. Run. const agent = new LlmAgent({ name: 'model_agent', model: 'gemini-2.5-flash', @@ -373,13 +583,32 @@ const agent = new LlmAgent({ }); ``` -Step 3 is the only part shaped by ADK rather than by the model, and all it does +Step 2 is three lines and no policy. The judge is a capability, like the database +connection: it can settle a rule stated in words, and *which* rules it is asked +are the ones the model names in `guards`. Nothing here says 25 dollars, and +nothing here decides that a memo has to name a failure. Change the ceiling in +`commerce.yaml` and this file does not move. + +It may well be the same Gemini model the agent runs on, and it is not the same +call. The judge is asked one rule about one set of attempted arguments, under a +system instruction of its own, outside the agent's conversation. So the agent +never sees the exchange: there is nothing in the transcript for it to argue with, +and no turn in which it can talk the gate round. + +The judge goes to the derivation rather than to each invocation, which is the +one non-obvious line in step 3. Whether a guarded action can be offered *at all* +depends on holding a judge, so the same object has to answer `runnable` and +answer the call. A tool derived with a judge and then invoked without one would +be advertised as callable and refused mid-call, which spends the agent's turn and +teaches it nothing. + +Step 4 is the only part shaped by ADK rather than by the model, and all it does is rename fields: a derived parameter already carries a JSON type and a description, which is the whole of a function declaration. ADK takes a plain schema object, so there is no schema library in here and nothing to keep in step with the derivation's types. -Step 2 is one call because every adapter has to make the same split over +Step 3 is one call because every adapter has to make the same split over `[...lookups, ...actions]`. A tool the runtime cannot run is still worth naming, yet offering it as callable spends a turn on a call that cannot succeed. `callableTools` makes that split in the library, and returning `withheld` @@ -409,56 +638,140 @@ the model was missing it, and the fix belongs in the model. The request below is phrased the way the person with the problem would phrase it. ADK logs an `INFO` line per model call; the transcripts leave those out, and -the two load-time warnings from +the three load-time warnings from [step 3](#3-check-what-the-model-declares) still print first. ```console $ bun agent.ts "Find the order for Morgan Ellis (morgan.ellis@example.com) that was placed on Labor Day. It was supposed to get free shipping but we had a glitch and the customer got charged. Please issue them a credit to offset the charge." - -> find_customer({"name":"Morgan Ellis","email":"morgan.ellis@example.com"}) + -> find_customer({"email":"morgan.ellis@example.com","name":"Morgan Ellis"}) <- {"entity":"Customer","fields":["customerId","name","email"],"rows":[["1","Morgan Ellis","morgan.ellis@example.com"]],"truncated":false} -> find_order({"customerId":1,"placedOn":"2026-09-07"}) <- {"entity":"Order","fields":["orderId","customerId","placedOn","total","status"],"rows":[["12345","1","2026-09-07","165.85","OPEN"]],"truncated":false} + -> find_line_item({"orderId":12345,"memo":"Shipping Charge","type":"fee"}) + <- {"entity":"LineItem","fields":["lineItemId","orderId","type","amount","memo"],"rows":[],"truncated":false} -> find_line_item({"orderId":12345,"type":"fee"}) <- {"entity":"LineItem","fields":["lineItemId","orderId","type","amount","memo"],"rows":[["li-12345-3","12345","fee","30","Shipping"]],"truncated":false} - -> issue_credit({"memo":"Credit for charged shipping fee on Labor Day order","order":"12345","amount":30}) - <- {"applied":true,"actedOn":{"order":["12345"]},"committedAt":"2026-09-13T18:16:11.020320Z"} -I issued a credit of 30.00 to order 12345 for Morgan Ellis, to offset the shipping charge. + -> issue_credit({"memo":"Credit for shipping charge applied in error on Labor Day order (original charge was 30.00)","amount":30,"order":"12345"}) + <- {"applied":false,"reason":"Action 'IssueCredit' is guarded by 'CreditUnderReviewThresholdWithJudge' (\"The credit amount requested must not exceed 25 dollars, which is the self-service ceiling for this desk. Read the amount as dollars.\"), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: The credit amount requested is 30 dollars, which exceeds the 25 dollar self-service ceiling for this desk. The model marks this rule 'escalate', so an approver may allow it; nothing here can. A credit over $25 is above the self-service ceiling. A supervisor decides it. No transaction was opened, so nothing was written.","whatToDo":"Correct what the reason describes, or report it. Nothing was written."} +The credit of $30 for the shipping charge was not applied. The reason given is that the credit amount exceeds the self-service ceiling of $25 for this desk. A supervisor needs to approve credits over $25. ``` -The agent made four tool calls, and the request named none of the things they +The agent made five tool calls, and the request named none of the things they took. A name and an email became a customer id; a holiday became a date; "the customer got charged" became a line of type `fee`; the amount to credit was read -off that line rather than supplied. Nowhere does the agent decide *how* to issue -a credit — that is one call, and the model owns what it does. +off that line rather than supplied. One of the five guessed at the wording of the +memo, came back with no rows, and was asked again by type alone. Nowhere does the +agent decide *how* to issue a credit — that is one call, and the model owns what +it does. -Check it with SQL: +Then it was told no, by a rule it had no part in writing, and it stopped. Order +12345 is untouched: ```console $ gcloud spanner databases execute-sql semantic_agent_demo \ --instance=my-instance --project=my-project \ - --sql="SELECT line_item_id, order_id, type, amount, memo FROM LineItem WHERE order_id = 12345 ORDER BY type" -line_item_id order_id type amount memo -17d5bf3c-624a-48f6-9556-8243df7c71b7 12345 credit -30 Credit for charged shipping fee on Labor Day order -li-12345-3 12345 fee 30 Shipping -li-12345-1 12345 item 89.99 Cast iron skillet -li-12345-2 12345 item 34.5 Enamel saucepan -li-12345-4 12345 tax 11.36 Sales tax -``` - -The credit exactly offsets the fee, which is what the request asked for. The -credit line's key was generated by the runtime, because the action's `affects` -says the call creates a `LineItem`; the profile's INSERT names it as -`@newLineItemKey`. Order 12345 is $135.85 afterwards, down from $165.85, and -nobody subtracted: the action's second statement re-summed the lines. - -What the agent is *not* doing is the more interesting half. It never writes SQL: -the statements are in the binding profile, authored once and reviewed there. It -cannot widen its own reach, because the tools it has are the ones the model -declares. It cannot compute a total — the action does that, in the same -transaction as the write. And it carries no opinion of its own about how to -treat a customer, because that opinion is in `commerce.yaml`. - -## 7. Run the same agent against AlloyDB + --sql="SELECT order_id, placed_on, total FROM Orders ORDER BY order_id" +order_id placed_on total +12345 2026-09-07 165.85 +12346 2026-08-20 18 +12347 2026-09-02 200 +``` + +Four things in that refusal are worth separating, because each comes from a +different place. The rule's *name and words* are quoted from `commerce.yaml`, so +an operator reading the transcript can go and find the rule. The *reason* is the +judge's, and it is about this call rather than about the rule in general. "The +model marks this rule `escalate`, so an approver may allow it; nothing here can" +is the runtime distinguishing a rule that ends the matter from one a person could +still overrule. And "No transaction was opened, so nothing was written" is a +statement about timing: the guards are settled before the transaction opens, so a +refusal is not a rollback and there is nothing to reconcile. + +The agent's own sentence adds nothing and invents nothing. That is the derived +instruction working — "read the reason it gives and repeat it plainly; if it says +a person has to decide, say so and stop". + +## 7. The other two outcomes + +`on_violation` carries three consequences, and the demo reaches all three. A +credit under the ceiling with a memo that names the failure passes every guard +and lands: + +```console +$ bun agent.ts "Dana Reyes says the stand mixer on order 12347 turned up with a dented casing. She's keeping it. Give her 20 dollars back." + -> issue_credit({"memo":"stand mixer with dented casing","order":"12347","amount":20}) + <- {"applied":true,"actedOn":{"order":["12347"]},"committedAt":"2026-09-14T17:13:48.303620Z"} +I issued a credit for $20 to order 12347 due to a stand mixer with a dented casing. +``` + +Three judgments were asked and all three held, so the write went through and +order 12347 is $180.00. Note that the agent called `issue_credit` directly: the +request named the order, so there was nothing to look up. + +A credit with no failure behind it trips the advisory rule, and `warn` means the +write lands anyway and says so: + +```console +$ bun agent.ts "Dana Reyes is asking for a 5 dollar credit on order 12347. She didn't say anything was wrong with it, she just asked for one as a gesture. Please put it through." + -> issue_credit({"order":"12347","memo":"Customer requested a credit as a gesture; no specific issue reported with the order.","amount":5}) + <- {"applied":true,"actedOn":{"order":["12347"]},"committedAt":"2026-09-14T17:19:09.854797Z","warnings":["'CreditMemoNamesAServiceFailure' (\"The memo argument of this call must name a specific thing that went wrong on the order: ...\") is advisory, and gemini-2.5-flash (us-central1) judged that it does not hold for this call: Your memo \"Customer requested a credit as a gesture; no specific issue reported with the order.\" does not name a specific thing that went wrong on the order."]} +The credit was applied to order 12347. However, there was a warning: the memo "Customer requested a credit as a gesture; no specific issue reported with the order." does not name a specific thing that went wrong on the order, which is usually required. I have applied a $5 credit to order 12347. +``` + +`applied: true` and a warning in the same answer is the case that is easy to +report badly. The write happened; a rule the model states went unmet; both are +true, and an agent that mentioned only the first would leave nobody to notice the +second. The derived instruction asks for both, and the agent gave both without +being told anything about credits. + +This is the one request in the four that does not always reach the tool. In +about a quarter of the runs the agent reads the advisory rule out of the tool +description, decides a memo naming no failure would be improper, and declines +before calling anything. That is the agent's judgment rather than the model's: +`warn` is the model saying this write should land and be flagged, and an agent +that refuses instead has quietly made an advisory rule a blocking one. Nothing +in the runtime stops it, which is worth knowing before an advisory rule is +relied on to let work through. + +A credit split to duck the ceiling is refused outright: + +```console +$ bun agent.ts "Order 12346 for Morgan Ellis is owed 40 dollars for a spatula set that arrived broken. I can only do 25 at a time on this desk, so put through the first 20 now and I'll do the rest tomorrow." + -> issue_credit({"order":12346,"memo":"Credit for broken spatula set. This is the first of two credits, totaling $40.","amount":20}) + <- {"applied":false,"reason":"Action 'IssueCredit' is guarded by 'CreditIsNotSplitToAvoidReview' (\"The credit requested must be the whole of what this order is owed, not one piece of a larger amount divided to stay under the 25-dollar self-service ceiling. ...\"), and gemini-2.5-flash (us-central1) judged that it does not hold for this call: Your memo states that this is the first of two credits, totaling $40, which indicates that the credit is a piece of a larger amount divided to stay under the 25-dollar self-service ceiling. Raise this as a single credit for the full amount and send it for supervisor review. No transaction was opened, so nothing was written.","whatToDo":"Correct what the reason describes, or report it. Nothing was written."} +The credit was not applied because it violates the 'CreditIsNotSplitToAvoidReview' rule. The system detected that this is part of a larger credit split to avoid the $25 self-service limit. You need to raise the full $40 credit as a single transaction and send it for supervisor review. Nothing was written. +``` + +$20 is under the ceiling and the memo names a real failure, so the first two +guards held. The third caught it, and no arithmetic could have: nothing about +`amount = 20` on order 12346 is out of policy. What is out of policy is the +*intent*, and the only evidence of it is the sentence the caller wrote. + +That run is also the one that shows the limit of judging. The judge sees the +attempted call and nothing else, so this rule fires only if the memo admits the +split — and the first time this demo was run, it did not. The agent wrote a memo +that named the breakage and nothing else, dropping the "first 20 of 40" framing +that was right there in the request, and the credit went through. The judge was +not wrong; it was given laundered evidence. + +The fix went into the model rather than the agent, because the model is where +the rule lives: + +```yaml + ... and, if this credit is part of a larger amount owed, say that + and give the total. A rule reads the memo, so a fact left out of it + is a fact the rule cannot weigh. +``` + +With that clause in the action's `ai_context`, the agent volunteered "This is +the first of two credits, totaling $40" and the rule fired. A judged rule over +caller-supplied text is only as good as the caller's candour. It catches an +honest mistake and deters a casual one; it does not stop a caller who has decided +to get around it. The rule that would hold regardless is an expression over what +is already stored — sum the credits already on this order — and that one is +waiting on the evaluator. + +## 8. Run the same agent against AlloyDB Everything so far ran against Spanner. This step runs the same request against AlloyDB, and the interesting part is the size of the change: one environment @@ -491,7 +804,7 @@ $ diff /tmp/spanner.txt /tmp/alloydb.txt > store: alloydb:my-project/us-central1/my-cluster/my-instance/semantic_agent_demo ``` -Seventy lines of tools, four tool names, every parameter, every description +Eighty-seven lines of tools, four tool names, every parameter, every description and the whole instruction: identical. The two lines that differ are the two that say which deployment this is. That is the claim, and the `diff` is the proof. @@ -611,27 +924,35 @@ exclusive, and a model can be published in one place and run in another. ### Run it `DEMO_PROFILE` picks the profile, the same way `--profile` does for `kcmd`. The -request is the one from [step 6](#6-run-it), word for word: +request is the credit that lands in +[step 7](#7-the-other-two-outcomes), word for word: ```bash # From outside the VPC, point the client at the address psql just used. export ALLOYDB_HOST="$PGHOST" -DEMO_PROFILE=alloydb bun agent.ts "Find the order for Morgan Ellis (morgan.ellis@example.com) that was placed on Labor Day. It was supposed to get free shipping but we had a glitch and the customer got charged. Please issue them a credit to offset the charge." +DEMO_PROFILE=alloydb bun agent.ts "Dana Reyes says the stand mixer on order 12347 turned up with a dented casing. She's keeping it. Give her 20 dollars back." ``` +The Labor Day request from [step 6](#6-run-it) is refused here for the reason it +is refused there, and the refusal is decided before a transaction opens, so it +never reaches the database. Exercising the far side takes a request that passes +every guard. + Check the result the same way, in the other dialect: ```bash psql "host=$PGHOST user=postgres dbname=semantic_agent_demo" -c \ "SELECT line_item_id, order_id, type, amount, memo - FROM order_line WHERE order_id = 12345 ORDER BY type" + FROM order_line WHERE order_id = 12347 ORDER BY type" ``` -A credit line of `-30.00` with a generated key, and `purchase_order.order_total` -down to `135.85` — the same outcome as -[step 6](#6-run-it), reached by different SQL against different tables, from the -same request through the same agent. +What that should show is a credit line of `-20.00` with a generated key and +`purchase_order.order_total` down to `180.00` — the outcome of +[step 7](#7-the-other-two-outcomes), reached by different SQL against different +tables, from the same request through the same agent. The note at the end of +this section says which parts of it have been run against a real cluster and +which have not. ### The same action without the model in the loop @@ -642,13 +963,21 @@ and the commit — so it is worth doing once on a new store before handing the store to a model: ```console -$ ../../../dist/kcmd action run IssueCredit --profile alloydb \ - --arg order=12345 --arg amount=30.00 --arg memo='Shipping charged in error' +$ ../../../dist/kcmd action run IssueCredit --profile alloydb --judge \ + --arg order=12345 --arg amount=20.00 --arg memo='Shipping charged in error' Running 'IssueCredit' on projects/my-project/locations/us-central1/clusters/my-cluster/instances/my-instance/databases/semantic_agent_demo... + rules stated in words go to gemini-2.5-flash (us-central1) order: '12345' -> Order 12345 -Committed at 2026-09-14T17:28:42.476Z. +Committed at 2026-09-14T18:16:59.516Z. ``` +The credit is $20 rather than the $30 that was charged, because $30 is over the +self-service ceiling and would be held for an approver on this store exactly as +it is on Spanner. Dropping `--judge` produces the refusal from +[step 3](#3-check-what-the-model-declares), word for word. Guards are settled +from the attempted call before any transaction opens, so which database is +underneath makes no difference to either outcome. + `order: '12345' -> Order 12345` is the reference being resolved: the argument arrives as text and the runtime reads the key back out of `purchase_order` before either statement runs, so an order that does not exist is refused here @@ -660,7 +989,7 @@ $ psql "host=$PGHOST user=$(gcloud config get-value account) dbname=semantic_age WHERE order_id = 12345 ORDER BY line_item_id" line_item_id | type | amount | memo --------------------------------------+--------+--------+--------------------------- - 3c6edaab-ed0a-4bae-864b-e3214365f852 | credit | -30.00 | Shipping charged in error + 59533395-db79-4c56-8b43-aa5fcc1b43e8 | credit | -20.00 | Shipping charged in error li-12345-1 | item | 89.99 | Cast iron skillet li-12345-2 | item | 34.50 | Enamel saucepan li-12345-3 | fee | 30.00 | Shipping @@ -669,15 +998,16 @@ $ psql "host=$PGHOST user=$(gcloud config get-value account) dbname=semantic_age ``` The UUID is the key the runtime generated, because the action's `affects` says -this call creates a `LineItem`. `order_total` reads `135.85`, recomputed from +this call creates a `LineItem`. `order_total` reads `145.85`, recomputed from those five lines rather than adjusted by the credit amount. > **What on this page is copied from a real run, and what is not.** Every `kcmd` > listing here is, including the `diff` above, the push refusal, and the > `action run` and `psql` output just above -- those two are from a live AlloyDB > cluster, so the connection, the IAM token as the password, the cluster CA, the -> `@name`-to-`$1` rewrite, both statements and the commit are all proven against -> a real database rather than only against unit tests. The **agent** transcript +> `@name`-to-`$1` rewrite, all three judged guards, both statements and the +> commit are all proven against a real database rather than only against unit +> tests. The **agent** transcript > under `alloydb` is not captured: `@google/adk` does not install here, so the > model-in-the-loop leg of this section has been run only under `spanner`. What > that leg adds over `action run` is the model choosing the action and its @@ -690,99 +1020,84 @@ Four files carry the business, and you can list them: | File | What it holds | | --- | --- | -| `catalog/EntryGroups/commerce_demo/commerce.yaml` | the ontology, the action, the three policy rules, the persona | +| `catalog/EntryGroups/commerce_demo/commerce.yaml` | the ontology, the action, the six policy rules, the persona | | `catalog/.../commerce.profiles/spanner.yaml` | tables, columns, the two SQL statements, the deployment target | | `catalog/.../commerce.profiles/alloydb.yaml` | the same four things, for the other database | | `schema.spanner.sql`, `schema.alloydb.sql` | three `CREATE TABLE`s, twice | -| the seed commands in [step 2](#2-create-the-store) and [step 7](#7-run-the-same-agent-against-alloydb) | two customers, three orders, six lines | +| the seed commands in [step 2](#2-create-the-store) and [step 8](#8-run-the-same-agent-against-alloydb) | two customers, three orders, six lines | `agent.ts` is not on that list, and neither is anything under `src/`. Swap those -files for a different business and the same 59 lines run it. That is the claim +files for a different business and the same 66 lines run it. That is the claim this demo makes, and the file list is how you check it. Note which rows doubled and which did not. Moving to a second database added a profile and a schema; it added no entity, no field, no action, no rule and no line of agent code. -The list stayed short because of two decisions. This case wanted a filter over a -coded field, and the fix went into `agent_tools.ts`, where it helps every model, -rather than into a hand-written tool here. It also wanted today's date, and that -went into `agent.ts` because it is a fact about the run rather than about -commerce. Each new use case pushes on the boundary in one of those two +The list stayed short because of three decisions, and each put a fact in a +different place. This case wanted a filter over a coded field, and the fix went +into `agent_tools.ts`, where it helps every model, rather than into a +hand-written tool here. It wanted today's date, and that went into `agent.ts` +because it is a fact about the run rather than about commerce. And it wanted +callers to disclose a split credit, which went into `commerce.yaml` because it +is a fact about this desk's policy. Each new use case pushes on the boundary in +one of those three directions, and the useful question is always which. ## What is not wired up yet -Start with the thing the run above got wrong. The third policy rule says a -credit over $25 is above the self-service ceiling and a supervisor decides it. -The agent wrote $30 and asked nobody. The agent did not disobey: the policy is -written down and never attached to anything. - -`commerce.yaml` declares three constraints, one per policy rule, and references -none of them. A constraint is inert until an action names it in -`guards`, and this runtime does not evaluate constraints yet — so naming one -makes the action *unrunnable* rather than checked. Try it: add -`guards: [CreditUnderReviewThreshold]` to the action and run `kcmd agent tools` -again. - -```console - action issue_credit (IssueCredit) [NOT RUNNABLE] - ... - This call is gated by CreditUnderReviewThreshold. - - Calling this will not work: Action 'IssueCredit' is guarded by - 'CreditUnderReviewThreshold', and this runtime does not evaluate - constraints yet. Running it would apply a write the model says must be - checked first, so it is refused rather than run unchecked. Report that - rather than retrying. - ... -``` - -Refusing is the point. The model says this write must be checked and the checker -is missing, so the write is refused rather than run unchecked. The tool is still -derived, still named and still described, because an action the model declares -should not vanish from what the model offers. `agent.ts` reports it and leaves -it unbound, so the agent has no call to make and nothing to retry. - -Re-run the same request with the guard attached and you get the reading half and -none of the writing half: - -```console -$ bun agent.ts "Find the order for Morgan Ellis (morgan.ellis@example.com) that was placed on Labor Day. ..." -(withheld) issue_credit: Action 'IssueCredit' is guarded by 'CreditUnderReviewThreshold', and this runtime does not evaluate constraints yet. Running it would apply a write the model says must be checked first, so it is refused rather than run unchecked. - -> find_customer({"email":"morgan.ellis@example.com","name":"Morgan Ellis"}) - <- {"entity":"Customer","fields":["customerId","name","email"],"rows":[["1","Morgan Ellis","morgan.ellis@example.com"]],"truncated":false} - -> find_order({"customerId":1,"placedOn":"2026-09-07"}) - <- {"entity":"Order","fields":["orderId","customerId","placedOn","total","status"],"rows":[["12345","1","2026-09-07","165.85","OPEN"]],"truncated":false} - -> find_line_item({"orderId":12345,"type":"fee"}) - <- {"entity":"LineItem","fields":["lineItemId","orderId","type","amount","memo"],"rows":[["li-12345-3","12345","fee","30","Shipping"]],"truncated":false} -The customer Morgan Ellis (ID 1) has an order (ID 12345) placed on 2026-09-07. This order includes a shipping fee of $30.00 (line item ID li-12345-3). - -I cannot issue credits or modify orders. A person will have to decide how to proceed with the credit. -``` - -It still did the work worth doing — found the order, found the charge, named the -amount — and order 12345 is still $165.85. - -So the demo reaches both ends of the policy story, write and refuse, and not the -middle. There are two rungs where there should be three, and the missing one is -"ask a person, then apply it if they say yes". - -The model already says which rule wants that rung. `CreditUnderReviewThreshold` -carries `on_violation: escalate`, and `escalate` means the write is held and an -approver decides. What is missing is not the declaration but anything that acts -on it. Most action frameworks ship a blanket "this action needs confirmation" -flag on the action instead; a rule that escalates only when a predicate is -violated says more, and costs more to honor, because something has to evaluate -the predicate. That evaluator is the next piece of work. The two warnings in -[step 3](#3-check-what-the-model-declares) report the same gap at load time. - -The other missing piece is access control. A business that runs an internal desk -agent usually wants a customer-facing one too, restricted to the caller's own -orders. That needs the caller's identity passed in the tool call and checked -below the agent. Only the internal one is here: the lookups take no caller -identity and there is nothing to scope them by, so the second agent cannot be -built from this model today. +**Nothing evaluates an expression.** This is the big one, and it is why +`commerce.yaml` guards on three judgments rather than on the arithmetic beside +them. `CreditWithinOrderTotal` and `OrderTotalMatchesLineItems` are catalogued +and not enforced, and the loader's first two warnings say so at load time. +`CreditUnderReviewThreshold` is the one with a working twin, which is why that +twin is marked temporary: when arithmetic can be settled by a query, settling it +by a model call is paying for nondeterminism. + +**No judge here reads the store, and one could.** `Judge` is an interface over +"settle this rule for this call"; the implementation this demo uses makes a +single model call with no tools, so it knows only what the caller passed. An +implementation holding a store connection would settle +`CreditWithinOrderTotal`, and a reviewing agent that queries before it answers +is how other systems in this space check exactly that class of rule. Two things +would need saying if one were written here. It would read outside the +transaction, because guards settle before the transaction opens so that a model +call does not hold write locks; two credits racing each other could therefore +each pass and jointly exceed the total. And it would spend a model call and a +query where a query alone suffices. + +**A rule about the result of a write has nowhere to run.** +`OrderTotalMatchesLineItems` is that rule: it constrains the state the write +leaves behind, and guards are settled before the write. Neither the expression +evaluator nor a store-reading judge fixes that, because the problem is when the +check runs rather than what it can see. Such a rule wants to be checked inside +the transaction or declared in the schema, and this runtime offers neither +binding point. + +**Every gate costs a model call.** Three guards means three round trips to Gemini +before a write, serially, and the loader's third warning names the cost. The +runtime stops at the first refusal, so a call that is going to be rejected does +not pay for the rest — but a call that succeeds pays for all three. An expression +evaluator would take the threshold check to zero, and in a store-level form could +push it into the same statement as the write. + +**`escalate` still has nobody to escalate to.** The rung is now visible rather +than missing: the runtime recognises `escalate`, holds the write, and says in the +refusal that an approver could allow it and that nothing here can. What does not +exist is the approver — a queue, a second caller with a different mandate, +anything that could take the held call and say yes. Today `escalate` and `reject` +both stop the write; the difference is only in what the caller is told. + +**A judged rule sees only what the caller wrote.** Covered in +[step 7](#7-the-other-two-outcomes): `CreditIsNotSplitToAvoidReview` fires +because the model tells callers to disclose a split. Nothing verifies that they +did. + +**Access control.** A business that runs an internal desk agent usually wants a +customer-facing one too, restricted to the caller's own orders. That needs the +caller's identity passed in the tool call and checked below the agent. Only the +internal one is here: the lookups take no caller identity and there is nothing to +scope them by, so the second agent cannot be built from this model today. ## Cleaning up @@ -794,7 +1109,7 @@ gcloud spanner databases delete "$DATABASE" \ --instance="$INSTANCE" --project="$PROJECT" ``` -If you also ran step 7, the AlloyDB cluster is the expensive half and goes +If you also ran step 8, the AlloyDB cluster is the expensive half and goes separately. It bills while it exists, so delete it rather than leaving it idle. ```bash diff --git a/toolbox/mdcode/demo/semantic-model/agent/agent.ts b/toolbox/mdcode/demo/semantic-model/agent/agent.ts index bf47d739..bc1c2ff1 100644 --- a/toolbox/mdcode/demo/semantic-model/agent/agent.ts +++ b/toolbox/mdcode/demo/semantic-model/agent/agent.ts @@ -7,15 +7,22 @@ // offset the charge." // // Nothing here mentions credits, orders, database tables or an operations desk. -// Four steps: create the runtime, derive the tools, adapt them to ADK, run. -// Point it at another semantic model and it is another agent, with no edit to -// this file. That is the property it exists to test -- so the moment something -// about this business has to be written here, the model was missing it and the -// fix belongs there. +// Five steps: create the runtime, hire a judge, derive the tools, adapt them to +// ADK, run. Point it at another semantic model and it is another agent, with no +// edit to this file. That is the property it exists to test -- so the moment +// something about this business has to be written here, the model was missing +// it and the fix belongs there. +// +// The judge is a capability rather than a policy: it can settle a rule stated +// in words, and WHICH rules it is asked are the ones the model names in +// `guards`. So supplying one here says nothing about commerce, the same way +// supplying a database connection does not. import {FunctionTool, InMemoryRunner, LlmAgent} from '@google/adk'; import {Type} from '@google/genai'; +import {ApiContext} from '../../../src/libts/gcp/context'; +import {GeminiJudge} from '../../../src/libts/gcp/gemini'; import {callableTools, modelTools} from '../../../src/libts/semantic/runtime/agent_tools'; import {createSemanticRuntimes} from '../../../src/libts/semantic/runtime/runtime'; import {closeStore} from '../../../src/libts/semantic/runtime/store'; @@ -42,15 +49,34 @@ process.env.GOOGLE_GENAI_USE_ENTERPRISE ??= 'true'; process.env.GOOGLE_CLOUD_PROJECT ??= runtime.store.project; process.env.GOOGLE_CLOUD_LOCATION ??= 'us-central1'; -// 2. Derive what the model offers, and keep what this binding can serve. -// `kcmd agent tools` prints all of it before a language model is involved. +// 2. Hire something that can settle a rule stated in words. Pointed at the +// same project the store is in, so the binding profile is still the only +// place that says where any of this runs. +// +// It may well be the same Gemini model the agent runs on, and it is not +// the same call: the judge is asked one rule about one set of attempted +// arguments, under a system instruction of its own, outside the agent's +// conversation. So there is nothing in the transcript for the agent to +// argue with, and no turn in which it can talk the gate round. +const judge = new GeminiJudge(ApiContext.default(), { + project: process.env.GOOGLE_CLOUD_PROJECT, + location: process.env.GOOGLE_CLOUD_LOCATION, + model: process.env.DEMO_JUDGE_MODEL, +}); + +// 3. Derive what the model offers, and keep what this binding can serve. +// `kcmd agent tools --judge` prints all of it before an agent exists. The +// judge goes to the derivation rather than to each call: it is what decides +// whether a guarded action is offerable at all, and a tool offered on the +// strength of a judge and then called without one would be refused +// mid-call. const {callable, withheld, instruction} = - callableTools(modelTools({runtime})); + callableTools(modelTools({runtime, judge})); for (const tool of withheld) { console.error(`(withheld) ${tool.name}: ${tool.unavailable}`); } -// 3. Adapt each one to ADK. A derived parameter already carries a JSON type and +// 4. Adapt each one to ADK. A derived parameter already carries a JSON type and // a description, which is the whole of a function declaration -- so this // changes the shape and none of the content. const tools = callable.map( @@ -69,8 +95,11 @@ const tools = callable.map( execute: (args: unknown) => tool.invoke(args as Record), })); -// 4. Run, printing each call and each answer so the transcript shows which -// tools the agent chose and what the store said back. +// 5. Run, printing each call and each answer so the transcript shows which +// tools the agent chose and what the store said back. A refusal and a +// warning both come back inside a tool answer, so both are already in the +// transcript; what the agent does with them is the derived instruction's +// business rather than this file's. // // The instruction is the model's, not this file's: what this business asks of // anything acting on it is in `ai_context` in commerce.yaml, and how to use a diff --git a/toolbox/mdcode/demo/semantic-model/agent/catalog/EntryGroups/commerce_demo/commerce.yaml b/toolbox/mdcode/demo/semantic-model/agent/catalog/EntryGroups/commerce_demo/commerce.yaml index 2cc5fb91..d6e3adba 100644 --- a/toolbox/mdcode/demo/semantic-model/agent/catalog/EntryGroups/commerce_demo/commerce.yaml +++ b/toolbox/mdcode/demo/semantic-model/agent/catalog/EntryGroups/commerce_demo/commerce.yaml @@ -2,8 +2,8 @@ # # The business is a small ecommerce operation: customers, their orders, and the # lines that make up an order. One thing can be done to it -- credit a customer -# against an order -- and three policy rules say when that is allowed. The -# README runs one request against it, the kind a support desk gets every day: a +# against an order -- and the rules below say when that is allowed. The README +# runs one request against it, the kind a support desk gets every day: a # customer was charged for shipping that should have been free. # # Nothing here says where the data lives, what the columns are called, or what @@ -14,12 +14,12 @@ # This file is the same bytes under both, which is the claim the pair exists to # make. # -# Three constraints are declared and none is referenced. A constraint is inert -# until an action names it in `guards`, and today's runtime evaluates none of -# them -- so naming one here would make the action unrunnable rather than -# checked. They are written down now because the policy exists now; wiring them -# up is the next step, not a forgotten one. Add `guards: [CreditUnderReviewThreshold]` -# to the action and re-run `kcmd agent tools` to see exactly what that costs today. +# The same policy is written twice, once in arithmetic and once in words. A +# constraint carries either an `expression` a query settles or a `judgment` a +# language model settles, and `IssueCredit` guards on the judgments, because +# those are the ones something can answer today. The expressions are declared +# and named by nothing, which makes them inert: a constraint takes effect where +# something references it. version: "0.2.0.dev0/google" @@ -115,12 +115,30 @@ semantic_model: affects: - {concept: LineItem, operation: create, fields: [type, amount, memo]} - {concept: Order, operation: modify, fields: [total]} + # Which rules this call must pass, and the only place a constraint + # takes effect. All three are judgments, so every gate here costs a + # model call -- the loader says so at load time, and it is the price of + # gating on the half of the policy something can settle today. + guards: + - CreditUnderReviewThresholdWithJudge + - CreditMemoNamesAServiceFailure + - CreditIsNotSplitToAvoidReview ai_context: instructions: >- Give the order as its number, the amount in dollars, and a memo saying why. Look the order up first if you were given a customer name rather than a number: an Order is identified by its key alone. + Say in the memo what actually went wrong on the order -- a late + delivery, a damaged item, a shipping charge applied in error -- + and, if this credit is part of a larger amount owed, say that and + give the total. A rule reads the memo, so a fact left out of it is + a fact the rule cannot weigh. + # Rules a query settles. All three are correct, none is referenced, and + # nothing evaluates one yet -- naming one in `guards` would make the action + # unrunnable rather than checked. They stay because they are the rules as + # they should end up: arithmetic a query settles costs no model call and + # cannot answer two identical calls differently. constraints: - name: CreditWithinOrderTotal expression: amount <= Order.total @@ -128,6 +146,13 @@ semantic_model: description: >- A credit cannot exceed the total of the order it credits. Lower the credit amount, or split it across the orders it actually covers. + # No counterpart in words, because the judge this demo hires is handed + # the attempted call and nothing else, and `Order.total` is in the + # store. That is this judge rather than judges in general: `Judge` is + # an interface, and one built over a store connection would settle + # this. What argues for leaving it an expression is that a query + # answers the comparison for nothing and answers it the same way + # twice. - name: CreditUnderReviewThreshold expression: amount <= 25 @@ -135,6 +160,13 @@ semantic_model: description: >- A credit over $25 is above the self-service ceiling. A supervisor decides it. + # The one rule here that a judge CAN settle, because the threshold is + # read off the call's own `amount`. Stated in words below as + # `CreditUnderReviewThresholdWithJudge` -- the same rule, down to the + # description, differing only in who settles it -- and that twin is + # what the demo guards on. When the expression evaluator lands, the + # twin is the one to drop, which is why its name says how it is + # settled and this one's does not. - name: OrderTotalMatchesLineItems expression: Order.total == SUM(LineItem.amount) @@ -142,3 +174,43 @@ semantic_model: description: >- An order's total must equal the sum of its line items, with credits subtracted. Nobody can approve an order that does not add up. + # No counterpart in words either, and out of reach of any judge, however + # much it can read: this constrains the state the write leaves behind, + # and guards are settled before the transaction opens. It wants to be + # checked inside the transaction, or declared in the schema. + + # Rules stated in words, settled by a judge reading the attempted call. + # These are what `IssueCredit` guards on, and they carry all three + # consequences: escalate, warn and reject. + - name: CreditUnderReviewThresholdWithJudge + judgment: >- + The credit amount requested must not exceed 25 dollars, which is the + self-service ceiling for this desk. Read the amount as dollars. + on_violation: escalate + description: >- + A credit over $25 is above the self-service ceiling. A supervisor + decides it. + + - name: CreditMemoNamesAServiceFailure + judgment: >- + The memo argument of this call must name a specific thing that went + wrong on the order: a late delivery, a damaged item, a shipping + charge applied in error. A memo saying only that the customer asked, + or that the credit is goodwill, or giving no reason at all, names no + failure and does not satisfy this rule. + on_violation: warn + description: >- + Say in the credit memo what actually went wrong with the order. + + - name: CreditIsNotSplitToAvoidReview + judgment: >- + The credit requested must be the whole of what this order is owed, + not one piece of a larger amount divided to stay under the 25-dollar + self-service ceiling. A memo argument calling the credit a part, a + half, an instalment, the first or second of several, or a remainder, + or naming a total larger than the amount argument, does not satisfy + this rule. + on_violation: reject + description: >- + Raise this as a single credit for the full amount and send it for + supervisor review. diff --git a/toolbox/mdcode/docs/semantic-model/actions.md b/toolbox/mdcode/docs/semantic-model/actions.md index 90c3ecdd..94b5eb6a 100644 --- a/toolbox/mdcode/docs/semantic-model/actions.md +++ b/toolbox/mdcode/docs/semantic-model/actions.md @@ -391,8 +391,29 @@ about handling a breach lives elsewhere. **Name fields model-qualified.** Write `LineItem.memo` rather than "the memo". `kcmd` resolves every `Entity.field` token in the text against the model and fails the push when the entity declares no such field, so a rename cannot leave -the sentence pointing at nothing. The qualified name also tells the judge -exactly which value to read. +the sentence pointing at nothing. The qualified name also tells the judge which +value to read. + +One caution comes with it. A guard is settled from the attempted call's +arguments and nothing else, so a sentence phrased as a rule about stored data +can be read as a rule the judge has no evidence for. The judge is told to answer +that such a rule does not hold and to say in its reason what is missing, which +refuses the call rather than passing it. That instruction binds a model rather +than the runtime, so the rule can come back held instead, which costs a rule +that never fires and says nothing. The rule above is settled correctly under +this wording, but a guard that reads awkwardly as a statement about the call is +worth rephrasing to name the argument, and worth testing against a case it +should refuse. A rule that truly needs stored rows — comparing a credit against +the order total, say — has no wording that reaches a judge given only the call, +and belongs in an `expression` unless the judge supplied can query the store +itself. Nothing in the `Judge` interface forbids one that does; the +implementation shipped here makes a single model call with no tools. + +A rule about the state a write *leaves behind* is a different matter, and no +judge settles it however much it can read. Guards are settled before the +transaction opens, so a rule such as "an order's total equals the sum of its +lines" has nothing to look at yet. That rule belongs inside the transaction or +in the schema. **Say what does not count.** A rule with no negative example is graded against whatever the model guesses the author had in mind. "A memo that states only that @@ -1218,10 +1239,31 @@ names a guard stated as an expression, nothing here evaluates one, and so the reported here instead — before any agent exists, rather than inside a transaction. -`kcmd agent tools` supplies no judge, so an action guarded by a judgment is -marked the same way and for the same reason: the derivation reports what the -runtime would do with the judge it holds, and it holds none. Passing a judge -through to the tools an agent is handed is the next step and is not taken yet. +An action guarded by a judgment is marked the same way when the derivation +holds no judge, and for the same reason: what is reported is what the runtime +*would* do with what it is holding, and with no judge it would refuse. Supply +one and the same action is offerable, with the same description and the same +parameters: + +```console +$ kcmd agent tools --judge +Rules stated in words go to gemini-2.5-flash (us-central1). +... + action issue_credit (IssueCredit) +``` + +The flag takes an optional model name, the same way +[`kcmd action run --judge`](#a-guard-settled-in-words) does. What it does not do +is call one: a judge settles a rule when an action runs, and printing what an +agent is offered runs none, so this listing costs nothing however many guarded +actions it names. + +The judge belongs to the derivation rather than to each invocation, which is the +one place this is easy to get wrong. Whether a guarded action can be offered *at +all* depends on holding a judge, so the same object has to answer `runnable` and +answer the call. A tool derived with a judge and then invoked without one would +be advertised as callable and refused mid-call, which is the drift the next +paragraph is about. The tool is still returned, still named and still described. An action the model declares should not vanish from what the model offers; what it is waiting @@ -1263,6 +1305,11 @@ if (!runtime.store) throw new Error(runtime.storeError); const {callable, withheld, instruction} = callableTools(modelTools({runtime})); ``` +`modelTools` also takes `judge`, and passing one is what makes an action guarded +by a judgment callable at all. `GeminiJudge` implements the seam over Vertex AI; +anything with a `decide` method does. Omit it and such an action is still +derived, still named and still described, and reported in `withheld`. + One call returns a runtime for every model document in the entry group. Each carries the store its deployment target names, the profile it was built under, and the document it was authored in, so a message about one model can say which @@ -1327,12 +1374,23 @@ not. The place to put it is the model. ### A worked example `demo/semantic-model/agent/` is an agent built this way, running against a live -operational store: a commerce model, a binding profile, and one file of 56 lines -that names no table, no column and no business term. Thirteen of those lines are -the adapter onto the agent framework. Its README walks the same four steps and -states what the run cannot yet do — the $30 credit it issues is over the model's -declared $25 self-service ceiling and is written anyway, because that ceiling is -an expression and nothing evaluates one. +operational store: a commerce model, a binding profile, and one file of 66 lines +that names no table, no column, no business term and no dollar threshold. +Thirteen of those lines are the adapter onto the agent framework. Its README +walks the same steps and reaches all three of `on_violation`'s outcomes against +that store: a $30 credit held because the model's $25 self-service ceiling is +`escalate`, a credit written with a warning because the memo names no service +failure, and a credit refused outright because the memo admits it is one piece of +a larger amount. + +It also states what that costs and what it cannot do. The model guards on three +judgments, so the demo pays three model calls per attempted write and loads with +the all-judged warning; the two rules that compare the call against stored rows +are declared but not enforced, one because the judge it hires reads no store and +one because it constrains the state after the write. And the split-credit +rule fires only because the model tells callers to disclose a split in the memo, +which makes it a check on honest mistakes rather than a control — the version +that would hold regardless is an expression over what is already stored. ## What is not modeled yet diff --git a/toolbox/mdcode/docs/semantic-model/reference.md b/toolbox/mdcode/docs/semantic-model/reference.md index d1627be0..5bf691c4 100644 --- a/toolbox/mdcode/docs/semantic-model/reference.md +++ b/toolbox/mdcode/docs/semantic-model/reference.md @@ -101,9 +101,14 @@ runs nothing. A tool the runtime cannot call is listed and marked `[NOT RUNNABLE]` rather than dropped, with the reason in its description, so a refusal is visible before any agent exists. To see what a guard costs today, add a constraint to an action's -`guards` and run this again. This command supplies no judge, so an action -guarded by a `judgment` is marked unrunnable here even though -`kcmd action run --judge` would settle it. +`guards` and run this again. What is offerable depends on what the caller holds, +so an action guarded by a `judgment` is marked unrunnable without `--judge` and +callable with it. + +| Flag | Effect | +|------|--------| +| `--profile [name]` | Read the model under this binding profile. Defaults to `default_profile`, else the model's inline bindings. | +| `--judge [model]` | List what an agent holding a judge is offered: an action guarded by a rule stated in words is callable rather than `[NOT RUNNABLE]`. Takes a Gemini model id, defaulting to `gemini-2.5-flash`, on the same region rule as [`action run --judge`](#action). No model is called either way — a judge settles a rule when an action runs, and this listing runs none. | A model whose profile names no Spanner database offers no tools, because calling one needs a store. That model is reported as offering none and the rest of the diff --git a/toolbox/mdcode/src/libts/semantic/runtime/agent_tools.ts b/toolbox/mdcode/src/libts/semantic/runtime/agent_tools.ts index 5793813b..162dc731 100644 --- a/toolbox/mdcode/src/libts/semantic/runtime/agent_tools.ts +++ b/toolbox/mdcode/src/libts/semantic/runtime/agent_tools.ts @@ -22,27 +22,24 @@ * tool it should not have gets back an outcome it has to report rather than a * knob it can turn. * - * One thing the runtime cannot yet do shows through here. A rule stated in - * words is settled by a judge and this module supplies none, and a rule stated - * as an expression is settled by nothing at all, so runAction refuses any - * action that names either in `guards` rather than running it unchecked. A tool for such an action would fail every - * time it was called, which is a bad thing to hand a caller that cannot see - * why. So a tool carries `runnable`, and an adapter binds the ones that are; - * the rest are still returned, named and explained, because an action the - * model declares should not vanish from a listing of what the model declares. + * What the runtime can settle shows through here, because it decides which + * tools are worth handing out. A guard stated as a `judgment` is settled by + * asking a judge, so passing one in is what makes the actions it gates + * callable. A guard stated as an `expression` is text nothing computes here, + * so runAction refuses any action naming one rather than running it unchecked, + * judge or no judge. A tool for such an action would fail every time it was + * called, which is a bad thing to hand a caller that cannot see why. So a tool + * carries `runnable`, and an adapter binds the ones that are; the rest are + * still returned, named and explained, because an action the model declares + * should not vanish from a listing of what the model declares. */ import {boundTable, spannerTable} from '../binding'; import {Action, Entity, fieldBinding, SemanticModel} from '../ir'; import {dialectFor} from './dialect'; -import { - ActionHandler, - ActionOutcome, - bindScalar, - runAction, - whyRefusedWithoutRunning, -} from './run_action'; +import {Judge} from './judge'; +import {ActionHandler, ActionOutcome, bindScalar, runAction, whyRefusedWithoutRunning,} from './run_action'; import {runtimeClient, SemanticRuntime} from './runtime'; @@ -76,10 +73,12 @@ export interface ActionTool { parameters: ToolParameter[]; /** * Whether calling this would reach the store. False when the runtime would - * refuse it before opening a transaction -- today, because the action names - * a guard that nothing available here settles, or because this binding - * supplies no executor. `invoke` still works and still reports the refusal; - * this is here so an adapter can decline to offer a tool that cannot work. + * refuse it before opening a transaction -- because the action names a guard + * nothing here can settle, or because this binding supplies no executor. + * Which guards can be settled depends on what was passed in: a judged guard + * needs a `judge`, and an expression needs an evaluator that does not exist + * yet. `invoke` still works and still reports the refusal; this is here so + * an adapter can decline to offer a tool that cannot work. */ runnable: boolean; /** Why `runnable` is false, in words a caller can report. */ @@ -112,7 +111,9 @@ export interface ToolResult { * The statements ran and the commit itself failed to answer. */ unknown?: boolean; - /** What the caller should do next, when the outcome permits only one thing. */ + /** + * What the caller should do next, when the outcome permits only one thing. + */ whatToDo?: string; /** * What a rule reported without stopping the write. An advisory guard whose @@ -133,6 +134,17 @@ export interface ActionToolOptions { * not wrap a call it could not roll back. */ handler?: ActionHandler; + /** + * Settles the guards the model states in words. An action guarded by a + * judgment is refused without one, so passing a judge here is what makes + * such an action offerable at all. + * + * The same judge answers `runnable` and the call, which is why it is passed + * to the derivation rather than to each invocation: a tool derived with a + * judge and then called without one would be advertised as runnable and + * refused mid-call. + */ + judge?: Judge; } @@ -155,8 +167,7 @@ function toolFor(action: Action, opts: ActionToolOptions): ActionTool { // function for the whole model, so passing it through unconditionally would // retract that claim for every action at once -- silently, since runAction // prefers a handler over the action's own statements. - const handler = - action.executor?.kind === 'sql' ? undefined : opts.handler; + const handler = action.executor?.kind === 'sql' ? undefined : opts.handler; // Asked of the runtime rather than worked out again here. Two copies of this // rule drift, and neither direction of the drift is visible: a tool said to // be runnable that refuses every call, or one withheld that would have run. @@ -165,7 +176,8 @@ function toolFor(action: Action, opts: ActionToolOptions): ActionTool { // model, then the runtime having no store, which is the same sentence on // every tool and says nothing about this one. const model = opts.runtime.model; - const blocked = whyRefusedWithoutRunning(model, action, handler) ?? + const blocked = + whyRefusedWithoutRunning(model, action, handler, opts.judge) ?? noStore(opts.runtime) ?? undefined; const tool: ActionTool = { name: snakeCase(action.name), @@ -179,6 +191,7 @@ function toolFor(action: Action, opts: ActionToolOptions): ActionTool { actionName: action.name, args, handler, + judge: opts.judge, }); return describeOutcome(outcome); }, @@ -205,8 +218,9 @@ function toolDescription( parts.push(`This call is gated by ${joinNames(gates)}.`); } if (blocked) { - parts.push(`Calling this will not work: ${blocked} Report that rather ` + - `than retrying.`); + parts.push( + `Calling this will not work: ${blocked} Report that rather ` + + `than retrying.`); } return parts.join('\n\n'); } @@ -543,8 +557,10 @@ function instructionFor(model: SemanticModel): string { 'caller\'s time. Never compute a total or a balance yourself; the ' + 'tools do that. When a tool reports that a write did not happen, read ' + 'the reason it gives and repeat it plainly; if it says a person has to ' + - 'decide, say so and stop, because you cannot approve it yourself. ' + - 'Finish by saying what you changed.'); + 'decide, say so and stop, because you cannot approve it yourself. When ' + + 'a write did happen and the tool returns warnings, the change landed ' + + 'and a rule still went unmet or unchecked: report both, because ' + + 'nobody else will. Finish by saying what you changed.'); return parts.join('\n\n'); } @@ -655,9 +671,7 @@ function filterDescription(entity: Entity, field: BoundField): string { function lookupDescription(entity: Entity, bound: BoundField[]): string { const parts: string[] = []; - parts.push( - entity.description?.trim() || - `Look up ${entity.name} records.`); + parts.push(entity.description?.trim() || `Look up ${entity.name} records.`); if (bound.length) { parts.push( `Returns ${bound.map(f => f.name).join(', ')}. Every argument is an ` + @@ -717,8 +731,8 @@ async function runLookup( if ('error' in value_) { return { ...empty, - problem: `${value_.error} No ${entity.name} has ${field.name} = '${ - value}'.`, + problem: + `${value_.error} No ${entity.name} has ${field.name} = '${value}'.`, }; } const bind = `f_${predicates.length}`; @@ -745,8 +759,8 @@ async function runLookup( let rows: Array>; try { rows = await client.withSession(async sessionName => { - const res = await client.executeQuery( - sessionName, {sql, params, paramTypes}); + const res = + await client.executeQuery(sessionName, {sql, params, paramTypes}); if (res.status < 200 || res.status >= 300) { throw new Error(res.message ?? `${res.status}`); } diff --git a/toolbox/mdcode/src/libts/semantic/runtime/judge.ts b/toolbox/mdcode/src/libts/semantic/runtime/judge.ts index 1de29b7e..ab667b2b 100644 --- a/toolbox/mdcode/src/libts/semantic/runtime/judge.ts +++ b/toolbox/mdcode/src/libts/semantic/runtime/judge.ts @@ -22,9 +22,12 @@ export interface JudgeRequest { // What the caller is trying to do, and why the model says it exists. action: string; actionDescription?: string; - // The arguments as the caller stated them, which is the whole of what a - // judge sees. It runs before the transaction opens and reads no store, so it - // is shown the proposal rather than its consequences. + // The arguments as the caller stated them, which is the whole of what this + // request carries. An implementation may consult whatever else it holds -- + // one built over a store connection can read the rows a rule names, and + // nothing here forbids it -- but a judge is asked before the transaction + // opens, so what it cannot be shown is the state the write would produce. A + // rule about that has no binding point here. arguments: Record; } diff --git a/toolbox/mdcode/src/tool/commands.ts b/toolbox/mdcode/src/tool/commands.ts index be264df7..480f22a9 100644 --- a/toolbox/mdcode/src/tool/commands.ts +++ b/toolbox/mdcode/src/tool/commands.ts @@ -15,7 +15,7 @@ import * as kc from '../libts/semantic/deploy_knowledge_catalog'; import * as deploySpannerLeg from '../libts/semantic/deploy_spanner'; import {googleDeploymentTargets} from '../libts/semantic/deployment_target'; import {ActionTool, EntityTool, modelTools} from '../libts/semantic/runtime/agent_tools'; -import {Action, ActionParameter} from '../libts/semantic/ir'; +import {Action, ActionParameter, constraintEvaluation, SemanticModel} from '../libts/semantic/ir'; import {provisionCustomTypes} from '../libts/semantic/kc_custom_types'; import {LoadedModel, loadSemanticModels} from '../libts/semantic/loader'; import {serializeModel} from '../libts/semantic/osi_converter'; @@ -1285,7 +1285,7 @@ function listActions( .join(', ')}`); } if (a.executor) { - console.log(` run: ${runLine(a)}`); + console.log(` run: ${runLine(a, model)}`); } else { console.log( ` run: bind an executor in a profile to run this.`); @@ -1300,6 +1300,9 @@ export interface AgentOptions { // `string|boolean` for the same reason the others are: cac yields `true` for // a bare `--profile` and `false` for `--no-profile`. profile?: string|boolean; + // `--judge [model]`: derive the listing as an agent holding a judge would + // see it. `true` for a bare `--judge`, which takes the default model. + judge?: string|boolean; } @@ -1318,6 +1321,14 @@ export interface AgentOptions { // print. `kcmd action run` calls the write half; the read half is a SELECT the // tool would issue, and `gcloud spanner databases execute-sql` will run it. // +// What the listing can call depends on what the caller holds, so `--judge` +// takes the same argument `kcmd action run` does. Without it an action guarded +// by a rule stated in words is marked NOT RUNNABLE, which is accurate for a +// caller holding no judge and misleading about an agent that holds one -- the +// point of this command is to show what the agent will be handed, and the +// judge is part of what decides that. No model is called here either way: a +// judge is asked when an action runs, and the listing runs none. +// // Returns a process exit code (0 on success). export async function agent( command: string, options: AgentOptions = {}): Promise { @@ -1336,6 +1347,14 @@ export async function agent( return 1; } + // Built the way `kcmd action run --judge` builds it, from the context this + // command already holds, so the listing and the run agree about who answers. + const judge = options.judge ? + new GeminiJudge( + ctx, typeof options.judge === 'string' ? {model: options.judge} : {}) : + undefined; + if (judge) console.log(`Rules stated in words go to ${judge.name}.`); + let incomplete = false; for (const runtime of opened) { const {model, store, storeError, profile, entryGroup} = runtime; @@ -1364,7 +1383,7 @@ export async function agent( console.log(` store: ${storeLine(store)}`); console.log(); - const {lookups, actions, instruction} = modelTools({runtime}); + const {lookups, actions, instruction} = modelTools({runtime, judge}); for (const tool of actions) printActionTool(tool); for (const tool of lookups) printLookupTool(tool); console.log(' instruction:'); @@ -1487,9 +1506,17 @@ function describeParameter(p: ActionParameter): string { // The command line that runs an action, with a placeholder per parameter. -function runLine(a: Action): string { +// +// A guard settled by judgment needs `--judge`, and the line says so, because a +// suggested command that is certain to be refused is worse than no suggestion: +// the reader tries it, reads a refusal, and has to work out that the fix is a +// flag this listing knew about all along. +function runLine(a: Action, model: SemanticModel): string { const args = a.parameters.map(p => ` --arg ${p.name}=<${p.type}>`).join(''); - return `kcmd action run ${a.name}${args}`; + const guards = new Set(a.guards ?? []); + const judged = (model.constraints ?? []).some( + c => guards.has(c.name) && constraintEvaluation(c) === 'judged'); + return `kcmd action run ${a.name}${judged ? ' --judge' : ''}${args}`; } diff --git a/toolbox/mdcode/src/tool/main.ts b/toolbox/mdcode/src/tool/main.ts index c8df7fd4..2864ab27 100644 --- a/toolbox/mdcode/src/tool/main.ts +++ b/toolbox/mdcode/src/tool/main.ts @@ -169,6 +169,9 @@ cli.command( .option( '--profile [name]', 'Read the model under this binding profile; defaults to default_profile, else the inline bindings') + .option( + '--judge [model]', + 'List what an agent holding a judge is offered, naming a Gemini model or taking the default; without it, an action guarded by a rule stated in words is marked NOT RUNNABLE. No model is called either way') .action(async (command, options) => { let exitCode = 1; try { diff --git a/toolbox/mdcode/tests/libts/semantic/runtime/agent_tools.test.ts b/toolbox/mdcode/tests/libts/semantic/runtime/agent_tools.test.ts index 5c7db068..34e44342 100644 --- a/toolbox/mdcode/tests/libts/semantic/runtime/agent_tools.test.ts +++ b/toolbox/mdcode/tests/libts/semantic/runtime/agent_tools.test.ts @@ -16,6 +16,7 @@ import * as path from 'node:path'; import {actionTools, callableTools, describeOutcome, entityTools, modelTools} from '../../../../src/libts/semantic/runtime/agent_tools'; import {Action, Constraint, Entity, SemanticModel} from '../../../../src/libts/semantic/ir'; import {loadModels} from '../../../../src/libts/semantic/loader'; +import {Judge} from '../../../../src/libts/semantic/runtime/judge'; import {SemanticRuntime} from '../../../../src/libts/semantic/runtime/runtime'; import * as spanner from '../../../../src/libts/gcp/spanner'; @@ -271,6 +272,66 @@ describe('what counts as runnable is the runtime\'s answer, not a copy', () => { expect(tool.runnable).toBe(false); expect(tool.unavailable).toContain('NoSuchRule'); }); + + // What the caller holds is half of the verdict. A judged guard is settled by + // asking, so whether such an action can be offered depends on whether a + // judge was passed to the derivation -- and the derivation has to say so + // both ways round, or an adapter either withholds a tool that works or + // offers one that is refused on its first call. + const judged: Constraint = { + name: 'CreditIsJustified', + judgment: 'The memo must name what went wrong.', + onViolation: 'reject', + }; + + const neverAsked: Judge = { + name: 'test judge', + decide: () => { + throw new Error('the derivation must not call a judge'); + }, + }; + + test('a judged guard withholds the tool when no judge was supplied', () => { + const [tool] = + actionTools({runtime: rt(guardedBy(judged, 'CreditIsJustified'))}); + expect(tool.runnable).toBe(false); + expect(tool.unavailable).toContain('no judge'); + }); + + test('a judge makes an action guarded by a judgment offerable', () => { + const [tool] = actionTools( + {runtime: rt(guardedBy(judged, 'CreditIsJustified')), + judge: neverAsked}); + expect(tool.runnable).toBe(true); + expect(tool.unavailable).toBeUndefined(); + }); + + test('deriving the tools asks the judge nothing', () => { + // `neverAsked` throws, so this passing is the assertion: a judge settles a + // rule when an action runs, and listing what an agent is offered runs + // none. A derivation that spent a model call per guarded action would make + // `kcmd agent tools` cost money to read. + const [tool] = actionTools( + {runtime: rt(guardedBy(judged, 'CreditIsJustified')), + judge: neverAsked}); + expect(tool.actionName).toBe('PlaceOrder'); + }); + + test('a judge does not make an expression computable', () => { + // Supplying a judge must not widen what is offered past what it settles. + // A caller sent to fetch a judge, who fetched one and was refused again, + // has been sent the wrong way. + const computable: Constraint = { + name: 'QuantityIsPositive', + expression: 'quantity > 0', + onViolation: 'reject', + }; + const [tool] = actionTools( + {runtime: rt(guardedBy(computable, 'QuantityIsPositive')), + judge: neverAsked}); + expect(tool.runnable).toBe(false); + expect(tool.unavailable).toContain('does not evaluate constraints'); + }); }); diff --git a/toolbox/mdcode/tests/tool/action.test.ts b/toolbox/mdcode/tests/tool/action.test.ts index a7977fbf..471a9fae 100644 --- a/toolbox/mdcode/tests/tool/action.test.ts +++ b/toolbox/mdcode/tests/tool/action.test.ts @@ -321,6 +321,28 @@ describe('kcmd action list', () => { expect(out).not.toContain('kcmd action run IssueCredit'); }); + test('the run line names --judge when a guard is settled by judgment', + async () => { + // Copying the line is the whole point of printing it. A judged guard + // refuses without a judge, so a line omitting the flag would send the + // reader to a refusal it could have predicted. + writeWorkspace(MODEL.replace( + ' - name: CreditIsPositive\n' + + ' expression: amount > 0\n', + ' - name: CreditIsPositive\n' + + ' judgment: The credit must be for a positive amount.\n' + + ' on_violation: reject\n')); + const code = await action('list', undefined); + expect(code).toBe(0); + const out = logs.join('\n'); + expect(out).toContain( + 'run: kcmd action run IssueCredit --judge ' + + '--arg order='); + // The other action names no guard at all, so it gains nothing. + expect(out).toContain( + 'run: kcmd action run NotifyCustomer --arg order='); + }); + test('says so when a model declares no actions', async () => { writeWorkspace(NO_ACTIONS); const code = await action('list', undefined);