SE Coaching Bot

A reference implementation of a secure, retrieval-grounded LLM assistant delivered into Slack. SE coaching on MEDDPICC deal qualification is the worked example; the reusable substance is the security gateway and the controls around a managed retrieval backend. All content is synthetic.

Results

Qualification — three-model comparison

Twenty transcripts, eight MEDDPICC elements each, graded deterministically plus an LLM-as-judge layer for gap identification.

Model	Conformance	Agreement	Gap recall	Severe recall	Precision	Judged	Tidiness
claude-sonnet-4-6	1.00	0.74	0.78	0.90	0.76	0.78	0/20 clean
claude-haiku-4-5	0.00	0.76	0.81	0.90	0.71	0.67	0/20 clean
gpt-oss-120b	1.00	0.70	0.72	0.70	0.73	0.69	20/20 clean

The headline finding is an accuracy/obedience split. Haiku scored the highest agreement at 0.76 but never once emitted instruction-conformant output; every response needed repair. GPT-oss was the mirror image: 20/20 clean but mid-pack on accuracy. That finding only exists because the parse records how much repair each response needed, rather than scoring presentation habits as capability failures.

Answer quality — three retrieval skills

Twelve scenarios per skill, scored by two independent judges. Sonnet 4.6 as the graded model, GPT-oss-120b as the coverage judge, Haiku 4.5 for agreement.

Metric	Objection	Positioning	Demo	Threshold
Point coverage	0.99	0.94	0.94	≥ 0.85
Trap rate	0.07	0.11	0.14	≤ 0.20
Invention rate	0.07 (6/83)	0.07 (9/121)	0.11 (10/87)	≤ 0.15
Refusal correctness	0.00	0.00	0.50	≥ 0.50
Inter-judge agreement	0.93	0.90	0.86	≥ 0.70

Refusal correctness breaches on objection and positioning. The breaches are real: the model refuses the artefact and then improvises the substance. See docs/evals/evals-README.md for the full write-up and per-scenario details.

Other graded layers

Layer	Result
Classification accuracy	0.93
Retrieval precision (k=6)	see eval scorecard
Ingestion boundary	all retrieved URIs under kb/; positive control passing

Judged rates are judge-dependent; compare models under the same judge and trust deltas, not levels. Full scorecards with per-item records, raw verdicts, and answers are in docs/evals/.

Architecture

Slack sends an app mention to API Gateway. A public receiver Lambda verifies the Slack signature, acknowledges within the three-second budget, adds an eyes reaction, and invokes the worker asynchronously. The worker walks the security gateway and, only if the request clears, runs the engine. Every outcome resolves the eyes to a terminal reaction with a reply in-thread.

graph TD
    Slack["Slack"]
    APIGW["API Gateway"]
    Receiver["Receiver — verify, ack, react"]
    Worker["Worker — gateway, engine"]
    Gateway["Security gateway — redact, screen, classify, RBAC, outbound redact"]
    Bedrock["Bedrock — Claude and Knowledge Base"]
    Reaper["SQS and Reaper — Case 3 fallback"]

    Slack --> APIGW
    APIGW --> Receiver
    Receiver -->|async| Worker
    Worker --> Gateway
    Worker --> Bedrock
    Worker -.->|on-failure| Reaper

    style Slack fill:#ecedef,stroke:#c0c2c6,color:#3d3d3a
    style APIGW fill:#E6F1FB,stroke:#185FA5,color:#0C447C
    style Receiver fill:#E1F5EE,stroke:#0F6E56,color:#085041
    style Worker fill:#E1F5EE,stroke:#0F6E56,color:#085041
    style Gateway fill:#E1F5EE,stroke:#0F6E56,color:#085041
    style Bedrock fill:#E6F1FB,stroke:#185FA5,color:#0C447C
    style Reaper fill:#ecedef,stroke:#c0c2c6,color:#3d3d3a

Loading

The receiver's IAM role can invoke the worker but cannot touch Bedrock, S3, or Secrets Manager. The public-facing function has the smallest possible blast radius.

The security gateway

src/se_coach/gateway/ — the differentiator.

Inbound pipeline in a fixed order: redact → injection screen → classify → RBAC. Retrieved chunks pass a separate context screen before reaching the prompt. Replies pass the redactor on the way out.

Two structural rules: the system/user prompt split is a security boundary (policy in system, user text and retrieved context in user), and retrieved text is never trusted as instruction. The injection screen is pattern-based and deterministic; it runs ahead of classification so a prompt-injection attempt is caught before the classifier spends a model call on it.

Four skills

Skill	Retrieval	Notes
qualify	no	MEDDPICC assessment against a pasted transcript
objection	yes	objection handling from the corpus
position	yes	competitive positioning
demo_plan	yes	demo planning

Skills are classified from natural language, not slash commands. The MEDDPICC rubric is policy, not corpus: it lives in content/policy/, loads into the system channel, and is excluded from Knowledge Base ingestion.

Failure modes

Three cases, all contracted. Case 1 (worker throws): try/finally resolves the reaction. Case 2 (reply post fails): bounded retry with backoff. Case 3 (worker never starts): Lambda on-failure destination routes to SQS; a reaper Lambda swaps the eyes for a cross and replies in-thread.

Content boundary

Only content/kb/ is synced to S3 for retrieval. The ingestion role's read scope is pinned to that prefix. content/policy/, content/eval/, and content/fixtures/ never reach the bucket. An ingestion-boundary eval (evals/boundary.py) proves this against the live Knowledge Base after every sync, guarded by a positive control.

Stack

Python 3.13 · uv · AWS Bedrock and Knowledge Bases · S3 · Lambda · API Gateway · Terraform · ruff · pytest · GitHub Actions (lint, tests, secret scanning, gated deploy).

Getting started

uv sync --group dev
cp .env.example .env      # fill in real values; never commit .env
make test                 # pytest
make lint                 # ruff check + format check

Running the eval harness

uv run python -m evals.run                   # offline pre-flight only (CI gate)
uv run python -m evals.run --graded          # pre-flight, then model-calling layers
  --only answer-quality                      # one section; comma-separated for several
  --models id1,id2                           # multi-model comparison
  --skill objection|position|demo_plan       # which scenario set
  --secondary-judge id                       # enables inter-judge agreement
  --fresh                                    # ignore checkpoint caches

Layout

src/se_coach/       the library: engine, gateway/, model and retrieval seams,
                    MEDDPICC schema, qualification parsing
src/receiver/       receiver Lambda — verify, ack, react, invoke
src/worker/         worker Lambda — gateway, engine, resolve
src/reaper/         DLQ consumer for invocations that never ran
evals/              eval harness; run.py is the entry point
tests/              test suite; fakes.py is apparatus, not tests
content/            synthetic content pack (contract: content/schema.md)
  kb/               knowledge base corpus (the ONLY subtree ingested to S3)
  policy/           MEDDPICC rubric (system prompt, not corpus)
  eval/             transcripts and scenarios with ground truth
  fixtures/         security fixtures (injection, PII, role matrix)
infra/              Terraform; S3 backend, CI-owned delivery
scripts/            build_lambda.sh — zip-plus-layer packaging
docs/               process.md, and eval scorecards under docs/evals/

What is not measured

The Lambda handlers are tested offline (test_receiver.py, test_worker.py, test_reaper.py) but there is no end-to-end integration test against the deployed stack. Case 3 (worker never starts) is contracted by the reaper but not exercised by any automated test. Both are accepted gaps, documented rather than hidden.

Licence

MIT.