refactor(libsy): share capability input and separate judge settings

[nachiketb-nvidia]

What

Share conversation selection between judge types and separate capability routing settings from the LLM judge's prompt, token limit, and solve-probability thresholds. TaskClassifierConfig now selects CapabilityJudgeConfig::Llm(LlmCapabilityConfig { ... }). Existing LLM behavior is preserved.

Why

A relative-advantage decision judge needs its own cutoff and evidence. Keeping LLM settings inside the LLM variant prevents them from becoming unused or misleading decision settings. Part of SWITCH-1687.

How

• Extract TaskInput::messages for the shared conversation window and reasoning removal. Append the LLM routing instruction only when building LLM judge messages.
• Keep affinity, classification frequency, conversation window, and error recovery on the shared config.
• Move the existing LLM settings, defaults, and validation into LlmCapabilityConfig.
• Preserve the flat serialized config and Python signatures. Update Rust callers, including stage/composite fixtures, to construct the LLM variant explicitly.

Validation

• 26 capability tests, four composite tests, and the stage judge-threshold test passed.
• Five focused runner config tests passed, including prompt/token settings and composite construction.
• Three Python classifier-config tests passed against a freshly built extension. The async tests required running outside the sandbox; inside it they timed out.
• Affected-crate Clippy passed with warnings denied; formatting passed.
• Existing client fail-open regression passed. No new test functions or live provider calls.

Notes for reviewers

Stack 1/2, based on main. Review and merge in order: #900 → #901. Start with TaskInput::messages, the config types, and the compatibility deserializer; other changes are mechanical Rust caller updates.

Includes the input refactor previously proposed in #899. Diff: seven files, +185 / −85 lines.

This changes the Rust construction API. Existing TOML and Python callers retain their behavior. The decision variant and implementation arrive together in the next PR.

Summary by CodeRabbit

• Configuration
  • Classifier settings now separate routing and conversation-window options from judge-specific thresholds, contract settings, and output-token limits.
  • LLM judge settings are grouped under a dedicated configuration, and the judge configuration types are publicly available.
  • Existing Python constructor parameters and defaults remain unchanged, and flat serialized settings continue to map to the nested configuration.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-900/
Built to branch gh-pages at 2026-10-01 22:37 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA-NeMo/Switchyard/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 00da9ab5-27d4-4eef-8bd0-2fe9d2e7953c

📥 Commits

Reviewing files that changed from the base of the PR and between 12922d5 and cfba169.

📒 Files selected for processing (7)

• crates/libsy-llm-client/tests/observability.rs
• crates/libsy/src/algorithms/composite.rs
• crates/libsy/src/algorithms/llm_class.rs
• crates/libsy/src/algorithms/stage.rs
• crates/libsy/src/lib.rs
• crates/switchyard-py/src/libsy_bindings.rs
• crates/switchyard-runner/src/algorithm.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

The classifier configuration separates routing and message-window options from LLM judge settings. The classifier, runner, Python bindings, and tests now use the nested judge configuration. Flat serialized settings map into the new structure.

Changes

Classifier judge configuration

Layer / File(s)	Summary
Define and validate judge configuration crates/libsy/src/algorithms/llm_class.rs, crates/libsy/src/lib.rs	TaskClassifierConfig now contains a judge configuration. CapabilityJudgeConfig selects the LLM judge, and LlmCapabilityConfig holds its thresholds, contract, and output-token limit. Deserialization maps flat settings into the nested form.
Use judge settings in classification crates/libsy/src/algorithms/llm_class.rs	Message selection uses the configured window. Classifier construction reads the contract, thresholds, and token limit from LlmCapabilityConfig. Tests use the nested configuration for policy setup and validation cases.
Update callers and configuration tests crates/switchyard-runner/src/algorithm.rs, crates/switchyard-py/src/libsy_bindings.rs, crates/libsy/src/algorithms/composite.rs, crates/libsy/src/algorithms/stage.rs, crates/libsy-llm-client/tests/observability.rs	Runner and Python binding code construct the nested configuration. Router, stage, and observability tests use the same configuration shape.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to cfba1

This refactor keeps the flat serialized configuration and the Python signatures unchanged, and no concrete merge-blocking risk was identified. The Rust construction API changes, and the callers shown in the change are updated.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 33 functions across 7 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: sharing capability input and separating judge settings in the libsy refactor.

• Fix all pre-merge checks with AI

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

_{I’m a rabbit with config tucked tight,
Nested judge settings hop into sight.
Thresholds and prompts find their place,
While routing keeps its own pace.
I nibble a test, then bound away!}

_{Comment @coderabbitai help to get the list of available commands.}

[ayushag-nv]

@nachiketb-nvidia Looks good. Just one thing, adding a CapabilityConfig , does it conflict with CustomClassifier / Judge how is that working now ?