升级 llm-compat 到 v0.10.0:根治总结/校对被拒绝检测误判 - #60
Merged
Conversation
在升级前锁定 7 格不变式轴表;长文含题材词与 on_all_refused 构造参数 在 v0.8.0 下预期失败,升级 v0.10.0 后应变绿。 Co-authored-by: Cursor <cursoragent@cursor.com>
修复拒绝检测对长文题材词的无约束子串误判(上游 issue #22)。 Co-authored-by: Cursor <cursoragent@cursor.com>
不使用 v0.10.0 默认 return_best,保持链耗尽时诚实标 failed; 删除未使用的 detect_provider import(v0.9.0 breaking)。 Co-authored-by: Cursor <cursoragent@cursor.com>
Required Gate v2 — 状态面板当前状态:pass · 可合并 当前裁决: Gate 历史(v1;来源为持久化
|
| Run | Attempt | Head | 状态 | 收件人动作 |
|---|---|---|---|---|
| 32572354986 | 1 | abe82fd |
pass |
可合并 |
| 32573032258 | 1 | 6f5d680 |
pass |
可合并 |
历史行按 run_id + run_attempt 去重并只增不删;删除本评论后可由 gate_terminal 制品重建。
⚙️ Review ledger state(机器状态记录,非评审结论)
机器状态明细
完整数据保存在 |
gate 主审 finding testing-global-client-state-leak: test_on_all_refused_raise_passed_at_construction 调真实 set_default_config, 把 mock 客户端写进 llm 模块的 _sync_client / _default_config 且不恢复。 已实证泄漏为真:临时关掉本 fixture 后,同会话内后置探针立刻断言失败 (global _sync_client leaked as MagicMock)。此前全量绿只是因为下游碰全局的 测试恰好都自己重设,把泄漏盖住了。 via [HAPI](https://hapi.run) Co-Authored-By: HAPI <noreply@hapi.run> Agent-Executor: cc Agent-Session: 9dd95690-cd40-438a-aed2-eb0c1279caa7 Agent-Effort: xhigh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
生产上「内容总结」失败率从 7 月的 0.7%(3/451)涨到 8 月的 4.7%(14/300),校对层
calibration_status=partial同期从 4.3%(11/256)涨到 7.9%(22/277)。根因不在模型能力,在依赖库
llm-compat≤v0.9.1 的拒绝检测:默认关键词表把「违反」「无法提供」「涉及敏感」这类普通题材词当作拒绝信号,并且做的是全文无长度、无位置约束
的子串匹配。任何正常长文只要正文里出现这些词,就被整条
content_fallbacks降级链判为「模型拒绝」,逐个模型试完后抛
All models refused。实测复现(绕过 llm-compat 直发同一网关、同一 prompt、同一模型):
模型返回的是一篇 13402 字的完整总结,只因正文里出现 4 次「违反」被整条链丢弃。失败率
上升是因为 8 月起法律类内容(刑事律师在线咨询等)转录量变高,不是缺陷本身变化。
已向上游提 issue 并获修复:zlxlabs/llm-compat#22
改动
llm-compatv0.8.0→v0.10.0。v0.10.0 把默认词表换成严格的拒绝言语行为正则(「第一人称两处调用侧改动:
on_all_refused="raise"。v0.10.0 把这个默认值从"raise"改成了"return_best"(链耗尽时把「推断层候选里正文最长的那个」救回来返回)。本仓不采用该默认:我们有诚实状态模型(
summary_status/calibration_status),链耗尽时宁可诚实标failed,也不能把一段疑似拒绝的正文当成总结写进数据库——那是本仓的 P1 红线「静默出错」。显式写死,不做成配置项。
detect_provider。v0.9.0 把它的返回值从str改成ProviderDetection(Breaking),本仓全仓只有这一处 import、无任何调用点,直接删掉。
未采用 v0.10.0 的其余新参数(
refusal_keywords_mode/refusal_max_content_length/refusal_head_window)——默认值已是本 PR 想要的行为,抄进调用方反而让日后上游调参失效。config/*、prompt、processors、coordinator、api 均未改动。跨版本 Breaking 核对
detect_provider()返回值str→ProviderDetectionregister_provider(..., caps=)注册期全 schema 校验build_request_payload等新增strict关键字参数llm.py:792调用不受影响mimo-*provider familyon_all_refused默认raise→return_best"raise"ChatResult新增refusal_suspected/refusal_evidence测试
新增
tests/unit/test_refusal_contract.py:7 格不变式轴表(正文长度 × 拒绝句式形态),调用真实的
llm_compat.refusal.detect_refusal,用 wire-format 响应 dict 驱动,不 mock 被测库。
finish_reason=content_filter(声明层)content=None+finish_reason=stop(畸形响应)另有两条:构造期 spy 断言
on_all_refused="raise"确实传入SyncLLMClient;终态异常链
ContentPolicyError → map_llm_compat_error → RetryableError保持不变。断言约束力已实证(不是恒真断言)——同一段语料在新旧两版下结果相反:
升级前该文件 3 红(两格长文误判 + 缺
on_all_refused),升级后 9 绿。全量
uv run pytest tests/unit:2769 passed。部署备注
合并后需重建 n305 镜像才在生产生效。