Skip to content

feat(chat): add hybrid semantic skill retrieval - #566

Open
Voitamin wants to merge 4 commits into
LazyAGI:mainfrom
Voitamin:codex/fix-skill-candidate-selection
Open

Voitamin wants to merge 4 commits into
LazyAGI:mainfrom
Voitamin:codex/fix-skill-candidate-selection

Conversation

@Voitamin

Copy link
Copy Markdown

What

  • 新 SkillRetriever:目录 ≤20 全量元数据;>20 走 lexical(BM25 风格,中英)+ dense(embed_main)两路 RRF 融合 TopK=8;embedding 缺失/报错/超时自动降级 lexical
  • 新 search_skills 工具:Loop 内二次发现,命中后动态 expose,下一轮 get_skill 可用
  • chat_service:显式选择绕过召回;candidates 模式召回后 expose;suppress 只表示「初始不预载」
  • 依赖 feat(agent): support dynamic skill visibility LazyLLM#1285

Why

原实现用 query 与 skill 路径名字面打分截 Top5,语义相近但字面不近的表达会在模型判断前被过滤;首轮漏召回后也没有二次发现入口。

Tests

  • test_skill_retriever.py:50 descriptor,中英/复合/CV 歧义/正逆序/降级/缓存/线程池
  • test_skill_runtime_integration.py:真实 ReactAgent + fake LLM,初始 expose→Loop get_skill、并发 session 隔离、Loop 内二次发现/纠错
  • 邻接回归 1223 passed, 1 skipped(live embedding 默认关闭)
  • live embedding:SiliconFlow bge-m3 7 条语义用例 Recall@5=7/7(小型 sanity check,非系统准确率)
  • 真实模型 smoke:DeepSeek-V4-Flash 5 场景,模型会主动 search/get,无关请求不滥用

边界(诚实声明)

  • 真实模型行为仅 5 条 smoke,非大规模评测
  • 完整 HTTP /chat E2E 未跑(Docker 环境受限)
  • 100/500/1000 规模 Recall 曲线未测
  • TopK=8 / 阈值=20 是实验起点,非调优最优

@Voitamin
Voitamin marked this pull request as ready for review August 20, 2026 08:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d437191f8a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +872 to +875
allowed_skill_ids = list(dict.fromkeys([
*(agent.available_skills or []),
*([workflow_skill_name] if workflow_skill_name else []),
]))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove explicitly excluded skills from the searchable catalog

When a bound skill is negated in the request, task profiling records it in task_profile.excluded_resources.skill_names, but this catalog still includes every agent.available_skills entry. The newly registered search_skills tool therefore remains able to expose and load a skill the user explicitly requested not to use, unlike the filtering already applied to excluded knowledge bases and workflows. Subtract the excluded skill IDs before constructing allowed_skills.

Useful? React with 👍 / 👎.

Comment on lines +143 to +145
if len(catalog) <= self._small_catalog_threshold:
hits = tuple(SkillRetrievalHit(item, 1.0, ('all',)) for item in catalog)
return self._result(hits, 'all', len(catalog), started)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor search result limits for small catalogs

For catalogs at or below the threshold, this branch returns every descriptor before limit is clamped or applied. Because the same retriever backs search_skills, a call such as search_skills(query, limit=1) against 20 skills returns and exposes all 20, contradicting the tool's documented maximum and defeating selective discovery. The full-catalog behavior may be appropriate for initial metadata injection, but tool searches still need to cap the exposed hits.

Useful? React with 👍 / 👎.

Comment on lines +303 to +304
except FutureTimeoutError as exc:
future.cancel()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Prevent timed-out embeddings from exhausting shared workers

If an embedding call has already started, Future.cancel() returns false and does not stop its thread. Four stalled backend calls can therefore occupy every worker in the class-level executor; all later retrievals queue behind them, wait the full configured timeout, and fall back to lexical retrieval even after new embedding requests could otherwise succeed. The timeout path needs cancellation at the embedding-client level or worker isolation/recovery that does not leave the shared pool occupied.

Useful? React with 👍 / 👎.

@Voitamin
Voitamin force-pushed the codex/fix-skill-candidate-selection branch from d437191 to 6376dbe Compare August 20, 2026 08:36

Copy link
Copy Markdown
Contributor

感谢这个 PR,整体方向和我们下一版本的 Skill Retrieval 规划比较一致。首轮只召回相关 Skill,并支持 Agent 在执行过程中通过 search_skills 补充发现,可以解决 Skill 数量较多时全部加载导致的上下文膨胀问题;同时 embedding 作为可选增强能力、未配置时使用关键词检索并支持自动降级,这个设计也比较合理。

建议后续主要考虑以下几点:

  • 优化 Skill 索引的生命周期。 当前每次 Query 仍可能重新扫描 Skill 目录、解析全部元数据;配置 embedding 时,冷启动或缓存失效还需要重新计算向量。后续可以缓存 Skill metadata 和关键词索引,embedding 索引按需启用,并在 Skill 安装、更新时增量维护。
  • 增加召回和上下文预算控制。 Retrieval 应允许返回 0 个 Skill,而不是一定填满 TopK;同时需要限制候选 Skill 的数量/Token,以及多次 search_skills 后候选集合的大小,避免运行过程中再次膨胀。
  • 调整 Skill 的上下文注入方式。 后续计划将召回的 Skill 作为一次用户 Query 内的临时 Runtime Context,在该 Query 对应的多轮模型调用中持续可见,但不写入 Conversation History。Session 中只保留实际使用过的 Skill ID,用于后续 Query 的复用和再次召回。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants