Skip to content

[Optimization] Optimize error message when embedding model change - #864

Open
chenhao0205 wants to merge 10 commits into
LazyAGI:mainfrom
chenhao0205:ch/opt-milvus
Open

chenhao0205 wants to merge 10 commits into
LazyAGI:mainfrom
chenhao0205:ch/opt-milvus

Conversation

@chenhao0205

@chenhao0205 chenhao0205 commented Nov 26, 2025

Copy link
Copy Markdown
Contributor

📌 PR 内容 / PR Description

  • 添加全局注册表EmbeddingModelRegistry,记录和校验数据库的embedding配置是否正确。
  • 校验逻辑:
    (1)是否数据库类型和数据库地址已在注册表存在,不存在则是全新数据库不校验
    (2)若数据库已存在,检查embed_key是否存在,不存在则不校验
    (3)若数据库和embed_key都存在,校验当前模型和模型输出维度是否和已注册信息一致,不一致则报错

✅ 变更类型 / Type of Change

  • [√] 性能优化 / Performance optimization

🧪 如何测试 / How Has This Been Tested?

  1. 运行basic_tests中涉及到document和数据库的全部测例

📷 截图 / Demo (Optional)

image

⚡ 更新后的用法示例 / Usage After Update

# 使用方法1:自动执行
from lazyllm import Document, OnlineEmbeddingModule
from lazyllm.tools.rag.transform import SentenceSplitter

# 第一次使用 - 自动注册
embedding_module1 = OnlineEmbeddingModule('qwen', embed_model_name='text-embedding-v1')

doc1 = Document(
    dataset_path='/path/to/data',
    embed={'my_embed': embedding_module1},  # embed_key='my_embed'
    store_conf=store_conf
)
doc1.create_node_group(name='context', transform=SentenceSplitter)
doc1.activate_group('context')  
# ↑ 在这里会自动注册: embed_key='my_embed', dimension=1536

# 第二次使用相同的 embedding - 自动校验通过
doc2 = Document(
    dataset_path='/path/to/more_data',
    embed={'my_embed': embedding_module1},  # 相同的 embed_key 和模型
    store_conf=store_conf
)
doc2.activate_group('context')  # ✓ 校验通过

# 尝试使用不同的 embedding - 自动拒绝
embedding_module2 = OnlineEmbeddingModule('qwen', embed_model_name='text-embedding-v4')
doc3 = Document(
    dataset_path='/path/to/data',
    embed={'my_embed': embedding_module2},  # 相同的 embed_key,但不同的模型!
    store_conf=store_conf
)
doc3.activate_group('context')  # ✗ 抛出维度不匹配错误
# 使用方法2:手动设置
registry = get_embedding_registry()

# 查询注册信息
model_info = registry.get('my_custom_embed')
print(f"Model: {model_info.model_name}, Dim: {model_info.dimension}")

# 手动校验
try:
    registry.validate('my_custom_embed', dimension=1024)  # 维度不匹配
except ValueError as e:
    print(f"Validation failed: {e}")

# 查看所有注册的模型
all_models = registry.get_all()
for embed_key, info in all_models.items():
    print(f"{embed_key}: {info.dimension}D - {info.model_name}")

# 取消注册
registry.unregister('my_custom_embed')

# 清空所有注册
registry.clear()

# 手动保存
registry.save()

⚠️ 注意事项 / Additional Notes

This branch had an error being deployed

1 failed deployment
protected cd638a82 Deployed Dec 1, 2025 by chenhao0205 via clone #3232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant