Integration: Kaapi<>Glific Evals & Guardrails

[Ayush8923]

Is your feature request related to a problem?
The current integration between Kaapi and Glific lacks evaluation and guardrails features, limiting its effectiveness and resulting in potential use cases not being addressed.

Describe the solution you'd like

• Implement evals and guardrails into the Kaapi<>Glific integration.
• Ensure that the new features enhance performance and usability.
• Aim for improved maintainability and scalability.

[Ayush8923]

We have made good progress on the Kaapi ↔ Glific evaluation and Guardrails integration.

Native Judge Metrics (Langfuse-free)

We have completed the implementation of the native evaluation pipeline, which is completely Langfuse-free and does not rely on any third-party evaluation service.

Related Issues

• Evaluation: Integrate llm-as-a-judge #959
• Evaluation: Separate out v2 evaluation endpoints (Langfuse-free) #1050
• Evaluation: Metric for prompt adherence #1046
• Evals: Decouple from Langfuse #1031
• Kaapi: Adherence to knowledge base metric #1045

From a development perspective, the implementation is complete. The native judge now evaluates all three metrics:

• Adherence to Prompt
• Adherence to Ground Truth
• Adherence to Knowledge Base

Currently, we are testing the implementation with real data and iterating on the judge prompts to improve scoring quality and consistency. The primary focus now is prompt tuning.

Documentation

Native Judge SRD

---

UI/UX

UI/UX discussions for the Glific integration are currently in progress.

Once the UI is finalized, we will share the high-level UI flow for how the evaluation experience will look inside Glific.

---

Next Step

The next task is to create separate V2 API endpoints for the prompt iteration flow so that we don't need to modify the existing evaluation endpoints and can continue experimenting independently.

Related Issue

• Evaluation: Update prompt iteration flow #1072

[Ayush8923]

In this, the evaluation flow is now live in Glific Production. Sharing the Glific frontend and backend repository links here for reference:
• Frontend: https://github.com/glific/glific-frontend
• Backend: https://github.com/glific/glific

we are tracking all the improvements and ongoing work in this feedback sheet: https://docs.google.com/spreadsheets/d/1BnNlNpyzclqVqBka7Io4fYdTnlbAxO49siUC8zQI7YE/edit?usp=sharing

The Guardrails flow is still pending. we will track the Guardrails-related work and issues in this issue: #1032