Embeddings: index syllabus and class data with metadata for cross-feature retrieval

[shreekardittakavi]

Spec: Doc Task Magnitude — "Class Information is parsed from the syllabus. This information is saved in vector DB, used as context for everything." Doc tech stack: "Embedded data should have metadata. Notes should be tagged as notes and such so that retrieval can be done cleanly."

Current state

Only note.embedding exists (a 768-dim pgvector column). Parsed syllabus data — parse_test_courses, parse_test_assignments, parse_test_contacts, parse_test_events, parse_test_grading_items, parse_test_concepts — is fully structured but never embedded. So the agent and the generators cannot retrieve "what does the grading policy say" or "which topics does this course cover" through the same path they use for notes.

Scope

• Embed syllabus-derived content (course overview, topic list, grading policy, policies, office hours) with metadata: { userId, sourceType: "syllabus", classId, section, chunkType }.
• Establish a single retrieval interface in lib/ that queries across note and syllabus vectors with metadata filters, so callers ask for "context for class X" rather than reaching into one table.
• userId filtering happens in the query, not in post-processing — this is an access-control boundary, not a relevance filter.
• Consumers: AskHamiz, quiz and flashcard generation, resource recommendations, and weak-topic explanations.
• Re-embed on syllabus re-parse; do not leave stale vectors from a superseded upload.

Related

• setup pincone for vector embeddings #16 (Pinecone setup) — decide pgvector vs Pinecone before building on top of this
• Add Topic-Based Retrieval and Chunked Embeddings for Flashcard Generation #57 (chunked embeddings for flashcard generation) — the chunking strategy should be shared, not per-feature

Acceptance criteria

• Syllabus content is embedded with source-type and class metadata
• One retrieval interface serves every consumer
• Retrieval cannot cross user boundaries
• Re-parsing a syllabus replaces its vectors rather than duplicating them