Resources: aggregation pipeline over allowlisted educational sites

[shreekardittakavi]

Spec: SDR 3.8.2 Resource Aggregation — "The system may scrape or gather data from educational resource sites to provide an overview of relevant materials." Doc: "Should scrape resource sites to give an overview?" (posed as an open question in the spec)

Scope

• Aggregation over an allowlist of educational sites only. No open-web crawling.
• Respect robots.txt, set a real User-Agent, rate-limit per host, and cache aggressively — this is student coursework, not a search engine.
• Extract title, description, and topic tags; store with provenance so a stale or dead source is traceable.
• Summarize the aggregate into a topic overview rather than dumping a link list.
• Prefer official APIs and open-license repositories over HTML scraping wherever one exists.

Open questions

• The spec itself marks this as uncertain. Worth confirming with the team whether it earns its complexity, or whether the curated registry plus YouTube recommendations covers the actual need.
• Check the terms of service for each site before adding it to the allowlist.

Acceptance criteria

• Aggregation runs only against allowlisted hosts
• robots.txt and rate limits are respected
• Results are cached and attributed to their source
• A dead or changed source degrades gracefully instead of breaking the page