Phase A: Daily OSINT ETL workflow (Overpass + open data to SQLite/JSON)

[jlucus]

Part of milestone v2.1.0 — Community Data Pipeline. Full plan: docs/02-project-scope/DATA_PIPELINE_PLAN.md.

Scope

Scheduled GitHub Action (daily) that fetches OSM Overpass, Oakland open data (Socrata), BART API, and library hours feeds; normalizes into the venues/evidence schema; dedupes; scores confidence; writes data/locations.db + public/data/locations.json; commits only on diff.

Acceptance criteria

• scripts/etl/ fetchers with conditional requests (ETag) and per-source timeouts — Overpass fetcher shipped in Phase A: daily OSINT ETL pipeline with auto-listed venues #27 (timeouts + mirror fallback + raw snapshot); ETag conditional requests and the Socrata/BART/hours-feed fetchers remain
• Canonical SQLite schema (venues, evidence) with derived publish tier (Phase A: daily OSINT ETL pipeline with auto-listed venues #27)
• Dedupe by geohash bucket + name similarity (Phase A: daily OSINT ETL pipeline with auto-listed venues #27)
• Commit-on-diff with data: prefix; CI-loop safety (Phase A: daily OSINT ETL pipeline with auto-listed venues #27 — verified byte-identical reruns)
• "Auto-listed" badge rendered on the site for machine-corroborated entries (Phase A: daily OSINT ETL pipeline with auto-listed venues #27)
• ODbL attribution shown in site footer + LICENSE-DATA (Phase A: daily OSINT ETL pipeline with auto-listed venues #27; remaining licensing docs in Phase E Phase E: Data licensing, attribution, and privacy updates #24)
• Workflow keepalive against GitHub's 60-day schedule auto-disable (Phase A: daily OSINT ETL pipeline with auto-listed venues #27 — daily data commits keep the repo active)

Remaining for close

Second corroboration source (Socrata city facilities) + ETag conditional requests, which restore the two-source auto-publish rule from the plan.