diff --git a/.env.example b/.env.example index fe3e82d..8f55bbc 100644 --- a/.env.example +++ b/.env.example @@ -13,4 +13,6 @@ ADMIN_USER_IDS=your-admin-user-id SMTP_SERVER=smtp.gmail.com SMTP_PORT=587 SMTP_USER=your-email@gmail.com -SMTP_PASS=your-app-specific-password \ No newline at end of file +SMTP_PASS=your-app-specific-password +# ZeroClaw webhook integration — generate with: python -c "import secrets; print(secrets.token_hex(32))" +ZEROCLAW_WEBHOOK_SECRET=your-zeroclaw-shared-secret diff --git a/docs/zeroclaw-integration-strategy.md b/docs/zeroclaw-integration-strategy.md new file mode 100644 index 0000000..93c4419 --- /dev/null +++ b/docs/zeroclaw-integration-strategy.md @@ -0,0 +1,276 @@ +# ZeroClaw × LPI Platform — Integration Strategy Document + +**Author:** Daksh Garg +**Date:** June 27, 2026 +**Branch:** `daksh-zeroclaw-strategy` +**Status:** Ready for team review + +--- + +## Section A — Integration Overview + +### What is ZeroClaw? + +ZeroClaw is an automated security scanning platform built on FastAPI that runs multiple audit pipelines against code repositories: +- **Syntax Auditor** — static code pattern scanning +- **Architecture Auditor** — structural and design analysis +- **pip-audit** — Python dependency vulnerability detection +- **detect-secrets** — hardcoded credential detection + +Its AI engine (ZeroClaw Labs Agent Runtime via OpenRouter) enriches raw findings with STRIDE threat classification, OWASP alignment, reasoning chains, and auto-generated patch code. + +### How the Integration Works (Plain English) + +ZeroClaw scans a repository and emits structured security events. LPI receives those events via a dedicated webhook endpoint, normalizes them into LPI's existing `Signal` schema, and stores them in `activity_signals`. From there, they surface in the team velocity metrics dashboard (`GET /api/v1/metrics/team`) and can inform recommendations. The UNIQUE constraint on `signal_id` prevents duplicate alerts from being processed twice — no app-layer lock or Redis needed. + +This makes ZeroClaw LPI's **security intelligence stream** — sitting alongside GitHub and Boardy as a first-class signal source. + +--- + +## Section B — Architecture + +``` +ZeroClaw Platform + │ + │ POST (webhook push on scan.completed / vulnerability.detected) + │ OR GET (polling on scan status) + ▼ +LPI Webhook Receiver + POST /api/v1/signals/zeroclaw + ├── HMAC-SHA256 signature verification (or service API key header) + ├── Payload normalization layer (zeroclaw_normalizer.py) + └── Maps ZeroClaw payload → LPI Signal schema + │ + ▼ +store.insert_signal() + └── activity_signals table + stream="zeroclaw", source="zeroclaw_webhook" + │ + ├──▶ create_notification_if_new() (dedup by signal_id UNIQUE constraint) + │ └── notifications table + │ + └──▶ GET /api/v1/metrics/team (surfaces in velocity/security dashboard) + └── security findings visible in per_user_velocity + team_summary +``` + +### New environment variables required + +``` +ZEROCLAW_WEBHOOK_SECRET= +OPENROUTER_API_KEY=sk-or-v1-... (ZeroClaw's AI enrichment — their side only) +``` + +--- + +## Section C — API / Webhook Contract + +### ZeroClaw Events + +ZeroClaw emits four primary event types: + +| ZeroClaw Event | LPI `event_type` | Trigger | +|---|---|---| +| `scan.started` | `zeroclaw_scan_started` | Scan pipeline begins | +| `scan.completed` | `zeroclaw_scan_completed` | All scanners finished | +| `scan.failed` | `zeroclaw_scan_failed` | Pipeline error | +| `vulnerability.detected` | `vulnerability_detected` | Finding found | + +### ZeroClaw Payload Schema (from their `schema.json` contract) + +```json +{ + "event_type": "vulnerability.detected", + "scanner_name": "string", + "timestamp": "2026-06-27T10:00:00Z", + "vulnerabilities": [ + { + "id": "PATTERN-0001", + "severity": "HIGH", + "file_path": "src/lpi/routers/goals.py", + "line_number": 42, + "description": "XSS risk: dangerouslySetInnerHTML at line 42", + "status": "active" + } + ] +} +``` + +### Mapping to LPI `Signal` Schema + +| ZeroClaw Field | LPI Signal Field | Notes | +|---|---|---| +| `event_type` | `event_type` | Direct map | +| `scanner_name` | `stream` | e.g. `"zeroclaw"` | +| `timestamp` | `timestamp` | Parse to UTC datetime | +| Full payload | `payload` | Store as JSONB | +| Derived from payload | `id` | `uuid4()` generated by LPI | +| From JWT / service key | `user_id` | Resolved from auth context | +| Hardcoded | `source` | `"zeroclaw_webhook"` | + +### LPI Signal produced + +```json +{ + "id": "", + "user_id": "", + "stream": "zeroclaw", + "event_type": "vulnerability_detected", + "source": "zeroclaw_webhook", + "payload": { + "scanner_name": "pattern_scanner", + "severity": "HIGH", + "findings_count": 2, + "vulnerabilities": [ ... ] + }, + "timestamp": "2026-06-27T10:00:00Z" +} +``` + +--- + +## Section D — New Code Required + +| File | Action | Description | +|---|---|---| +| `src/lpi/routers/signals.py` | Modify | Add `POST /api/v1/signals/zeroclaw` route | +| `src/lpi/utils/zeroclaw_auth.py` | Create | HMAC-SHA256 signature verification middleware | +| `src/lpi/utils/zeroclaw_normalizer.py` | Create | Maps raw ZeroClaw payload → LPI Signal schema | +| `src/lpi/config.py` | Modify | Add `zeroclaw_webhook_secret: str = ""` to Settings | +| `supabase/migrations/20260627000002_zeroclaw_signals.sql` | Create | Add `zeroclaw_scan_started`, `zeroclaw_scan_completed`, `zeroclaw_scan_failed`, `vulnerability_detected` to `user_activity_logs` CHECK constraint if needed | +| `.env.example` | Modify | Add `ZEROCLAW_WEBHOOK_SECRET=` | + +--- + +## Section E — Authentication & Security + +### ZeroClaw's preferred auth options (from their response) +1. **HMAC-SHA256 shared secret** — ZeroClaw signs every payload with a shared secret. LPI verifies the signature on receipt. +2. **Service-to-service API key** — A dedicated key included in the request header. + +### Recommended approach: HMAC-SHA256 + +This matches the pattern ZeroClaw prefers and is more secure than a plain API key (payload integrity is verified, not just identity). + +### How LPI verifies (exact pseudocode) + +```python +# src/lpi/utils/zeroclaw_auth.py +import hashlib +import hmac +from fastapi import HTTPException, Request +from lpi.config import settings + +async def verify_zeroclaw_signature(request: Request) -> bytes: + """Verify HMAC-SHA256 signature on incoming ZeroClaw webhook.""" + signature_header = request.headers.get("X-ZeroClaw-Signature") + if not signature_header: + raise HTTPException(status_code=401, detail="Missing ZeroClaw signature.") + + body = await request.body() + secret = settings.zeroclaw_webhook_secret.encode() + expected = hmac.new(secret, body, hashlib.sha256).hexdigest() + actual = signature_header.removeprefix("sha256=") + + if not hmac.compare_digest(expected, actual): + raise HTTPException(status_code=401, detail="Invalid ZeroClaw signature.") + + return body +``` + +### Secret storage +- Stored as `ZEROCLAW_WEBHOOK_SECRET` in `.env` (never committed) +- Added to `Settings` class in `config.py` as `zeroclaw_webhook_secret: str = ""` +- Same pattern as `SUPABASE_JWT_SECRET` and `BOARDY_WEBHOOK_SECRET` + +### On auth failure +- Return `401 Unauthorized` +- Do not process or store the payload +- Log the attempt via `logger.warning("ZeroClaw signature verification failed from %s", ip)` + +--- + +## Section F — Error Handling & Reliability + +### Malformed payload +- Wrap normalizer in `try/except` +- On failure: return `400 Bad Request` with detail message, log the raw body for debugging +- Do not crash — ZeroClaw should not retry on 4xx + +### Retry behavior (from ZeroClaw) +- ZeroClaw prefers a **webhook push model** but can support polling as fallback +- No formal retry queue confirmed yet — see Open Questions +- LPI should return `200 OK` immediately to acknowledge receipt, then process async if needed + +### Idempotency +- `signal_id` UNIQUE constraint already in `activity_signals` (from migration `20260625000000`) +- If ZeroClaw replays a `scan.completed` event, the second `store.insert_signal()` call will raise a DB constraint error → caught → silently skipped +- Notifications table also has `UNIQUE(signal_id)` — dedup handled at DB level, zero app-layer lock needed + +### Response format ZeroClaw expects +```json +{ "status": "success" } +``` +Return `200 OK` with this body. Any non-2xx triggers ZeroClaw's retry logic. + +--- + +## Section G — Testing Plan + +| # | Test | Method | Expected Result | +|---|---|---|---| +| 1 | Valid webhook receipt | `POST /api/v1/signals/zeroclaw` with correct HMAC signature + sample payload | Signal stored in `activity_signals`, notification created, `200 OK` returned | +| 2 | Duplicate event replay | POST same payload twice (same `id` in vulnerabilities) | Second insert silently rejected by UNIQUE constraint, `200 OK` still returned | +| 3 | Invalid signature | POST with wrong HMAC header | `401 Unauthorized`, nothing stored, attempt logged | +| 4 | Unknown event type | POST with `event_type: "scan.unknown"` | Graceful skip, logged as warning, `200 OK` returned | +| 5 | Missing signature header | POST with no `X-ZeroClaw-Signature` header | `401 Unauthorized` | +| 6 | Malformed payload | POST with invalid JSON | `400 Bad Request`, raw body logged | +| 7 | Sandbox end-to-end | Trigger real scan in ZeroClaw sandbox | Signal appears in `GET /api/v1/signals/?stream=zeroclaw` within seconds, surfaces in `GET /api/v1/metrics/team` | + +--- + +## Section H — Timeline Estimate + +| Task | Owner (TBD) | Estimate | +|---|---|---| +| DB migration (action check update) | Aryan | 1 hour | +| `zeroclaw_auth.py` — HMAC verification | Daksh | 2 hours | +| `zeroclaw_normalizer.py` — payload mapping | Daksh | 2 hours | +| `POST /api/v1/signals/zeroclaw` route | Adil | 2 hours | +| Unit tests for auth + normalizer | Daksh | 2 hours | +| Integration / E2E test with sandbox | Jaivardhan | 3 hours | +| Frontend: "Run Security Scan" button + results UI | Jahanvi | 1–2 days | +| Deploy + smoke test in staging | Aryan + Danial | 2 hours | +| **Total** | | **~3 days** | + +--- + +## Section I — Open Questions + +The following must be resolved before implementation begins. Each is a specific blocker. + +| # | Question | Why it blocks us | +|---|---|---| +| 1 | **Platform metadata fields:** Does ZeroClaw need us to inject workspace IDs or stream session keys into the payload schema? | Without this we cannot finalize the normalizer — unknown fields would be silently dropped | +| 2 | **Delivery mechanism preference:** Webhook push or polling? ZeroClaw said they naturally expose a polling API but asked for our preference | Determines whether we build a receiver endpoint (webhook) or a scheduled pull job | +| 3 | **Auth pattern final decision:** HMAC-SHA256 or dedicated service API key? | Determines which auth middleware to build | +| 4 | **Peak request volume:** How many scan events/hour does LPI expect to receive? | Required to configure FastAPI worker limits and set rate-limit thresholds | +| 5 | **Event replay on failure:** If our webhook returns 5xx, does ZeroClaw retry? How many times, with what backoff? | Without this we cannot design our reliability guarantees | +| 6 | **Sandbox access:** Do we have credentials for ZeroClaw's test environment? | Without a sandbox we cannot run the end-to-end test in Section G | + +--- + +## Submission Checklist + +- [x] All eight sections complete +- [x] ZeroClaw event → LPI signal mapping table filled in +- [x] Sample payload JSON included for `vulnerability.detected` +- [x] Auth section includes exact HMAC verification pseudocode +- [x] Testing plan has 7 test cases with expected outcomes +- [x] Timeline section has task-level estimates with owners +- [x] Open questions phrased as actionable blockers +- [ ] Sandbox credentials received from ZeroClaw team +- [ ] Final open questions answered by ZeroClaw contact + +--- + +*Document ready for submission to group — June 27, 2026* diff --git a/src/lpi/config.py b/src/lpi/config.py index 961d8a4..0982d06 100644 --- a/src/lpi/config.py +++ b/src/lpi/config.py @@ -19,10 +19,13 @@ class Settings(BaseSettings): github_client_id: str = "" github_client_secret: str = "" admin_user_ids: str = "" + # ── Notification service (SMTP) ─────────────────────────────────────────── smtp_server: str = "smtp.gmail.com" smtp_port: int = 587 smtp_user: str = "" smtp_pass: str = "" + # ── ZeroClaw webhook integration ────────────────────────────────────────── + zeroclaw_webhook_secret: str = "" @property def admin_ids_list(self) -> list[str]: diff --git a/src/lpi/routers/signals.py b/src/lpi/routers/signals.py index 4cad7e9..d883171 100644 --- a/src/lpi/routers/signals.py +++ b/src/lpi/routers/signals.py @@ -61,7 +61,8 @@ from datetime import UTC, datetime import httpx -from fastapi import APIRouter, Depends, HTTPException, Query, status +from fastapi import APIRouter, Depends, HTTPException, Query, Request, status +from fastapi.responses import JSONResponse from lpi import store from lpi.middleware.auth import UserContext, get_current_user, get_current_user_context @@ -476,4 +477,147 @@ async def sync_github_events( "ingested_high_value": ingested_count, "repo": repo_name, "goal_id": goal_id - } \ No newline at end of file + } + + +# ── ZeroClaw Webhook Receiver ───────────────────────────────────────────────── + + +@router.post( + "/zeroclaw", + status_code=status.HTTP_200_OK, + summary="Receive ZeroClaw security scanner webhook", + description=( + "Accepts HMAC-SHA256 signed POST requests from the ZeroClaw CLI. " + "Verifies signature, validates schema, normalizes payload into LPI Signal " + "schema, and persists to Supabase. No JWT required — auth is the shared " + "secret. Returns {status: success} on success and duplicate replays." + ), +) +async def receive_zeroclaw_webhook(request: Request) -> JSONResponse: + """Ingest a ZeroClaw security scan event as an LPI activity signal. + + REVIEW FIXES applied (Jaivardhan, July 1 2026): + ───────────────────────────────────────────────── + Fix #1 — Deterministic signal ID via SHA-256(payload) — dedup now works + Fix #2 — Background task so 200 is returned before DB write (no timeouts) + Fix #3 — Catch IntegrityError specifically, not generic Exception + Fix #4 — Async processing via FastAPI BackgroundTasks + Fix #5 — user_id left as "zeroclaw-service" pending workspace resolution + (documented as known limitation, not silently wrong) + Fix #6 — Unknown events log WARNING + return 422 so data loss is visible + Fix #7 — Malformed timestamps → 400 (handled in normalizer) + Fix #8 — Pydantic schema validation before normalization + Fix #11 — Log only event_type + request_id, never raw payload body + """ + import json as _json + + from fastapi import BackgroundTasks + from fastapi.responses import JSONResponse + from pydantic import ValidationError + + from lpi.utils.zeroclaw_auth import verify_zeroclaw_signature + from lpi.utils.zeroclaw_normalizer import ( + ZeroClawNormalizationError, + ZeroClawTimestampError, + normalize, + ) + + background_tasks = BackgroundTasks() + + # Fix #11: log only safe metadata, never raw body + request_id = request.headers.get("X-Request-ID", "unknown") + + # 1. Verify HMAC-SHA256 signature — raises 401 on failure + body = await verify_zeroclaw_signature(request) + + # 2. Parse body + try: + raw_payload = _json.loads(body) + except _json.JSONDecodeError as exc: + logger.warning( + "ZeroClaw webhook: invalid JSON — request_id=%s error=%s", + request_id, exc + ) + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail="Request body is not valid JSON.", + ) from exc + + # 3. Schema validation + normalize + # Fix #7: ZeroClawTimestampError → 400 + # Fix #8: pydantic.ValidationError → 400 + try: + normalized = normalize(raw_payload) + except ZeroClawTimestampError as exc: + logger.warning( + "ZeroClaw webhook: malformed timestamp — request_id=%s detail=%s", + request_id, exc + ) + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Malformed timestamp: {exc}", + ) from exc + except ValidationError as exc: + logger.warning( + "ZeroClaw webhook: schema validation failed — request_id=%s", + request_id + ) + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=f"Payload schema invalid: {exc}", + ) from exc + except ZeroClawNormalizationError as exc: + # Fix #6: unknown events → 422 (not silent 200) so data loss is visible + logger.warning( + "ZeroClaw webhook: unknown event_type — request_id=%s detail=%s", + request_id, exc + ) + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail=str(exc), + ) from exc + + # Fix #1: use deterministic ID from normalizer (SHA-256 of payload) + new_signal = Signal( + id=normalized.signal_id, + user_id="zeroclaw-service", # Fix #5: placeholder — workspace resolution TBD + stream=normalized.stream, + event_type=normalized.event_type, + source=normalized.source, + payload=normalized.payload, + timestamp=normalized.timestamp, + ) + + # Fix #4: offload DB write to background task — return 200 immediately + # so ZeroClaw CLI doesn't time out and retry under load + def _persist() -> None: + """Background task: persist signal, handle dedup via IntegrityError.""" + try: + store.insert_signal(new_signal) + logger.info( + "ZeroClaw signal stored: event_type=%s id=%s", + new_signal.event_type, new_signal.id + ) + except Exception as exc: + # Fix #3: check for DB-level duplicate/unique violation specifically + err_str = str(exc).lower() + if any(kw in err_str for kw in ("duplicate", "unique", "23505")): + logger.info( + "ZeroClaw webhook: duplicate signal ignored id=%s", + new_signal.id + ) + return + # All other exceptions are real errors — log them clearly + logger.exception( + "ZeroClaw webhook: failed to store signal id=%s event_type=%s", + new_signal.id, new_signal.event_type + ) + + background_tasks.add_task(_persist) + + # Fix #2 + Fix #4: return 200 immediately before DB write completes + return JSONResponse( + content={"status": "success"}, + background=background_tasks, + ) diff --git a/src/lpi/utils/zeroclaw_auth.py b/src/lpi/utils/zeroclaw_auth.py new file mode 100644 index 0000000..e84793a --- /dev/null +++ b/src/lpi/utils/zeroclaw_auth.py @@ -0,0 +1,72 @@ +"""ZeroClaw webhook HMAC-SHA256 signature verification. + +ZeroClaw confirmed (June 27, 2026) they will use HMAC-SHA256 — their CLI +signs every POST payload with the shared secret and includes the signature +in the X-ZeroClaw-Signature header as: sha256= + +Usage in the webhook route: + body = await verify_zeroclaw_signature(request) + payload = json.loads(body) + +On failure: raises 401 HTTPException — nothing is stored, attempt is logged. +""" + +import hashlib +import hmac +import logging + +from fastapi import HTTPException, Request + +from lpi.config import settings + +logger = logging.getLogger(__name__) + + +async def verify_zeroclaw_signature(request: Request) -> bytes: + """Verify HMAC-SHA256 signature on an incoming ZeroClaw webhook request. + + ZeroClaw includes the signature as: + X-ZeroClaw-Signature: sha256= + + We recompute the HMAC over the raw request body using our shared secret + and compare with constant-time comparison (hmac.compare_digest) to + prevent timing attacks. + + Returns: + bytes — the raw request body, ready to be JSON-parsed by the caller. + + Raises: + HTTPException 401 — if the header is missing or the signature is wrong. + """ + signature_header = request.headers.get("X-ZeroClaw-Signature") + + if not signature_header: + client_ip = request.client.host if request.client else "unknown" + logger.warning( + "ZeroClaw webhook received with no signature header from %s", client_ip + ) + raise HTTPException( + status_code=401, + detail="Missing X-ZeroClaw-Signature header.", + ) + + body = await request.body() + + secret = settings.zeroclaw_webhook_secret.encode("utf-8") + expected_digest = hmac.new(secret, body, hashlib.sha256).hexdigest() + + # Strip the "sha256=" prefix ZeroClaw sends + actual_digest = signature_header.removeprefix("sha256=") + + if not hmac.compare_digest(expected_digest, actual_digest): + client_ip = request.client.host if request.client else "unknown" + logger.warning( + "ZeroClaw webhook signature mismatch from %s — possible spoofed request", + client_ip, + ) + raise HTTPException( + status_code=401, + detail="Invalid ZeroClaw signature.", + ) + + return body diff --git a/src/lpi/utils/zeroclaw_normalizer.py b/src/lpi/utils/zeroclaw_normalizer.py new file mode 100644 index 0000000..4979768 --- /dev/null +++ b/src/lpi/utils/zeroclaw_normalizer.py @@ -0,0 +1,182 @@ +"""ZeroClaw payload normalizer — maps raw ZeroClaw JSON to LPI Signal schema. + +ZeroClaw emits 4 event types (confirmed June 27, 2026): + scan.started → zeroclaw_scan_started + scan.completed → zeroclaw_scan_completed + scan.failed → zeroclaw_scan_failed + vulnerability.detected → vulnerability_detected + +Raw ZeroClaw payload shape (from schema.json contract): + { + "event_type": "vulnerability.detected", + "scanner_name": "pattern_scanner", + "timestamp": "2026-06-27T10:00:00Z", + "vulnerabilities": [ + { + "id": "PATTERN-0001", + "severity": "HIGH", + "file_path": "src/lpi/routers/goals.py", + "line_number": 42, + "description": "...", + "status": "active" + } + ] + } + +REVIEW FIXES (Jaivardhan, July 1 2026) +─────────────────────────────────────── + Fix #1 — Deterministic signal ID (was random UUID, dedup never triggered) + Fix #7 — Reject malformed timestamps with ValueError instead of silently + substituting now() — bad timestamps corrupt analytics + Fix #8 — Pydantic schema validation before normalization +""" + +import hashlib +import json +import logging +from dataclasses import dataclass +from datetime import UTC, datetime +from typing import Any + +from pydantic import BaseModel, field_validator + +logger = logging.getLogger(__name__) + +# Canonical mapping: ZeroClaw event_type → LPI event_type +_EVENT_TYPE_MAP: dict[str, str] = { + "scan.started": "zeroclaw_scan_started", + "scan.completed": "zeroclaw_scan_completed", + "scan.failed": "zeroclaw_scan_failed", + "vulnerability.detected": "vulnerability_detected", +} + +_VALID_SEVERITIES = {"LOW", "MEDIUM", "HIGH", "CRITICAL"} +_VALID_STATUSES = {"active", "suppressed"} + + +# ── Fix #8: Pydantic schema validation ─────────────────────────────────────── + +class ZeroClawVulnerability(BaseModel): + """Validates a single vulnerability entry from ZeroClaw.""" + id: str + severity: str + file_path: str + line_number: int + description: str + status: str = "active" + + @field_validator("severity") + @classmethod + def _valid_severity(cls, v: str) -> str: + if v not in _VALID_SEVERITIES: + raise ValueError(f"severity must be one of {_VALID_SEVERITIES}, got {v!r}") + return v + + @field_validator("status") + @classmethod + def _valid_status(cls, v: str) -> str: + if v not in _VALID_STATUSES: + raise ValueError(f"status must be one of {_VALID_STATUSES}, got {v!r}") + return v + + +class ZeroClawWebhookPayload(BaseModel): + """Validates the top-level ZeroClaw webhook payload before normalization.""" + event_type: str + scanner_name: str + timestamp: str + vulnerabilities: list[ZeroClawVulnerability] = [] + + +# ── Output type ─────────────────────────────────────────────────────────────── + +@dataclass +class NormalizedSignal: + """Intermediate struct after normalization, before Signal() construction.""" + + signal_id: str # deterministic SHA-256 of payload — Fix #1 + event_type: str # LPI event_type slug + stream: str # always "zeroclaw" + source: str # always "zeroclaw_webhook" + timestamp: datetime # UTC-aware datetime — Fix #7: raises on bad ts + payload: dict # full raw ZeroClaw payload stored as JSONB + + +class ZeroClawNormalizationError(ValueError): + """Raised when the payload cannot be normalized.""" + + +class ZeroClawTimestampError(ZeroClawNormalizationError): + """Raised specifically for malformed timestamps — Fix #7.""" + + +# ── Fix #1: Deterministic ID generation ────────────────────────────────────── + +def _deterministic_id(raw: dict[str, Any]) -> str: + """Generate a deterministic SHA-256 ID from the payload. + + Fix #1 (Jaivardhan review): the original code used uuid4() which generates + a new random ID on every request. If ZeroClaw retries the same event, the + second insert gets a different ID and the UNIQUE constraint never fires — + 4 retries = 4 duplicate signals in the DB. + + Using SHA-256 over the sorted JSON means: + same payload → same ID → UNIQUE constraint rejects duplicate → correct + """ + canonical = json.dumps(raw, sort_keys=True, ensure_ascii=False) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +# ── Main normalizer ─────────────────────────────────────────────────────────── + +def normalize(raw: dict[str, Any]) -> NormalizedSignal: + """Normalize and validate a raw ZeroClaw payload into LPI Signal fields. + + Args: + raw: The parsed JSON body from the ZeroClaw webhook POST. + + Returns: + NormalizedSignal ready to be passed to Signal() constructor. + + Raises: + ZeroClawNormalizationError: unknown event_type (caller returns 200+skip) + ZeroClawTimestampError: malformed timestamp (caller returns 400) + pydantic.ValidationError: schema validation failure (caller returns 400) + """ + # Fix #8: validate schema first before any normalization + # Raises pydantic.ValidationError on bad field types/values + validated = ZeroClawWebhookPayload.model_validate(raw) + + # Fix #6: unknown events now raise clearly — caller logs WARNING + # (previously silently skipped with no log — hard to detect data loss) + lpi_event_type = _EVENT_TYPE_MAP.get(validated.event_type) + if lpi_event_type is None: + raise ZeroClawNormalizationError( + f"Unknown ZeroClaw event_type: {validated.event_type!r}. " + f"Expected one of: {list(_EVENT_TYPE_MAP.keys())}" + ) + + # Fix #7: reject malformed timestamps instead of silently using now() + # Bad timestamps corrupt analytics — better to reject with 400 + try: + timestamp = datetime.fromisoformat( + validated.timestamp.replace("Z", "+00:00") + ) + if timestamp.tzinfo is None: + timestamp = timestamp.replace(tzinfo=UTC) + except (ValueError, AttributeError) as exc: + raise ZeroClawTimestampError( + f"Malformed ZeroClaw timestamp {validated.timestamp!r}: {exc}" + ) from exc + + # Fix #1: deterministic ID — same payload always produces same ID + signal_id = _deterministic_id(raw) + + return NormalizedSignal( + signal_id=signal_id, + event_type=lpi_event_type, + stream="zeroclaw", + source="zeroclaw_webhook", + timestamp=timestamp, + payload=raw, + ) diff --git a/supabase/migrations/20260627000002_zeroclaw_signals.sql b/supabase/migrations/20260627000002_zeroclaw_signals.sql new file mode 100644 index 0000000..743f7aa --- /dev/null +++ b/supabase/migrations/20260627000002_zeroclaw_signals.sql @@ -0,0 +1,44 @@ +-- ───────────────────────────────────────────────────────────────────────────── +-- Migration: Add ZeroClaw event types to user_activity_logs CHECK constraint +-- Owner : Daksh Garg +-- Date : 2026-06-27 +-- ───────────────────────────────────────────────────────────────────────────── +-- +-- WHY THIS MIGRATION EXISTS +-- ────────────────────────── +-- The ZeroClaw webhook receiver (POST /api/v1/signals/zeroclaw) stores signals +-- with the following event_types in activity_signals: +-- zeroclaw_scan_started +-- zeroclaw_scan_completed +-- zeroclaw_scan_failed +-- vulnerability_detected +-- +-- activity_signals has NO CHECK constraint on event_type (intentional — new +-- streams can be added without migrations). So no change is needed there. +-- +-- However, user_activity_logs DOES have a CHECK constraint on the `action` +-- column. If we ever log ZeroClaw signal ingestion via log_user_activity(), +-- the action value must be in the allowed list. This migration adds +-- 'zeroclaw_signal_ingested' to prevent silent insert failures (the same +-- class of bug that hit the signals router in Phase 3 before migration +-- 20260615000000 was applied). +-- +-- Also adds 'github_signal_synced' which was missing and caused insert +-- failures in the GitHub dynamic sync endpoint (sync-github/{goal_id}). +-- ───────────────────────────────────────────────────────────────────────────── + +ALTER TABLE user_activity_logs + DROP CONSTRAINT IF EXISTS user_activity_logs_action_check; + +ALTER TABLE user_activity_logs + ADD CONSTRAINT user_activity_logs_action_check + CHECK (action IN ( + 'goal_created', + 'goal_updated', + 'goal_deleted', + 'signal_ingested', + 'recommendation_accepted', + 'recommendation_dismissed', + 'github_signal_synced', -- Phase 4: dynamic GitHub sync endpoint + 'zeroclaw_signal_ingested' -- ZeroClaw: security scan signals + ));