Driveway Watcher

Experimental -- This is a personal project under active development. Expect rough edges.

Watches a driveway RTSP camera, detects parked vehicles with YOLO, identifies make/model and license plate, and can automatically file SF311 blocked-driveway reports with human-in-the-loop approval.

Approval workflow

When a vehicle parks in the driveway, the PWA shows the detection with make/model, scene description, and Approve / Reject buttons. On approval, the SF311 report is filed automatically.

What it does

• Real-time vehicle detection via RTSP camera stream (GStreamer + YOLOv8)
• Persistent object tracking (ByteTrack) across frames for vehicles, people, and dogs
• Parking detection state machine (MOVING -> PARKING -> PARKED) using motion analysis + Gemma 3 vision model
• License plate recognition (fast-alpr)
• Vehicle make/model classification (3-model ConvNeXt ensemble)
• Driveway polygon zone detection (Shapely) to distinguish driveway vs street parking
• Email notifications when vehicles park in the driveway
• SF311 automated filing via Playwright browser automation with approval workflow
• Progressive Web App (PWA) for mobile access with live video, detection history, and approve/reject UI
• WebSocket real-time updates between CV pipeline, web server, and phone clients

Architecture

RTSP Camera
    |
GStreamer (hardware decode) -> Frame Queue
    |
YOLO + ByteTrack (detection/tracking)
    |
    +-- Per-vehicle: LPR + Make/Model classification (parallel)
    +-- Gemma 3: Scene description + parking classification
    +-- Parking detector: Motion analysis + Gemma fusion
    |
FastAPI Web Server (port 8000)
    +-- MJPEG live stream
    +-- SQLite detection history
    +-- WebSocket approval workflow
    +-- PWA frontend
    |
SF311 Submitter (Playwright, on approval)

Requirements

• Python 3.11+ (3.13 recommended)
• Apple Silicon Mac recommended (MPS acceleration for ML models)
• RTSP camera on your local network
• ~3GB disk for ML model weights (downloaded automatically on first run)
• uv package manager

Setup

1. Clone and install dependencies:

git clone <repo-url>
cd driveway-watcher
uv sync

2. Install Playwright browsers (for SF311 automation):

uv run python -m playwright install --with-deps

3. Create a .env file with your configuration:

# Camera
RTSP_URL=rtsp://user:pass@your-camera-ip:554/stream1

# Driveway zone polygon (comma-separated x,y coordinates)
DRIVEWAY_POLYGON=960,865,1920,865,1920,1295,960,1295

# Email notifications
EMAIL_ENABLED=true
SENDER_EMAIL=you@gmail.com
SENDER_PASSWORD=your-app-password
RECIPIENT_EMAIL=recipient@example.com
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587

# SF311 automation
SF311_ENABLED=true
SF311_ADDRESS="123 Your St"
SF311_NEIGHBORHOOD="Your Neighborhood"
SF311_CONTACT_NAME="Your Name"

# Web UI
APP_PASSWORD=your-password
APPROVAL_REQUIRED=true
AUTO_APPROVE_TIMEOUT=300

# CORS (comma-separated origins for remote access)
CORS_ORIGINS=http://localhost:8000

4. Model weights download automatically on first run (~3GB total for ConvNeXt ensemble + Gemma 3).

Running

./start_web.sh

This starts the GStreamer pipeline and web server. Access the web UI at http://localhost:8000. The live feed takes a few minutes to appear while the RTSP stream connects and ML models load.

The PWA can be installed on your phone for mobile monitoring. Access via your machine's local IP or a Cloudflare tunnel for remote access.

Project structure

gstreamer_web_pipeline.py   # Main entry point: GStreamer -> YOLO -> detectors -> web
web_server.py               # FastAPI backend: video streaming, API, WebSocket, auth
start_web.sh                # Launch script (sets GStreamer env vars, sources .env)

app/
  tracker.py                # ByteTrack object tracking + vehicle result cache
  parking_detector.py       # Parking state machine (motion + Gemma classification)
  dog_detector.py           # Dog detection and stopped-state tracking
  person_detector.py        # Person detection and tracking
  lpr.py                    # License plate recognition (fast-alpr)
  make_model.py             # Vehicle make/model classifier (ConvNeXt ensemble)
  scene.py                  # Scene description (BLIP / Gemma 3 multimodal)
  polygon.py                # Driveway zone polygon (Shapely)
  frame_annotator.py        # OpenCV frame annotation (bboxes, labels, states)
  email_notifier.py         # SMTP email notifications
  sf311_submitter.py        # Playwright SF311 form automation
  cv_client.py              # WebSocket client for approval workflow

static/                     # PWA frontend (vanilla JS)
deprecated/                 # Superseded code preserved for reference

SF311 Form Automation

The system can automatically file blocked-driveway reports with San Francisco's 311 service. When a vehicle is detected parked in the driveway zone:

1. Detection appears in the PWA with photo, make/model, and plate
2. User approves or rejects via the mobile UI
3. On approval, Playwright opens the SF311 form and fills in the details
4. Confirmation is sent back through the WebSocket

SF311 Configuration

Add to your .env:

SF311_ENABLED=true
SF311_ADDRESS="123 Your St"        # required
SF311_CROSS_STREET="2nd St"        # optional
SF311_NEIGHBORHOOD="South Beach"   # optional
SF311_CONTACT_NAME="Your Name"     # optional
SF311_CONTACT_EMAIL="you@example.com"  # optional
SF311_HEADLESS=true                # false to watch the browser

Testing

Set SF311_HEADLESS=false so you can watch Playwright fill the form. The browser will open visibly and you can verify each field before it submits. To do a dry run, comment out the submit click in app/sf311_submitter.py → _submit_form().

Troubleshooting

• No reports submitted: Check SF311_ENABLED=true and that vehicles are in PARKED state over the driveway polygon
• Form selectors broken: SF311 occasionally changes their form UI. Set SF311_HEADLESS=false, watch which field fails in the logs, and update selectors in app/sf311_submitter.py
• Cooldown: Same vehicle only triggers one report every 5 minutes to prevent spam
• Logs: grep "[SF311]" web_pipeline.log