Web Scraping API

A web scraping API built with Node.js, Express, and Puppeteer, with user authentication backed by Supabase and a React frontend dashboard.

Features

• Headless browser scraping via Puppeteer with a browser pool
• CSS selector and XPath extraction, including attribute and multi-element support
• API key authentication per user
• Usage tracking and rate limiting
• React + Vite frontend with signup, login, and dashboard
• Supabase for auth and data storage

Prerequisites

• Node.js 18+
• A Supabase project (free tier works)

Setup

1. Clone and install

git clone <repo-url>
cd WebScrapingAPI
npm install
cd frontend && npm install && cd ..

2. Set up Supabase

Run the SQL in database/supabase-setup.sql in your Supabase project's SQL editor. This creates the required tables and RLS policies.

3. Configure environment variables

Backend — copy and fill in .env:

cp .env.example .env

Frontend — copy and fill in frontend/.env.local:

cp frontend/.env.example frontend/.env.local

See .env.example and frontend/.env.example for all required variables and where to find them in your Supabase project settings.

4. Start the servers

Backend:

npm start          # production
npm run dev        # development with auto-restart

Frontend (separate terminal):

cd frontend
npm run dev

The API runs on http://localhost:3000 and the frontend on http://localhost:5173 by default.

API Usage

All scraping requests require an API key obtained from the dashboard after signing up.

Authentication

Include your API key in the X-API-Key header:

X-API-Key: wh_api_your_key_here

Scrape endpoint

POST /api/scrape

{
  "url": "https://example.com",
  "selectors": {
    "title": {
      "selector": "h1"
    },
    "price": {
      "selector": ".price"
    },
    "links": {
      "selector": "a.nav-link",
      "multiple": true,
      "attribute": "href"
    },
    "heading_xpath": {
      "selector": "//h2[@class='main']",
      "type": "xpath"
    }
  }
}

Response:

{
  "success": true,
  "data": {
    "title": "Example Domain",
    "price": "$99.99",
    "links": ["/home", "/about"],
    "heading_xpath": "Welcome"
  },
  "metadata": {
    "url": "https://example.com",
    "timestamp": "2025-07-13T...",
    "processingTime": 1800
  }
}

Selector options:

Field	Type	Default	Description
selector	string	—	CSS selector or XPath expression
type	"css" | "xpath"	auto-detected	Force selector type
multiple	boolean	false	Return all matches as an array
attribute	string	—	Extract an attribute instead of text content

Other endpoints

Method	Path	Description
GET	/health	Server health check
GET	/api/status	API status
GET	/api/pool	Browser pool status
GET	/api/metrics	Performance metrics
POST	/api/auth/register	Register a new user
POST	/api/auth/login	Log in
GET	/api/auth/profile	Get profile and API key (requires session token)
POST	/api/auth/regenerate-api-key	Issue a new API key

curl example

export TEST_API_KEY=wh_api_your_key_here

curl -X POST http://localhost:3000/api/scrape \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $TEST_API_KEY" \
  -d '{
    "url": "https://news.ycombinator.com",
    "selectors": {
      "title": { "selector": ".titleline > a", "multiple": true },
      "points": { "selector": ".score", "multiple": true }
    }
  }'

You can also use the provided test-scraping.sh script — set TEST_API_KEY in your environment before running it.

Project Structure

.
├── server.js                  # Entry point
├── src/
│   ├── config/
│   │   ├── config.js          # App configuration
│   │   └── supabase.js        # Supabase client setup
│   ├── middleware/
│   │   ├── auth.js            # API key authentication, usage tracking
│   │   └── validation.js      # Request validation
│   ├── routes/
│   │   ├── auth.js            # Auth endpoints
│   │   └── scrape.js          # Scrape endpoint
│   ├── services/
│   │   ├── browserPool.js     # Puppeteer browser pool
│   │   ├── pooledScraper.js   # Scraping logic
│   │   └── poolMonitor.js     # Pool health monitoring
│   └── utils/
│       ├── logger.js
│       ├── selectorUtils.js   # CSS/XPath selector handling
│       └── userAgents.js      # User agent rotation
├── frontend/                  # React + Vite frontend
│   └── src/
│       ├── components/        # Dashboard, Login, Signup
│       ├── contexts/          # AuthContext
│       └── lib/supabase.ts    # Supabase client
├── database/
│   └── supabase-setup.sql     # Schema and RLS policies
├── .env.example               # Backend env template
└── frontend/.env.example      # Frontend env template

Common Issues

• Browser launch fails: Puppeteer downloads Chromium automatically on npm install. If it fails, check you have sufficient disk space and that your OS allows sandboxed processes (see --no-sandbox flag in src/config/config.js for Docker/Linux environments).
• Timeout errors: Some sites take longer to load. Increase PAGE_TIMEOUT in .env.
• Selector not found: Use your browser's DevTools to verify the selector against the live page. Dynamic content may require waiting for a specific element.
• Supabase errors on startup: Ensure all three Supabase env vars are set and that you have run the setup SQL.