Crawler.API

A .NET 8 ASP.NET Core read API for multi-retailer PC-parts price comparison. It serves searchable product catalogs and historical price snapshots from SQL Server—not the crawler that collects that data.

Originally built as the backend for GoGeeks, an Indian PC-hardware price tracker covering retailers such as MD Computers, PrimeABGB, and Amazon. The goal was a fast, cache-friendly search API in front of a crawled product database, with Lucene for low-latency keyword filtering and SQL full-text as a fallback.

Why it exists

Price comparison only works if product search is fast and price history is easy to query. This API was extracted so the storefront could:

• Search and filter a large product catalog without hammering SQL on every request
• Return the current best price per product from the latest snapshot
• Expose price history for charts and alerts
• Generate an XML sitemap for SEO

The ingestion/crawler that writes into the Crawler database lives outside this repository.

Architecture

HTTP Client
    │
    ▼
ProductsController / SnapshotsController
    │
    ▼
ProductService / SnapshotService
    │
    ├── LuceneSearchService  (fast path when index + cache are warm)
    ├── CacheService         (in-memory product list)
    └── UnitOfWork / Repositories
            │
            ▼
        AppDbContext (EF Core) → SQL Server

Background: LuceneIndexService (hosted service)
    loads products → memory cache → on-disk Lucene index (./LuceneIndex)

Request flow (product search)

1. If the Lucene index exists and the AllProducts cache entry is present → Lucene returns matching IDs, then results are filtered and paginated from the in-memory list.
2. Otherwise → SQL Server full-text search via EF.Functions.Contains on product name.

Cross-cutting concerns

Concern	Implementation
API versioning	api/v{version}/... (default 1.0)
Rate limiting	AspNetCoreRateLimit (1000 requests / 24h per IP)
Logging	NLog → rolling files (./logs); optional Elasticsearch
Docs	Swagger UI in Development
Caching	IMemoryCache via CacheService<T>

Domain model

• Product — catalog item (name, SKU, image)
• Snapshot — price observation for a product on a Platform (link, price, timestamp)
• Watchlist — mapped in EF for future use; no public endpoints yet

API surface

Method	Route	Description
GET	/api/v1/Products	Search/list products (keywords, brands, categories, pagination)
GET	/api/v1/Products/{id}	Product by ID with best price
GET	/api/v1/Products/Sitemap	XML sitemap using SiteSettings:BaseUrl
GET	/api/v1/Products/{productId}/Snapshots	Price history for a product

Pagination metadata is returned in the X-Pagination response header.

Getting started

Prerequisites

• .NET 8 SDK
• SQL Server with the Crawler schema populated (this repo does not include migrations or seed data)
• SQL Server full-text search recommended for the database fallback path

Configure

cp Crawler.API/appsettings.Example.json Crawler.API/appsettings.Development.json
# edit ConnectionStrings:DefaultConnection and SiteSettings:BaseUrl

Or use user secrets / environment variables:

ConnectionStrings__DefaultConnection="Server=...;Database=Crawler;..."
SiteSettings__BaseUrl="https://www.your-frontend.com"

Optional Elasticsearch logging:

# PowerShell
$env:NLOG_ELASTIC_URI="http://localhost:9200"
# then uncomment the elastic logger rule in Crawler.API/nlog.config

Run

dotnet restore
dotnet build
dotnet run --project Crawler.API

Swagger (Development): https://localhost:7075/swagger

Test

dotnet test

Project layout

Path	Role
Crawler.API/	Web API
Crawler.API/Controllers/	HTTP endpoints
Crawler.API/Services/	Business logic, Lucene, cache, logging
Crawler.API/Repositories/	Data access + unit of work
Crawler.API/Data/	EF Core AppDbContext
Crawler.API/Models/	Entities, DTOs, enums
Crawler.API.Tests/	Unit tests

Tech stack

ASP.NET Core 8 · EF Core 7 · SQL Server · Lucene.NET · NLog · FluentValidation · AspNetCoreRateLimit · Swashbuckle · xUnit

Security notes

• Do not commit real connection strings or logging credentials.
• Prefer user secrets / environment variables for local and production config.
• If you forked from a private history that once contained secrets, rotate those credentials even if the current tree is clean.

License

MIT — see LICENSE.

Contributors

Thanks to everyone who helped shape this API, including collaborators on the original private project.