docsync

A documentation crawling and synchronization engine built with Python, Crawlee, Playwright, and uv.

docsync crawls documentation websites, extracts clean Markdown content, and maintains incremental synchronization state.

It is designed for:

• Documentation backups
• Offline documentation archives
• AI dataset preparation
• Knowledge base generation
• Internal documentation mirrors
• Automated documentation synchronization

---

Features

Feature	Support
Python	3.13+
Crawlee Python	1.9.1
HTTP crawling	Yes
Playwright crawling	Yes
BeautifulSoup extraction	Yes
Chromium support	Yes
Firefox support	Yes
WebKit support	Yes
Sitemap discovery	Yes
Incremental synchronization	Yes
Request rate limiting	Yes
Concurrency control	Yes
Markdown conversion	Yes
Language-aware crawling	Yes
CLI application	Yes

---

Architecture

flowchart TD

A[CLI Entry Point] --> B[Configuration Loader]

B --> C[Crawler Runtime]

C --> D[HTTP Crawler]
C --> E[Playwright Crawler]

D --> F[HTML Extraction]
E --> F[HTML Extraction]

F --> G[Language Detection]

G --> H[Markdown Converter]

H --> I[Output Storage]

C --> J[Request Queue]

C --> K[Synchronization State]

Loading

---

Installation

Requirements

Before installation:

• Git
• Python 3.13+
• uv package manager

---

Install uv

Linux / macOS

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows PowerShell

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Verify:

uv --version

---

Install docsync from GitHub

Clone repository:

git clone https://github.com/USERNAME/docsync.git
cd docsync

Install project:

uv sync

This automatically installs:

• docsync dependencies
• Crawlee Python
• BeautifulSoup crawler support
• Playwright crawler support
• curl impersonation support
• Required Python packages

---

Crawlee Installation

docsync uses Crawlee for Python as the crawling engine.

No separate Crawlee application is required.

Crawlee is installed automatically with:

uv sync

Verify:

uv run python -c "import crawlee; print(crawlee.__version__)"

Expected:

1.9.1

---

Playwright Browser Installation

Playwright browser binaries are installed separately.

For Chromium:

uv run playwright install chromium

Supported browsers:

• Chromium
• Firefox
• WebKit

---

Verify Installation

Run:

uv run docsync --help

If the CLI help appears, installation is complete.

---

Quick Start

HTTP Crawl

For static documentation:

uv run docsync https://example.com/docs

---

Playwright Crawl

For JavaScript-rendered websites:

uv run docsync \
  https://example.com/docs \
  --mode playwright \
  --browser-type chromium

---

Crawler Modes

HTTP Mode

Fast crawling mode for static websites.

uv run docsync URL --mode http

---

Playwright Mode

Browser automation mode for JavaScript applications.

uv run docsync URL \
  --mode playwright \
  --browser-type chromium

---

Language Support

Supported languages:

Language	Code
English	en
Turkish	tr

Example:

uv run docsync \
  https://example.com/docs \
  --language tr

---

CLI Configuration

Option	Description
--output-dir	Markdown output directory
--state-dir	Synchronization state directory
--max-concurrency	Maximum parallel requests
--max-requests	Request limit
--requests-per-minute	Rate limit
--language	Target language
--refresh-hours	Refresh interval
--mode	HTTP or Playwright
--browser-type	Browser engine

Example:

uv run docsync \
  https://example.com/docs \
  --output-dir ./output \
  --state-dir ./storage \
  --max-concurrency 4 \
  --max-requests 5000 \
  --requests-per-minute 60 \
  --language en \
  --mode playwright \
  --browser-type chromium

---

Project Structure

docsync/

├── src/
│   └── docsync/
│       ├── cli.py
│       ├── crawler.py
│       ├── crawler_runtime.py
│       ├── config.py
│       ├── inventory.py
│       ├── language.py
│       ├── sitemap.py
│       └── models.py
│
├── tests/
│
├── output/
│
├── storage/
│
├── pyproject.toml
│
└── uv.lock

---

Output Structure

Generated Markdown:

output/

└── pages/
    ├── index.md
    ├── getting-started.md
    └── api-reference.md

Synchronization data:

storage/

The state directory keeps crawl progress and synchronization metadata.

---

Development

Install development environment:

uv sync

Run tests:

uv run pytest -q

Format:

uv run ruff format .

Lint:

uv run ruff check .

Type checking:

uv run mypy .

---

Build Release Package

Build package:

uv build

Install locally:

uv tool install .

Verify:

docsync --help

---

Global CLI Installation

After publishing a GitHub Release:

uv tool install docsync

Then docsync can run from any directory:

docsync https://example.com/docs

No project folder is required after installation.

---

Technology Stack

Technology	Purpose
Python	Application runtime
Crawlee	Web crawling engine
Playwright	Browser automation
BeautifulSoup	HTML parsing
Pydantic	Data validation
uv	Dependency management
Ruff	Code quality
MyPy	Static typing
Pytest	Testing

---

License

MIT License