A cross-platform (Windows / Linux / macOS) Wikidot farm-wiki archiving tool, powered by Go. It produces backups that are compatible with the WikiComma archive format — the same directory layout and byte-for-byte the same file contents. The backup can be directly imported for use with the ProjectWikit Engine.
wikit captures essentially everything a Wikidot wiki exposes:
- Domain
- Global site ID
- Slug (lowercase unix name)
- Home page
- Language
- Global page ID
- Name
- Title
- Rating
- Tags
- Parent page
- Forum discussion thread ID
- Lock status
- Last-updated timestamp
- Full revision list
- Votes and voters (per-user up/down)
- Attached files list
- Global file ID
- Name
- Original URL
- Size (human-readable and in bytes)
- MIME type and reported content type
- Author (numeric ID)
- Upload timestamp
- The raw file bytes (
files/<page>/<file_id>)
- Global revision ID
- Per-page revision number
- Timestamp
- Author (numeric ID)
- Change flags
- Commentary
- The full wiki source text of every revision (compressed in
pages/<name>.7z)
- Title, description, global ID
- Time of last post
- Total posts and threads
- Last poster (numeric ID)
- Global ID, title, description
- Created when / by whom
- Last post time / last poster
- Post count, sticky flag, lock status
- Full nested post tree
- Global ID, title, author, timestamp
- Last edit time / last editor
- Every post revision's HTML content (compressed in
forum/<cat>/<thread>.7z) - Nested replies (recursive tree)
- Display name and username slug
- Account creation date
- Account type (e.g. Pro)
- Activity / karma level
- (Users are bucketed by
id >> 13.)
Linux / macOS
curl -fsSL https://raw.githubusercontent.com/kakushi-w/wikit/main/install.sh | sh
Windows (PowerShell)
irm https://raw.githubusercontent.com/kakushi-w/wikit/main/install.ps1 | iex
Open a new terminal, then run wikit.
Prefer to do it by hand? Download a binary from the
Releases page and run
wikit install once.
wikit backup all # back up every wiki in config.json
wikit backup <name> [name...] # back up specific wikis, Separate multiple wikis with spaces
# a name not in the config is fetched from
# https://<name>.wikidot.com
-c, --config <path> config file (default ./config.json or $WIKIT_CONFIG)
--base-dir <path> override base_directory
--bucket-size <n> ratelimit bucket size
--refill-seconds <n> ratelimit refill seconds
--delay-ms <n> delay between jobs (ms)
--max-jobs <n> maximum simultaneous wikis
--user-cache <n> user-info cache freshness (seconds)
--http-proxy <s> http proxy: host:port or host:port:user:password
--socks-proxy <s> socks proxy: host:port
--no-update-check do not check for a newer wikit release
--refresh-votes after backup, bulk-refresh page ratings/votes
--scheme <s> default scheme for wikis whose url omits one (default https)
--keep-removed keep pages that disappeared from the sitemap
The normal scan can't see rating/vote changes (they don't bump a page's
timestamp), so stored ratings go stale. --refresh-votes updates them after the
backup, rewriting only page metadata:
wikit backup scp-wiki --refresh-votes
wikit update # download and install the latest release
wikit update --check # only report whether a newer version exists
wikit version # print the installed version
After each backup run, wikit also does a cached (once-a-day) check and prints a
one-line notice if a newer version is available — disable with
--no-update-check or WIKIT_NO_UPDATE_CHECK=1.
config.json format:
{
"base_directory": "/data",
"wikis": [ { "name": "scp-wiki", "url": "https://scp-wiki.wikidot.com" } ],
"ratelimit": { "bucket_size": 60, "refill_seconds": 60 },
"delay_ms": 200,
"user_list_cache_freshness": 86400,
"http_proxy": null,
"socks_proxy": null,
"refresh_votes": false,
"scheme": "https",
"keep_removed": false
}refresh_votes, scheme and keep_removed can be set here or overridden
per-run with the matching flags (the flag wins). keep_removed keeps pages
deleted from the wiki instead of removing them from the local archive.
Each wiki's url is optional (omit it to derive https://<name>.wikidot.com)
and sets that wiki's protocol: a url written with http:// or https:// is
used as-is, so wikis can mix protocols. scheme is only the default for wikis
whose url omits one (and for bare command-line names); set it to http when
your default wiki is HTTP-only.
"wikis": [
{ "name": "a" }, // https://a.wikidot.com
{ "name": "b", "url": "http://b.example.com" } // http, custom domain
]A config file is only required for backup all, which reads the wiki list from
it. When you name wikis explicitly (wikit backup <name> ...) and there is no
config.json, wikit runs with these same defaults built in — archiving into a
wikit_data folder created in the current working directory (where you launched
the command). Use --base-dir (and the other override flags) to adjust them
without writing a config, or pass -c <path> to point at one. Naming a config
with -c that does not exist is an error.
<base_directory>/
_users/<bucket>.json users bucketed by id >> 13
_users/pending.json
<wiki>/
http_cookies.json
meta/site.json
meta/sitemap.json
meta/pages/<name>.json page metadata (":" -> "_")
meta/file_map.json meta/page_id_map.json meta/pending_*.json
meta/forum/category/<id>.json
meta/forum/<cat>/<thread>.json
pages/<name>.7z compressed page revisions (<rev>.txt)
files/<page>/<file_id> raw attachments
forum/<cat>/<thread>.7z compressed post html (<post>/<rev|latest>.html)
go build -o wikit ./cmd/wikit
Cross-compile (the matching 7-Zip binary is embedded per platform):
GOOS=windows GOARCH=amd64 go build -o wikit.exe ./cmd/wikit
GOOS=linux GOARCH=amd64 go build -o wikit ./cmd/wikit
go test ./...