Skip to content

feat(zst): own the full .rt <-> .rt.zst round trip in-tree (zstd-19) - #135

Merged
bandrel merged 14 commits into
masterfrom
feat/zstd-19-rt
Jul 28, 2026
Merged

feat(zst): own the full .rt <-> .rt.zst round trip in-tree (zstd-19)#135
bandrel merged 14 commits into
masterfrom
feat/zstd-19-rt

Conversation

@bandrel

@bandrel bandrel commented Jul 28, 2026

Copy link
Copy Markdown
Owner

What this adds

Compress and decompress .rt tables at zstd level 19 inside the codebase, instead of telling users to shell out to zstd -19 -T0 --rm.

  • Write side (new): a zst_compress module, exposed as a crackalack_rt2zst binary and as a --zst[=LEVEL] flag on crackalack_sort.
  • Read side: crackalack_lookup ingests .rt.zst directly at all four dispatch points, with no flags — the wrapper suffix is stripped before filename params are parsed, so compressed tables stay self-describing.
  • Decompression is streamed (~128 KB input buffer), so it never holds the compressed file in RAM.

This supersedes the older unmerged feat/zstd-ingest branch, which was read-only and had conflicts with the RAR-ingest work that landed in #128. Those conflicts are resolved here.

Measured

On a real 16 MB NTLM table (RTX 3080 Ti / M3 Max as noted):
Compression ratio at L19 16,000,000 → 8,728,071 = 1.83x
Compress throughput ~3.2 MB/s single-threaded (one-time cost; -T N parallelizes)
Decompress throughput ~470 MB/s
Lookup wall time, .rt vs .rt.zst identical (0.75s vs 0.77s; 0.86s vs 0.86s over 8 tables)
Peak RSS overhead fixed ~8.8 MB per concurrent loader = zstd's L19 window, not proportional to table size

The RSS overhead was confirmed to be the decompression window by varying the level: L1 costs +1.1 MB, L10 +4.7 MB, L19 +8.8 MB, matching zstd's 512 KB / 4 MiB / 8 MiB windows. Loaders are capped at 8 (compute_load_thread_count()), so the worst case is ~71 MB regardless of table size — noise against the multi-GB budget lookup auto-detects.

Mask and Markov tables compress far better (7–10x), since their restricted index spaces are more compressible.

Correctness

  • Byte-identical round trip on a real 16 MB table, verified with cmp.
  • crackalack_sort --zst=19 emits bytes identical to crackalack_rt2zst -l 19.
  • All four table filename forms verified end-to-end — plain, mask (%l%l%l%l%d), Markov (-mk1000000), combined mask+Markov, and custom charset (!1-61626378797a) — each cracking a gen_known_hash target from a .rt.zst-only directory with results identical to the plain-.rt control, and each round-tripping byte-identical.
  • CPU suite (including both zst test groups) and the Metal GPU suite pass. Re-verified after rebasing onto master.

Per the branch notes it was also exercised on Linux/OpenCL and CUDA; Windows is not wired up (see below).

Notes and caveats

🤖 Generated with Claude Code

bandrel and others added 14 commits July 28, 2026 16:40
@bandrel @claude
feat(zst): add zst_decompress module + unit test + build wiring
423ef98 
libzstd-backed in-memory decompression of .rt.zst tables, mirroring
rtc_decompress. -lzstd wired into linux/cuda/macos builds. Verified by
round-trip CPU unit test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@bandrel @claude
feat(lookup): ingest .rt.zst tables
2757131 
Rebase of feat/zstd-ingest onto post-RAR master. The per-call-site
parse_name copies are replaced by strip_wrapper_suffix(), a widening of
the RAR feature's strip_rar_suffix() that handles both wrappers, and
count_tables() keeps .rti2 in its extension list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
docs(zst): document .rt.zst ingest (build dep + zstd -19 creation)
526db98 
GPU e2e verified on dell5: lookup cracks a known hash from a .rt.zst-only
directory identically to the raw .rt, and corrupt .zst files are skipped
with a clear error (no crash).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@bandrel @claude
feat(zst): add streaming zst_compress module
4c42fac 
Compresses a raw .rt (from disk or memory) into a zstd frame at level 19
by default. Pledges the source size so the frame header carries the
content size, which zst_decompress() requires. Writes via temp file plus
rename and never touches the source.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
fix(zst): stop zst_compress() from hanging on premature EOF
6ac9cf2 
The streaming read loop never detected a zero-byte fread() that arrived
before `total` bytes were consumed (e.g. the source file shrinks between
the initial size probe and the read loop). consumed stayed short of
total forever, so `last` never became true and the outer loop spun
without end.

Extract the boundary check into zst_is_premature_eof() (zst_compress.h)
so the read loop fails cleanly with a distinct error code instead, and
so the regression test can exercise the exact hang-triggering condition
directly without racing a real truncation against the read loop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
feat(zst): add crackalack_rt2zst tool
ab15e59 
Compresses .rt to .rt.zst at level 19 by default, with -d for the reverse
so the round trip is verifiable with cmp and the benchmark can time the
same zst_decompress() the lookup loader uses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
fix(zst): add crackalack_rt2zst to make clean's rm list
e0d6543 
make clean enumerated every sibling binary by name but omitted the new
tool, leaving it behind in the project root after a clean build --
breaking the documented "make clean removes all binaries" contract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
feat(sort): add --zst[=LEVEL] to emit .rt.zst
c8a5f7a 
Compresses the sorted table straight from the in-memory buffer, so no
re-read is needed, and removes the raw .rt only after the compressed
output is renamed into place. Already-sorted tables are still compressed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
bench(zst): add zstd level-sweep harness and baseline results
401233d 
Sweeps compression levels through crackalack_rt2zst measuring ratio,
throughput, peak RSS, and round-trip byte identity, then compares
end-to-end lookup time across .rt/.rtc/.rt.zst.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
perf(zst): stream decompression to cap peak RSS
972a069 
The one-shot path held the compressed file and the decompressed table in
memory at once, roughly 1.5x the table size at typical ratios, which the
lookup RAM budget does not account for. Streaming the input caps the peak
at the decompressed size plus one input window.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
fix(zst): detect frames whose declared size understates real content
af39fd1 
The streaming zst_decompress() rewrite treated "destination buffer full"
as "frame complete", but ZSTD_decompressStream stops writing once the
buffer fills regardless of whether the frame's own decoder state still
has more to flush. A corrupted/understated Frame_Content_Size field could
therefore produce a silently truncated table instead of an error, unlike
the one-shot API it replaced. Track ZSTD_decompressStream's own return
value (0 means the frame is genuinely finished) and fail with the existing
error code 8 if the buffer filled before the frame did.

Adds test_zst_decompress_rejects_undersized_content_size(), which patches
a real frame's content-size field down by one chain and asserts
zst_decompress() rejects it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
fix: whole-branch review fixes for zstd-19 .rt compression support
532781e 
- Guard all zstd wiring (Makefile SRCS/BINARIES/link rules, crackalack_lookup.c,
  crackalack_sort.c) behind -DHAVE_ZSTD, excluded from BUILD=windows, mirroring
  the existing HAVE_UNRAR pattern -- Windows has no <zstd.h>/-lzstd and
  zst_compress.c relies on fseeko/ftello, which MinGW lacks.
- Fix bench_zstd.py's vacuous "cracked" substring check (both success and
  failure messages contain "cracked") by parsing the real crack count.
- Clear stale pot files before each format's lookup run in bench_zstd.py,
  fixing a "already cracked" early-exit that had been silently producing
  identical, meaningless format_comparison rows; regenerate the streaming
  benchmark with real data.
- Document in the README that crackalack_verify only reads .rt, so tables
  must be verified before --zst compression (or decompressed first).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
fix(windows): guard test_zst calls in cpu_tests_common.c behind HAVE_…
28a384e 
…ZSTD

Round-2 fix for the Windows build gap: cpu_tests_common.c (linked into
crackalack_unit_tests, which is part of the Windows BINARIES set)
unconditionally called six test_zst_* functions, while the Makefile only
adds test_zst.o/zst_decompress.o/zst_compress.o to UNITTEST_OBJS for
non-Windows builds. This left six undefined references at link time for
crackalack_unit_tests.exe. Wrap the #include and both test-calling/reporting
blocks in #ifdef HAVE_ZSTD, mirroring the existing crackalack_lookup.c and
crackalack_sort.c guards.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bandrel @claude
docs(zst): document .rt.zst in the agent guide; bump to v1.5.7
7abdd56 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bandrel
bandrel merged commit a68097c into master Jul 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bandrel