Conversation
The NetNTLMv1 .rtc -> .rt.zst migration derived its output path from the source path and deleted each .rtc after a verified round trip. That is correct when the source tree IS the working copy, but it made the script unusable against the complete second copy of the .rtc set on the NAS: pointing --base-dir at it would have written outputs onto the NAS and deleted the backup. Add --dest-dir, which mirrors the source layout relative to --base-dir under a separate root and never deletes the source. This lets every worker read .rtc from the NAS over its own 2.5 GbE link while the destination host only receives .rt.zst, so it no longer serves source reads for the whole fleet and can reclaim the space its .rtc copy occupies. Because the source is immutable in this mode, a deleted .rtc can no longer signal progress, so pending_parts() also treats a non-empty output as done (a zero-length one means an interrupted write, not a finished part). Verified on dell3 against the real binaries: all 4 parts of 0-1000/5 converted from the NAS source are byte-identical to the .rt.zst the in-place pipeline already produced for those parts, all 4 NAS sources survive, nothing was written into the NAS tree, and scratch is clean. Also commits the script and its first test suite, which until now existed only in dell3:~/projects and dell4:/tmp. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On 2026-07-30 two drivers ended up running against range 3001-4095 after a
relaunch. Worker A converted a part and deleted its .rtc; worker B, whose file
list still contained that part, ran rtc2rt against the now-missing source,
failed, and fail() removed the good .rt.zst worker A had just written --
because it deleted zst_path unconditionally. 32 parts were left with neither a
.rtc nor a .rt.zst and had to be rebuilt from the NAS backup. Nothing was
permanently lost only because that backup exists.
Two fixes:
- convert_one_part() now records whether zst_path existed before it started
and only cleans it up on failure if this invocation created it. Cleanup of
our own partial output is unchanged.
- Add driver_lock(): an exclusive non-blocking flock on <manifest>.lock, held
for the whole run, so a second driver on the same manifest (i.e. the same
range) exits with a clear message instead of racing. Ranges were already the
isolation boundary between hosts, but nothing enforced one driver per range
on a single host.
Adds 7 tests. Verified they catch the real defect: reintroducing the
unconditional delete fails exactly 2 of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records the two failure modes that actually cost data during the NetNTLMv1 campaign, so the next person does not rediscover them: --dest-dir is mandatory when the source tree is a backup (without it the script deletes each .rtc), and one driver per range is enforced by an flock on <manifest>.lock. Also notes that the driver's stdout is block-buffered when redirected, so manifest lines rather than log lines are the progress signal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The NetNTLMv1
.rtc->.rt.zstmigration lost 32 outputs in production on 2026-07-30. Two driver processes ended up on range3001-4095after a relaunch. Worker A converted a part and deleted its.rtc; worker B, still holding that part in its file list, ranrtc2rtagainst the now-missing source, failed — andfail()deleted the good.rt.zstworker A had just written, because it removedzst_pathunconditionally. Those parts ended up with neither a.rtcnor a.rt.zst, recoverable only because a complete second copy of the.rtcset exists on a NAS.Using that backup as a source was itself unsafe: the script derived the output path from the source path and deleted each
.rtcafter conversion, so pointing--base-dirat the backup would have written outputs onto it and consumed it.What
--dest-dirmirrors the source layout under a separate root and never deletes the source, so a read-only copy can be the source. Without it, the previous in-place behaviour is unchanged. Because the source is then immutable,pending_parts()also treats a non-empty output as done (a zero-length one means an interrupted write).fail()no longer deletes an output it did not create — it records whetherzst_pathexisted before this invocation started. Cleanup of its own partial output is unchanged.driver_lock()takes an exclusive non-blockingflockon<manifest>.lockfor the whole run, so a second driver on the same range exits with a clear message instead of racing. Ranges were already the isolation boundary between hosts, but nothing enforced one driver per range on a host.Verification
unittest(no pytest) so they run on the hosts' barepython3. Green on macOS and on all three migration hosts.0-1000/5converted from the NAS source arecmp-identical to the.rt.zstthe in-place pipeline had already produced for those parts, with the NAS sources intact and nothing written into the source tree.rc=1withanother driver already holds .../m.jsonl.lock.zstd -tfirst.make cpu-tests+./crackalack_cpu_tests(all pass) andmake macos.🤖 Generated with Claude Code