Skip to content

fix(storage): stop compaction from destroying the log on Windows - #378

Open
greit0n wants to merge 1 commit into
sirrobot01:mainfrom
greit0n:fix/compaction-swap-loses-log
Open

greit0n wants to merge 1 commit into
sirrobot01:mainfrom
greit0n:fix/compaction-swap-loses-log

Conversation

@greit0n

@greit0n greit0n commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #377.

The problem

Store.Compact() deletes the live log, then renames the compacted copy into its place while
that copy is still open
, discarding the results of both calls:

s.log = newLog          // newLog is still OPEN
...
_ = oldLog.Close()
_ = os.Remove(oldPath)              // original destroyed first
_ = os.Rename(newLogPath, oldPath)  // fails: source still open. Error discarded.
s.log.path = oldPath                // recorded as success regardless

Go opens files on Windows without FILE_SHARE_DELETE, so renaming a file the process itself still
holds open fails with a sharing violation. This isn't an occasional antivirus problem — on Windows
the rename fails every time, and because the original was removed one line earlier, every
compaction that runs there destroys the store.

The error is discarded and s.log.path is updated anyway, so the index ends up describing a log
that no longer exists. That surfaces as failed to read during compaction: EOF on every
subsequent 5-minute tick, forever, since nothing revisits the decision. openAppendLog uses
O_CREATE, so a restart silently recreates the log empty and throws away the data still
sitting in the .compact file.

The fix

  • Close both logs before the swap — required on Windows for either path to be renameable.
  • Drop the os.Remove. os.Rename replaces an existing destination on POSIX and Windows alike,
    so removing it first only created a window where a failed rename lost everything.
  • Check the rename. On failure, reopen the untouched original so the store keeps serving the
    pre-compaction state, and return the error instead of reporting success.
  • Retry the rename briefly. Antivirus, indexers and backup agents hold transient handles; the Go
    toolchain retries module-cache renames for exactly this reason (cmd/go: 'Access is denied' when renaming module cache directory [1.13 backport] golang/go#37802).
  • Adopt an orphaned <name>.db.compact at startup when the log is missing or empty, so stores
    already broken by this bug recover their data instead of restarting empty.

Tests

Three new tests in pkg/storage/hybrid/store_compaction_test.go. Two of them fail on main on
Windows — including the plain success path, which is the point:

--- FAIL: TestCompactSucceedsAndLeavesNoTempFile
    compaction temp file must not be left behind
--- FAIL: TestRecoverOrphanedCompaction
    data must be recovered from the orphaned .compact file: key keeper: key not found

All three pass with this change, and the existing suite is unaffected (go test ./... green).

TestCompactKeepsLogWhenSwapFails forces an unrenameable destination by holding it open; it skips
on platforms where that rename legitimately succeeds, so it stays meaningful on Linux/macOS
without being flaky.

Notes for review

  • Behaviour on POSIX is unchanged apart from the swap no longer being destructive and errors now
    being surfaced.
  • The startup recovery only triggers when the main log is missing or header-only, so it can't
    shadow a healthy log. The .compact file is fully written and synced before any swap is
    attempted, and Compact() holds the store lock, so it is never a partial snapshot.
  • Running in production on Windows since 2026-08-03. On first start with the patched binary the
    previously orphaned queue.db and repair_runs.db were adopted and are now normal .db files
    again.

🤖 Generated with Claude Code

Compact() removed the live log and then renamed the compacted copy over
it while that copy was still open, discarding both errors.

Go opens files on Windows without FILE_SHARE_DELETE, so renaming a file
the process still holds open fails with a sharing violation. The rename
therefore fails every time on Windows, not just under antivirus
interference - and because the original was already removed, every
compaction that runs there destroys the store. The error was discarded
and s.log.path was updated as if the swap had succeeded, leaving the
index describing a log that no longer exists, which surfaces as
"failed to read during compaction: EOF" on every subsequent 5-minute
tick until the db directory is deleted by hand.

Close both logs before the swap, drop the unnecessary os.Remove
(os.Rename replaces an existing destination on POSIX and Windows
alike), check the rename and fall back to the untouched original on
failure, and retry the rename briefly since antivirus and indexers hold
transient handles - the Go toolchain retries module-cache renames for
the same reason (golang/go#37802).

Also adopt an orphaned <name>.db.compact at startup when the log is
missing or empty. openAppendLog uses O_CREATE, so without this a store
already broken by this bug silently restarts empty and discards the
surviving data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store compaction destroys the log on Windows: rename of a still-open file always fails, after the original has already been deleted

1 participant