Skip to content

Use hard links instead of copying when writing cache data to output #190

Description

@sakuro

Summary

When writing cached data to an output path (cache.FileSystem.WriteTo) or from a temp download file to the final output (transfer.copyFile), we currently always read+write the full file contents. On the same filesystem, and when the cache entry is not compressed, a hard link (os.Link) would avoid duplicating disk space while producing an equivalent file.

Background

  • internal/cache/filesystem.go: WriteTo reads (and decompresses if needed) the cached entry and writes it to outputPath via os.WriteFile.
  • internal/transfer/downloader.go: copyFile reads the temp download file and writes it to output.
  • The default download cache config has no CompressionThreshold set (never compressed) — see internal/config/config.go — so MOD/game file cache entries are good hard-link candidates by default. api and info_json caches compress by default, so those entries cannot be hard-linked as-is.

Proposed approach

  • Attempt os.Link(src, dst) first; on failure (cross-device EXDEV, filesystem without hard link support, etc.) fall back to the existing copy behavior. Prefer this try-then-fallback approach over pre-checking "does the OS/filesystem support hard links" or "are src/dst on the same filesystem" — those checks are platform-specific and can't reliably predict success (e.g. bind mounts, overlayfs).
  • Only attempt the hard link when the cache entry is uncompressed. Compressed entries must be inflated and written as before.
  • Handle the case where dst already exists — os.Link fails if the destination exists (unlike os.WriteFile, which overwrites). Remove the existing destination first, or link to a temp name and os.Rename over the destination for atomicity.
  • Note: transfer.copyFile copies from a temp directory created by os.MkdirTemp("", ...), which is often on a different filesystem than the output path, so the hard link attempt will frequently fall back to copy there — the main benefit is in cache.FileSystem.WriteTo (cache hit path).

Risks / things to verify

  • A hard-linked output file shares the same inode as the cache entry. If anything modifies the downloaded output file in place after writing (rather than replacing it), it would silently corrupt the cache entry too. Confirm no code path does in-place modification of downloaded MOD/game files.
  • Deleting a cache entry (Delete) only removes one directory entry, so it does not affect an already hard-linked output file — this should be fine, but worth confirming as part of implementation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions