feature request: Content-Defined Chunking

[martin4861]

Newer versions of Bazel support Content-Defined Chunking (CDC) to incrementally upload and download artifacts. Client-side this can be enabled on Bazel 8.7 and 9.1 using --experimental_remote_cache_chunking. In order to make this work, bazel-remote would need to support CDC as well.

There is also an excellent blog post about CDC, which is implemented in BuildBuddy: https://www.buildbuddy.io/blog/content-defined-chunking/

[vogelsgesang]

+1 - I would also like to use CDC with the bazel-remote cache.

Afaict, the "SpliceBlob" part was already implemented in af08f8a. The missing piece is SpliceBlob, regarding which the commit noted:

The SplitBlob rpc is not yet supported, I am waiting for details of a common chunking algorithm to be decided.

I admittedly don't quite understand this note, though. My understanding of the protocol was that the server shouldn't need to care about the chunking algorithm. The client decides on whichever chunking algorithm it prefers. And then informs the server about the chunks via SpliceBlob. The server should remember that splicing and just return it verbatim when asked via SplitBlob

[mostynb]

The SplitBlobRequest message specifies the blob it wants split, and a preferred chunking function id. The server decides how to split, if at all.

I suppose the main choices are which chunking function (or functions) to implement, and perhaps more complicated: how blobs should be stored.

[vogelsgesang]

indeed, you are right. I missed that bazelbuild/remote-apis@de5501d added a chunking_function and still had the older version in mind where this field didn't exist, yet