Add support for the compressed-blobs REAPI proposal

README.md

@@ -14,7 +14,7 @@ commodity hardware and AWS servers. Outgoing bandwidth can exceed 15 Gbit/s on t
 ## HTTP/1.1 REST API
 
 Cache entries are set and retrieved by key, and there are two types of keys that can be used:
-1. Content addressed storage (CAS), where the key is the lowercase SHA256 hash of the stored value.
+1. Content addressed storage (CAS), where the key is the lowercase SHA256 hash of the entry.
    The REST API for these entries is: `/cas/<key>` or with an optional but ignored instance name:
    `/<instance>/cas/<key>`.
 2. Action cache, where the key is an arbitrary 64 character lowercase hexadecimal string.
@@ -24,6 +24,14 @@ Cache entries are set and retrieved by key, and there are two types of keys that
 Values are stored via HTTP PUT requests, and retrieved via GET requests.
 HEAD requests can be used to confirm whether a key exists or not.
 
+If GET requests specify `zstd` in the `Accept-Encoding` header, then
+zstandard-encoded data may be returned.
+
+To upload zstandard compressed data, PUT requests must set
+`Content-Encoding: zstd` and include a custom `X-Digest-SizeBytes` header
+with the size of the uncompressed entry. The key must also refer to
+the uncompressed entry.
+
 If the `--enable_ac_key_instance_mangling` flag is specified and the instance
 name is not empty, then action cache keys are hashed along with the instance
 name to produce the action cache lookup key. Since the URL path is processed
@@ -93,6 +101,19 @@ which can be enabled with the `--experimental_remote_asset_api` flag.
 To use this with Bazel, specify
 [--experimental_remote_downloader=grpc://replace-with-your.host:port](https://docs.bazel.build/versions/master/command-line-reference.html#flag--experimental_remote_downloader).
 
+### Byte Stream compressed-blobs
+
+This version of bazel-remote supports the
+[Byte Stream compressed-blobs proposal](https://github.com/bazelbuild/remote-apis/pull/168),
+which provides a way for clients to upload and download CAS blobs compressed
+with zstandard, in order to improve network efficiency.
+
+Uploaded CAS blobs are stored in a zstandard compressed format by default,
+which can increase the effective cache size and reduce load on the server
+if clients also download blobs in zstandard compressed form. If you would
+rather store CAS blobs in uncompressed form, add `--storage_mode uncompressed`
+to your configuration.
+
 ## Usage
 
 If a YAML configuration file is specified by the `--config_file` command line
@@ -115,39 +136,44 @@ DESCRIPTION:
    A remote build cache for Bazel.
 
 GLOBAL OPTIONS:
-   --config_file value                Path to a YAML configuration file. If this flag is specified then all other flags are ignored. [$BAZEL_REMOTE_CONFIG_FILE]
-   --dir value                        Directory path where to store the cache contents. This flag is required. [$BAZEL_REMOTE_DIR]
-   --max_size value                   The maximum size of the remote cache in GiB. This flag is required. (default: -1) [$BAZEL_REMOTE_MAX_SIZE]
-   --host value                       Address to listen on. Listens on all network interfaces by default. [$BAZEL_REMOTE_HOST]
-   --port value                       The port the HTTP server listens on. (default: 8080) [$BAZEL_REMOTE_PORT]
-   --grpc_port value                  The port the gRPC server listens on. Set to 0 to disable. (default: 9092) [$BAZEL_REMOTE_GRPC_PORT]
-   --profile_host value               A host address to listen on for profiling, if enabled by a valid --profile_port setting. (default: "127.0.0.1") [$BAZEL_REMOTE_PROFILE_HOST]
-   --profile_port value               If a positive integer, serve /debug/pprof/* URLs from http://profile_host:profile_port. (default: 0, ie profiling disabled) [$BAZEL_REMOTE_PROFILE_PORT]
-   --http_read_timeout value          The HTTP read timeout for a client request in seconds (does not apply to the proxy backends or the profiling endpoint) (default: 0s, ie disabled) [$BAZEL_REMOTE_HTTP_READ_TIMEOUT]
-   --http_write_timeout value         The HTTP write timeout for a server response in seconds (does not apply to the proxy backends or the profiling endpoint) (default: 0s, ie disabled) [$BAZEL_REMOTE_HTTP_WRITE_TIMEOUT]
-   --htpasswd_file value              Path to a .htpasswd file. This flag is optional. Please read https://httpd.apache.org/docs/2.4/programs/htpasswd.html. [$BAZEL_REMOTE_HTPASSWD_FILE]
-   --tls_enabled                      This flag has been deprecated. Specify tls_cert_file and tls_key_file instead. (default: false) [$BAZEL_REMOTE_TLS_ENABLED]
-   --tls_ca_file value                Optional. Enables mTLS (authenticating client certificates), should be the certificate authority that signed the client certificates. [$BAZEL_REMOTE_TLS_CA_FILE]
-   --tls_cert_file value              Path to a pem encoded certificate file. [$BAZEL_REMOTE_TLS_CERT_FILE]
-   --tls_key_file value               Path to a pem encoded key file. [$BAZEL_REMOTE_TLS_KEY_FILE]
-   --idle_timeout value               The maximum period of having received no request after which the server will shut itself down. (default: 0s, ie disabled) [$BAZEL_REMOTE_IDLE_TIMEOUT]
-   --max_queued_uploads value         When using proxy backends, sets the maximum number of objects in queue for upload. If the queue is full, uploads will be skipped until the queue has space again. (default: 1000000) [$BAZEL_REMOTE_MAX_QUEUED_UPLOADS]
-   --num_uploaders value              When using proxy backends, sets the number of Goroutines to process parallel uploads to backend. (default: 100) [$BAZEL_REMOTE_NUM_UPLOADERS]
-   --s3.endpoint value                The S3/minio endpoint to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_ENDPOINT]
-   --s3.bucket value                  The S3/minio bucket to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_BUCKET]
-   --s3.prefix value                  The S3/minio object prefix to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_PREFIX]
-   --s3.access_key_id value           The S3/minio access key to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_ACCESS_KEY_ID]
-   --s3.secret_access_key value       The S3/minio secret access key to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_SECRET_ACCESS_KEY]
-   --s3.disable_ssl                   Whether to disable TLS/SSL when using the S3 proxy backend. (default: false, ie enable TLS/SSL) [$BAZEL_REMOTE_S3_DISABLE_SSL]
-   --s3.iam_role_endpoint value       Endpoint for using IAM security credentials. By default it will look for credentials in the standard locations for the AWS platform. [$BAZEL_REMOTE_S3_IAM_ROLE_ENDPOINT]
-   --s3.region value                  The AWS region. Required when not specifying S3/minio access keys. [$BAZEL_REMOTE_S3_REGION]
-   --s3.key_version value             Set to 1 for the legacy flat key format, or 2 for the newer format that reduces the impact of S3 rate limits. (default: 1) [$BAZEL_REMOTE_S3_KEY_VERSION]
-   --disable_http_ac_validation       Whether to disable ActionResult validation for HTTP requests. (default: false, ie enable validation) [$BAZEL_REMOTE_DISABLE_HTTP_AC_VALIDATION]
-   --disable_grpc_ac_deps_check       Whether to disable ActionResult dependency checks for gRPC GetActionResult requests. (default: false, ie enable ActionCache dependency checks) [$BAZEL_REMOTE_DISABLE_GRPS_AC_DEPS_CHECK]
-   --enable_ac_key_instance_mangling  Whether to enable mangling ActionCache keys with non-empty instance names. (default: false, ie disable mangling) [$BAZEL_REMOTE_ENABLE_AC_KEY_INSTANCE_MANGLING]
-   --enable_endpoint_metrics          Whether to enable metrics for each HTTP/gRPC endpoint. (default: false, ie disable metrics) [$BAZEL_REMOTE_ENABLE_ENDPOINT_METRICS]
-   --experimental_remote_asset_api    Whether to enable the experimental remote asset API implementation. (default: false, ie disable remote asset API) [$BAZEL_REMOTE_EXPERIMENTAL_REMOTE_ASSET_API]
-   --help, -h                         show help (default: false)
+   --config_file value                      Path to a YAML configuration file. If this flag is specified then all other flags are ignored. [$BAZEL_REMOTE_CONFIG_FILE]
+   --dir value                              Directory path where to store the cache contents. This flag is required. [$BAZEL_REMOTE_DIR]
+   --max_size value                         The maximum size of the remote cache in GiB. This flag is required. (default: -1) [$BAZEL_REMOTE_MAX_SIZE]
+   --storage_mode value                     Which format to store CAS blobs in. Must be one of "zstd" or "uncompressed". (default: "zstd") [$BAZEL_REMOTE_STORAGE_MODE]
+   --host value                             Address to listen on. Listens on all network interfaces by default. [$BAZEL_REMOTE_HOST]
+   --port value                             The port the HTTP server listens on. (default: 8080) [$BAZEL_REMOTE_PORT]
+   --grpc_port value                        The port the gRPC server listens on. Set to 0 to disable. (default: 9092) [$BAZEL_REMOTE_GRPC_PORT]
+   --profile_host value                     A host address to listen on for profiling, if enabled by a valid --profile_port setting. (default: "127.0.0.1") [$BAZEL_REMOTE_PROFILE_HOST]
+   --profile_port value                     If a positive integer, serve /debug/pprof/* URLs from http://profile_host:profile_port. (default: 0, ie profiling disabled) [$BAZEL_REMOTE_PROFILE_PORT]
+   --http_read_timeout value                The HTTP read timeout for a client request in seconds (does not apply to the proxy backends or the profiling endpoint) (default: 0s, ie disabled) [$BAZEL_REMOTE_HTTP_READ_TIMEOUT]
+   --http_write_timeout value               The HTTP write timeout for a server response in seconds (does not apply to the proxy backends or the profiling endpoint) (default: 0s, ie disabled) [$BAZEL_REMOTE_HTTP_WRITE_TIMEOUT]
+   --htpasswd_file value                    Path to a .htpasswd file. This flag is optional. Please read https://httpd.apache.org/docs/2.4/programs/htpasswd.html. [$BAZEL_REMOTE_HTPASSWD_FILE]
+   --tls_enabled                            This flag has been deprecated. Specify tls_cert_file and tls_key_file instead. (default: false) [$BAZEL_REMOTE_TLS_ENABLED]
+   --tls_ca_file value                      Optional. Enables mTLS (authenticating client certificates), should be the certificate authority that signed the client certificates. [$BAZEL_REMOTE_TLS_CA_FILE]
+   --tls_cert_file value                    Path to a pem encoded certificate file. [$BAZEL_REMOTE_TLS_CERT_FILE]
+   --tls_key_file value                     Path to a pem encoded key file. [$BAZEL_REMOTE_TLS_KEY_FILE]
+   --idle_timeout value                     The maximum period of having received no request after which the server will shut itself down. (default: 0s, ie disabled) [$BAZEL_REMOTE_IDLE_TIMEOUT]
+   --max_queued_uploads value               When using proxy backends, sets the maximum number of objects in queue for upload. If the queue is full, uploads will be skipped until the queue has space again. (default: 1000000) [$BAZEL_REMOTE_MAX_QUEUED_UPLOADS]
+   --num_uploaders value                    When using proxy backends, sets the number of Goroutines to process parallel uploads to backend. (default: 100) [$BAZEL_REMOTE_NUM_UPLOADERS]
+   --http_proxy.url value                   The base URL to use for a http proxy backend. [$BAZEL_REMOTE_HTTP_PROXY_URL]
+   --gcs_proxy.bucket value                 The bucket to use for the Google Cloud Storage proxy backend. [$BAZEL_REMOTE_GCS_BUCKET]
+   --gcs_proxy.use_default_credentials      Whether or not to use authentication for the Google Cloud Storage proxy backend. (default: false) [$BAZEL_REMOTE_GCS_USE_DEFAULT_CREDENTIALS]
+   --gcs_proxy.json_credentials_file value  Path to a JSON file that contains Google credentials for the Google Cloud Storage proxy backend. [$BAZEL_REMOTE_GCS_JSON_CREDENTIALS_FILE]
+   --s3.endpoint value                      The S3/minio endpoint to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_ENDPOINT]
+   --s3.bucket value                        The S3/minio bucket to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_BUCKET]
+   --s3.prefix value                        The S3/minio object prefix to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_PREFIX]
+   --s3.access_key_id value                 The S3/minio access key to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_ACCESS_KEY_ID]
+   --s3.secret_access_key value             The S3/minio secret access key to use when using S3 proxy backend. [$BAZEL_REMOTE_S3_SECRET_ACCESS_KEY]
+   --s3.disable_ssl                         Whether to disable TLS/SSL when using the S3 proxy backend. (default: false, ie enable TLS/SSL) [$BAZEL_REMOTE_S3_DISABLE_SSL]
+   --s3.iam_role_endpoint value             Endpoint for using IAM security credentials. By default it will look for credentials in the standard locations for the AWS platform. [$BAZEL_REMOTE_S3_IAM_ROLE_ENDPOINT]
+   --s3.region value                        The AWS region. Required when not specifying S3/minio access keys. [$BAZEL_REMOTE_S3_REGION]
+   --s3.key_version value                   Set to 1 for the legacy flat key format, or 2 for the newer format that reduces the impact of S3 rate limits. (default: 1) [$BAZEL_REMOTE_S3_KEY_VERSION]
+   --disable_http_ac_validation             Whether to disable ActionResult validation for HTTP requests. (default: false, ie enable validation) [$BAZEL_REMOTE_DISABLE_HTTP_AC_VALIDATION]
+   --disable_grpc_ac_deps_check             Whether to disable ActionResult dependency checks for gRPC GetActionResult requests. (default: false, ie enable ActionCache dependency checks) [$BAZEL_REMOTE_DISABLE_GRPS_AC_DEPS_CHECK]
+   --enable_ac_key_instance_mangling        Whether to enable mangling ActionCache keys with non-empty instance names. (default: false, ie disable mangling) [$BAZEL_REMOTE_ENABLE_AC_KEY_INSTANCE_MANGLING]
+   --enable_endpoint_metrics                Whether to enable metrics for each HTTP/gRPC endpoint. (default: false, ie disable metrics) [$BAZEL_REMOTE_ENABLE_ENDPOINT_METRICS]
+   --experimental_remote_asset_api          Whether to enable the experimental remote asset API implementation. (default: false, ie disable remote asset API) [$BAZEL_REMOTE_EXPERIMENTAL_REMOTE_ASSET_API]
+   --help, -h                               show help (default: false)
 ```
 
 ### Example configuration file
@@ -157,6 +183,9 @@ GLOBAL OPTIONS:
 dir: path/to/cache-dir
 max_size: 100
 
+# The form to store CAS blobs in ("zstd" or "uncompressed"):
+#storage_mode: zstd
+
 host: localhost
 # The port to use for HTTP/HTTPS:
 #port: 8080

WORKSPACE

@@ -969,15 +969,15 @@ go_repository(
 go_repository(
     name = "com_github_klauspost_compress",
     importpath = "github.com/klauspost/compress",
-    sum = "h1:dB4Bn0tN3wdCzQxnS8r06kV74qN/TAfaIS0bVE8h3jc=",
-    version = "v1.11.3",
+    sum = "h1:kz40R/YWls3iqT9zX9AHN3WoVsrAWVyui5sxuLqiXqU=",
+    version = "v1.11.4",
 )
 
 go_repository(
     name = "com_github_mostynb_go_grpc_compression",
     importpath = "github.com/mostynb/go-grpc-compression",
-    sum = "h1:emrImHjSPW/H4aarNxuiPG1uN1WgIlVdR3M0LH5HH5E=",
-    version = "v1.1.2",
+    sum = "h1:wlscKqxoQsZQxUi6uTER+Gh9MbcUFUa4Cw5A69pj/c8=",
+    version = "v1.1.4",
 )
 
 go_repository(
@@ -1070,3 +1070,10 @@ go_repository(
     sum = "h1:y7Vn4YH/rfUHOCwNhvkAcA0gMQvFdKzSE8Ri3qtcFlc=",
     version = "v1.10.1",
 )
+
+go_repository(
+    name = "com_github_mostynb_zstdpool_syncpool",
+    importpath = "github.com/mostynb/zstdpool-syncpool",
+    sum = "h1:tSmtZSiSAfffEgy9ziKvOZOOlccUwF2Ar78+CeOH+8A=",
+    version = "v0.0.2",
+)

cache/cache.go

@@ -4,7 +4,6 @@ import (
 	"crypto/sha256"
 	"encoding/hex"
 	"io"
-	"path/filepath"
 )
 
 // EntryKind describes the kind of cache entry
@@ -33,6 +32,16 @@ func (e EntryKind) String() string {
 	return "raw"
 }
 
+func (e EntryKind) DirName() string {
+	if e == AC {
+		return "ac.v2"
+	}
+	if e == CAS {
+		return "cas.v2"
+	}
+	return "raw.v2"
+}
+
 // Logger is designed to be satisfied by log.Logger.
 type Logger interface {
 	Printf(format string, v ...interface{})
@@ -53,14 +62,20 @@ func (e *Error) Error() string {
 // Proxy is the interface that (optional) proxy backends must implement.
 // Implementations are expected to be safe for concurrent use.
 type Proxy interface {
-	// Put should make a reasonable effort to proxy this data to the backend.
-	// This is allowed to fail silently (eg when under heavy load).
-	Put(kind EntryKind, hash string, size int64, rdr io.ReadCloser)
 
-	// Get should return the cache item identified by `hash`, or an error
-	// if something went wrong. If the item was not found, the io.ReadCloser
-	// will be nil.
-	Get(kind EntryKind, hash string) (io.ReadCloser, int64, error)
+	// Put makes a reasonable effort to upload the cache item identified by
+	// `hash` with logical size `size`, whose data is readable from `rc` to
+	// the proxy backend. The data available in `rc` is in the same format
+	// as used by the disk.Cache instance.
+	//
+	// This is allowed to fail silently (for example when under heavy load).
+	Put(kind EntryKind, hash string, size int64, rc io.ReadCloser)
+
+	// Get returns an io.ReadCloser from which the cache item identified by
+	// `hash` can be read, its logical size, and an error if something went
+	// wrong. The data available from `rc` is in the same format as used by
+	// the disk.Cache instance.
+	Get(kind EntryKind, hash string) (rc io.ReadCloser, size int64, err error)
 
 	// Contains returns whether or not the cache item exists on the
 	// remote end, and the size if it exists (and -1 if the size is
@@ -87,7 +102,6 @@ func TransformActionCacheKey(key, instance string, logger Logger) string {
 	return newKey
 }
 
-// Key returns the proper cache key for an entry kind and hash.
-func Key(kind EntryKind, hash string) string {
-	return filepath.Join(kind.String(), hash[:2], hash)
+func LookupKey(kind EntryKind, hash string) string {
+	return kind.String() + "/" + hash
 }

cache/disk/BUILD.bazel

@@ -10,6 +10,7 @@ go_library(
     visibility = ["//visibility:public"],
     deps = [
         "//cache:go_default_library",
+        "//cache/disk/casblob:go_default_library",
         "//genproto/build/bazel/remote/execution/v2:go_default_library",
         "//utils/tempfile:go_default_library",
         "@com_github_djherbis_atime//:go_default_library",
@@ -28,6 +29,7 @@ go_test(
     embed = [":go_default_library"],
     deps = [
         "//cache:go_default_library",
+        "//cache/disk/casblob:go_default_library",
         "//cache/httpproxy:go_default_library",
         "//genproto/build/bazel/remote/execution/v2:go_default_library",
         "//utils:go_default_library",

cache/disk/casblob/BUILD.bazel

@@ -0,0 +1,19 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["casblob.go"],
+    importpath = "github.com/buchgr/bazel-remote/cache/disk/casblob",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//utils/zstdpool:go_default_library",
+        "@com_github_klauspost_compress//zstd:go_default_library",
+        "@com_github_mostynb_zstdpool_syncpool//:go_default_library",
+    ],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["casblob_test.go"],
+    embed = [":go_default_library"],
+)