Problem
The framed upload path (and the streaming-copy pump) accumulates a complete internal part of ciphertext in memory before uploading it to the backend. Internal part size is forced to grow with client part size because internal part numbers are statically strided: client part N owns internal numbers [(N-1)*20+1 .. N*20] (MAX_INTERNAL_PARTS_PER_CLIENT = 20, s3proxy/crypto.py — the stride × 500 client parts fills S3's 10,000-part ceiling, so the cap is load-bearing and cannot simply be raised).
Consequences (all measured/documented in crypto.py governor math):
- 512 MB barman part → 26 MB internal parts → ~56 MB peak per request
- 4 GB PutObject → 204 MB internal parts → ~417 MB peak
- multi-GB streaming copies → internal chunks up to ~238 MB → ~500 MB+ honest peak;
copy_governor_clamped_reserve() deliberately lets these overshoot their reservation ("the pod memory limit is the backstop")
This is the single reason proxy pods need 1 Gi limits: fleet peaks of 640–1020 MiB vs a 192 MB governed budget are dominated by these buffered parts (+ the transport copy, see #127).
Fix
Upload internal parts with an async-generator body that yields sealed 8 MiB frames as encryption produces them, instead of concatenating the whole part first. AES-GCM framing is already sequential; nothing requires the full part in memory. Content-Length is known up front (ciphertext_size()), so no chunked-encoding surprises.
Peak per in-flight request becomes ~2 frames (≈16–24 MB) independent of object/part size. No changes to part numbering, sidecar metadata, or the 10k-part budget.
Retry trade-off
A generator body can't be replayed by botocore's internal retry:
- Copies (do first — this is where the ~500 MB peaks live): source is re-readable via ranged GET; the pump can rebuild the stream and retry the internal part itself. No client-visible change.
- Uploads (second): a mid-part backend failure surfaces as a failed client part instead of being retried internally from the buffer. Backup clients (rclone/scylla-agent, barman) already retry failed parts as a matter of course. Optionally keep buffering for parts ≤ some threshold where it's cheap.
Validation
memory_copy / memory_usage integration suites + OOM Proof CI job: assert new O(frame) ceilings.- Update governor formulas (
streaming_upload_peak, copy_chunk_peak) and their calibrated docstring measurements; the clamp machinery (*_governor_clamped_reserve) should become nearly vestigial.
- Failure-injection test: kill backend connection mid-internal-part, assert copy self-retries and upload surfaces a clean retryable error to the client.
Outcome
Worst-case honest peak ~25 MB/request. Combined with #127 this makes 512 Mi (or lower) pod limits safe by arithmetic; today they'd OOM (observed peaks 640–1020 MiB).
Backup impact & rollout
Expected impact on backups: none in the happy path; backups still complete in all paths. The only observable difference is during backend hiccups: a mid-part failure that botocore retries invisibly today becomes a client-visible failed part that rclone/barman re-send (~one client part of traffic) — the same retry behavior they already exhibit routinely. Copies get strictly better (self-healing internal retry from the re-readable source). No changes to data format, part numbering, or sidecar metadata: objects uploaded before the change remain fully compatible, and rollback is a plain image revert.
Rollout order: copies first (where the ~500 MB peaks live, no retry-semantics change), uploads second. Deploy between backup runs, watch one full daily backup cycle (Failed counters + PART_PLAINTEXT_MISMATCH + memory peaks) before the next step. Sequence after #127.
Problem
The framed upload path (and the streaming-copy pump) accumulates a complete internal part of ciphertext in memory before uploading it to the backend. Internal part size is forced to grow with client part size because internal part numbers are statically strided: client part N owns internal numbers
[(N-1)*20+1 .. N*20](MAX_INTERNAL_PARTS_PER_CLIENT = 20,s3proxy/crypto.py— the stride × 500 client parts fills S3's 10,000-part ceiling, so the cap is load-bearing and cannot simply be raised).Consequences (all measured/documented in
crypto.pygovernor math):copy_governor_clamped_reserve()deliberately lets these overshoot their reservation ("the pod memory limit is the backstop")This is the single reason proxy pods need 1 Gi limits: fleet peaks of 640–1020 MiB vs a 192 MB governed budget are dominated by these buffered parts (+ the transport copy, see #127).
Fix
Upload internal parts with an async-generator body that yields sealed 8 MiB frames as encryption produces them, instead of concatenating the whole part first. AES-GCM framing is already sequential; nothing requires the full part in memory. Content-Length is known up front (
ciphertext_size()), so no chunked-encoding surprises.Peak per in-flight request becomes ~2 frames (≈16–24 MB) independent of object/part size. No changes to part numbering, sidecar metadata, or the 10k-part budget.
Retry trade-off
A generator body can't be replayed by botocore's internal retry:
Validation
memory_copy/memory_usageintegration suites + OOM Proof CI job: assert new O(frame) ceilings.streaming_upload_peak,copy_chunk_peak) and their calibrated docstring measurements; the clamp machinery (*_governor_clamped_reserve) should become nearly vestigial.Outcome
Worst-case honest peak ~25 MB/request. Combined with #127 this makes 512 Mi (or lower) pod limits safe by arithmetic; today they'd OOM (observed peaks 640–1020 MiB).
Backup impact & rollout
Expected impact on backups: none in the happy path; backups still complete in all paths. The only observable difference is during backend hiccups: a mid-part failure that botocore retries invisibly today becomes a client-visible failed part that rclone/barman re-send (~one client part of traffic) — the same retry behavior they already exhibit routinely. Copies get strictly better (self-healing internal retry from the re-readable source). No changes to data format, part numbering, or sidecar metadata: objects uploaded before the change remain fully compatible, and rollback is a plain image revert.
Rollout order: copies first (where the ~500 MB peaks live, no retry-semantics change), uploads second. Deploy between backup runs, watch one full daily backup cycle (Failed counters + PART_PLAINTEXT_MISMATCH + memory peaks) before the next step. Sequence after #127.