docs: record the fetch-width measurement - #123
Merged
Merged
Conversation
Two arms on a quiet box, one per FETCH_WORKERS width, each with a 1 Hz sampler: width 1 delivered 37% of width 3's throughput — against the 80% this section set as the bar for switching — while writing slightly more per byte pulled and peaking slightly higher. The width stays 3. The write column is the finding that matters for draining a backlog: written bytes per byte read barely move with the width (2.39 against 2.56), so the platter cost is a property of the pipeline — payload written, extracted, and unpacked — and not of concurrency. That ratio, not the fetch width, is what a reduction would have to move.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the measurement
docs/MAINTENANCE.mdasked for under "Measuring whether one fetch stream beats three" — the section previously said nobody had run it.Result: the width stays 3
Two arms on a quiet box (ingest held down,
io full avg101.6% before the first), same walk, same gate, onlyFETCH_WORKERSdiffering, each with a 1 Hz sampler over/proc/pressure/ioand fields 3/7 of/sys/block/{sda,sdb}/stat:io full avg10Width 1 delivered 37% of width 3's throughput against the 80% bar this section set for switching, and bought nothing back. The arms were not the same releases and that cuts against the result rather than for it: arm A included a 27.59 GB RAR-packed release that needed 55 volumes unpacked locally, arm B included none, and arm A still had the lower churn ratio and lower peak.
Why it matters beyond the flag
Written bytes per byte read barely move with the width, so the platter cost is a property of the pipeline —
payload.zipwritten, extracted, and (for a packed release) its RAR volumes unpacked before deletion — not of concurrency. ~2.4 logical bytes written per byte pulled, before RAID1 mirrors it. That ratio is the one a reduction would have to move, and the doc now says so instead of pointing atFETCH_WORKERS.No code changed: the width-one branch was an experiment and was discarded.