Conversation
The retryable status set in htsparse.c was 408, 409, 500, 502 and 504. A 429 or a 503 therefore got one request and the link was dropped, and Retry-After was parsed nowhere. Measured against a local server with --retries=3: the 500 and the 408 were each requested four times, the 429 and the 503 exactly once. Adding the two codes on their own would have made it worse. A retry re-records the link and the next free socket takes it, with no backoff, so all four attempts on the 500 landed inside the same second. The wait therefore comes first: back_set_retry_after() withholds every new launch until the delay is up, through the same non-blocking gate that already carries -%c and --pause, which both the look-ahead and the main loop already honour. The server picks the Retry-After value, so --max-retry-after (-%J, 60 seconds by default) caps what the engine obeys, and the knob itself stops at an hour. A 503 naming no delay stays fatal, because it is as likely a host that is down for good and retrying it triples the requests made to it. One int at the htsblk tail and one at the httrackp tail, both ABI-stable, so the soname does not move. Closes #1738 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Review found that the new hold swallowed the user's Ctrl-C: with --max-retry-after=20 a SIGINT took 17.3s to be honoured, against 1ms with the wait waived. back_pluggable_sockets_strict() returned 0 ahead of the socket arithmetic every wait loop reads, and sig_leave() only sets opt->state.stop, which nothing on that path looked at. --pause has the same defect on master today, measured at 16.1s, so the guard covers both gates rather than only the new one. Both now answer in tens of milliseconds. Also from review: the crawl test could not see the HTTP-date form at all, because its date case was a 429, which retries whether or not the header parsed. It is a 503 now, joined by a 503 carrying an unusable value, so dropping the date branch or swapping the parser for atoi() both go red. A gate self-test covers the hold, the cap and the stop without a signal, so Windows runs it too. Signed-off-by: Xavier Roche <roche@httrack.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…thing Under WSL2 the suite runs a native Windows python, whose text-mode stdout ends every line with CRLF. Test 490 counts requests with grep -c '^HIT GET /rate.html$', which a trailing CR makes match nothing, so the leg failed reporting 0 requests while its own dump showed all three. The engine was right; the counting was not. Pin stdout to LF the way local-server.py and proxytestlib.py already do. Also regenerate html/httrack.man.html, which the man-page-sync guard requires to move whenever man/httrack.1 does. Signed-off-by: Xavier Roche <roche@httrack.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The number was free when the branch was cut; master has since landed 490 and 491. 231_test-names sees the collision on the merged tree and reds five legs with it, all from that one cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
added a commit
that referenced
this pull request
Sep 23, 2026
…to label one (#1757) #1755 added `--max-retry-after` to the engine, and no front end offers it. WinHTTrack cannot even label a field for one, because `lang.def` is this repo's. `COptionTab4::GetTip()` shows no tooltip at all when `LANGSEL()` returns the empty string for a key the catalogs lack. This adds the field to WebHTTrack's Flow control page, next to Retries. It also adds the two `LANG_` keys, translated in all 30 catalogs, and the `MaxRetryAfter` row in `winprofile-keys.tsv`. The row names `web` as its only owner. The other two gain their letter when each lands its own field, the way `PauseFiles` gained `droid` in #1331. An empty box passes no flag, so the engine's own 60 seconds apply. A front end that substituted 60 would write a value nobody chose, which is the `MaxRate` lesson in `doc/winprofile-ini.md`. Test 493 probes a fresh session for that before it posts anything. --------- Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HTTrack never retried a 429 or a 503. The retryable status set in
htsparse.cwas 408, 409, 500, 502 and 504, andRetry-Afterwas parsed nowhere. Against a local server with--retries=3the 500 and the 408 were each requested four times, the 429 and the 503 exactly once.Adding the two codes alone would have been worse than dropping the link. A retry re-records the link and the next free socket takes it, so all four attempts on the 500 landed inside the same second. The wait therefore comes first, reusing the launch gate that already carries
-%cand--pause.--max-retry-aftercaps what the engine obeys, because the server picks the value (-%J, 60s by default). A 503 naming no usable delay stays fatal. Both new fields sit at a struct tail, so the soname does not move.Review turned up a second defect, which the last commit fixes: the hold swallowed the user's Ctrl-C.
--pausedoes the same on master today, measured at 16.1 seconds, so the guard covers both gates.Closes #1738