Add retry_request / aretry_request helpers for body-phase errors

CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+### Added
+- `retry_request` and `aretry_request` helpers, which run the retry loop at the client level so that errors raised while reading the response body (such as `ReadTimeout` and `RemoteProtocolError`) are retried. This is something `RetryTransport` cannot do, as transports return before the body is read. They reject a client that already uses `RetryTransport` to avoid retrying every request twice.
+- `validate_response` option on `Retry` to retry when a callback rejects an otherwise-successful response (for example, content-level blocks such as a CAPTCHA or authorization wall).
+- `request.extensions["retry"]` support, allowing the `Retry` configuration to be overridden per request; the resolved `Retry` is also attached to `response.extensions["retry"]` for introspection.
+- `Retry.copy_with` to derive a modified `Retry` (mirroring `httpx.URL.copy_with`), convenient for per-request overrides via `request.extensions["retry"]`.
+
 ## [0.5.0] - 2026-04-20
 
 ### Added
@@ -113,7 +121,7 @@ from `httpx.BaseTransport` and `httpx.AsyncBaseTransport`.
 ### Added
 - Initial release
 
-[Unreleased]: https://github.com/will-ockmore/httpx-retries/compare/0.3.0...HEAD
+[Unreleased]: https://github.com/will-ockmore/httpx-retries/compare/0.5.0...HEAD
 [0.3.0]: https://github.com/will-ockmore/httpx-retries/releases/tag/0.3.0
 [0.2.4]: https://github.com/will-ockmore/httpx-retries/releases/tag/0.2.4
 [0.2.3]: https://github.com/will-ockmore/httpx-retries/releases/tag/0.2.3

README.md

@@ -68,6 +68,11 @@ with httpx.Client(transport=transport) as client:
     response = client.get("https://example.com")
 ```
 
+> **Errors while reading the response body** (such as `ReadTimeout` part-way through a download) happen
+> after the transport has returned, so `RetryTransport` can't retry them. This is a niche case,
+> but if you read very large or slow bodies and need those retried, see the `retry_request` / `aretry_request` helpers in
+> [Why wasn't my `ReadTimeout` retried?](https://will-ockmore.github.io/httpx-retries/faq/#why-wasnt-my-readtimeout-retried).
+
 ## Features
 
 HTTPX Retries builds on the patterns users will expect from `urllib` and `requests`. The typical approach has been

docs/api.md

@@ -2,7 +2,8 @@
     options:
         members:
             - RetryTransport
-            - AsyncRetryTransport
             - Retry
+            - retry_request
+            - aretry_request
         filters:
           - "!^_"

docs/faq.md

@@ -59,22 +59,35 @@ Not retried by [RetryTransport][httpx_retries.RetryTransport]:
 
 - Any exception raised during `response.read()`, `response.aread()`, or iteration of a streaming response — including `ReadTimeout` mid-body and `RemoteProtocolError("peer closed connection...")`.
 
-If you need to retry body-phase errors today, do it at the call site:
+!!! tip "Most requests don't need this! Keep using `RetryTransport`"
+    A body-phase error can only occur while the body is in transit, so for the small responses typical of most API traffic that window is tiny and almost every failure is already caught by [RetryTransport][httpx_retries.RetryTransport] at the header or status-code stage. The helpers below earn their keep in a narrower set of cases: reading **large or slow, non-streamed** bodies — big exports and downloads, or payloads proxied from a flaky origin — where a mid-body `ReadTimeout` or a truncated `RemoteProtocolError` is more likely. For ordinary requests, prefer [RetryTransport][httpx_retries.RetryTransport] for simplicity.
+
+To retry body-phase errors, use [retry_request][httpx_retries.retry_request] (or its async counterpart [aretry_request][httpx_retries.aretry_request]). These helpers drive the retry loop at the *client* level, where the body is read, so the same [Retry][httpx_retries.Retry] configuration covers body-phase errors as well as the header-phase errors and retryable status codes that [RetryTransport][httpx_retries.RetryTransport] already handles:
 
 ```python
 import httpx
+from httpx_retries import Retry, retry_request
+
+with httpx.Client() as client:
+    response = retry_request(client, "GET", "https://example.com", retry=Retry(total=5, backoff_factor=0.5))
+```
 
-retryable = (httpx.ReadTimeout, httpx.RemoteProtocolError)
+```python
+import httpx
+from httpx_retries import aretry_request
 
-for attempt in range(5):
-    try:
-        response = client.get("https://example.com")
-        break
-    except retryable:
-        if attempt == 4:
-            raise
+async with httpx.AsyncClient() as client:
+    response = await aretry_request(client, "GET", "https://example.com")
 ```
 
+A plain client is all you need — the helpers run the full retry loop themselves, so there's no need to also install [RetryTransport][httpx_retries.RetryTransport]. Passing a client that uses [RetryTransport][httpx_retries.RetryTransport] raises a `ValueError`, because it would retry every request twice.
+
+!!! warning "These helpers buffer the full response body"
+    Because the body is read before the helper returns, `retry_request` and `aretry_request` are not suitable for streaming. An error raised while iterating a streaming response (`client.stream(...)`) happens after the body has started arriving and cannot be retried transparently — bytes already handed to your code can't be recalled. For streaming, catch the error and re-issue the request yourself.
+
+!!! note "Only idempotent methods are retried by default"
+    Like [RetryTransport][httpx_retries.RetryTransport], the helpers only retry methods in `Retry.allowed_methods` (by default `HEAD`, `GET`, `PUT`, `DELETE`, `OPTIONS`, `TRACE`); other methods are sent once. You can opt a method in with `Retry(allowed_methods=[...])`, but take care: a body-phase retry re-issues the *entire* request, and because the server has already started responding it has most likely processed the original — so only enable non-idempotent methods such as `POST` when duplicate side effects are acceptable.
+
 ## Retrying on response content
 
 Sometimes a server returns a valid response but the body or custom headers signals a failure - for example, a block page, a CAPTCHA redirect, or an authorization wall. This commonly occurs if access may be blocked at the content level rather than the HTTP status level.

httpx_retries/__init__.py

@@ -1,4 +1,5 @@
+from .helpers import aretry_request, retry_request
 from .retry import Retry
 from .transport import RetryTransport
 
-__all__ = ["Retry", "RetryTransport"]
+__all__ = ["Retry", "RetryTransport", "aretry_request", "retry_request"]

httpx_retries/helpers.py

@@ -0,0 +1,147 @@
+import inspect
+from typing import Any
+
+import httpx
+
+from .retry import Retry
+from .transport import RetryTransport, _retry_operation, _retry_operation_async
+
+# Arguments accepted by `Client.send` rather than `Client.build_request`; forwarded to send if provided.
+_SEND_KWARGS = ("auth", "follow_redirects")
+
+
+def _client_retries(client: httpx.Client | httpx.AsyncClient) -> bool:
+    """Return True if the client already retries via a [RetryTransport][httpx_retries.RetryTransport].
+
+    The helpers run the retry loop themselves, so combining them with a retrying transport would retry every
+    request twice. Detection reaches into httpx's private attributes and degrades to ``False`` if they are absent.
+    """
+    mounts = getattr(client, "_mounts", {})
+    transports = [getattr(client, "_transport", None), *mounts.values()]
+    return any(isinstance(transport, RetryTransport) for transport in transports)
+
+
+def retry_request(
+    client: httpx.Client,
+    method: str,
+    url: httpx.URL | str,
+    *,
+    retry: Retry | None = None,
+    **kwargs: Any,
+) -> httpx.Response:
+    """
+    Send a request with retries, including errors raised while reading the response body.
+
+    Unlike [RetryTransport][httpx_retries.RetryTransport], which can only observe what flows through its
+    `handle_request` method (the response *headers*), this helper drives the retry loop at the client level.
+    Because `httpx.Client.send` reads the body before returning, body-phase errors such as `httpx.ReadTimeout`
+    and `httpx.RemoteProtocolError("peer closed connection...")` are caught here and retried.
+
+    ```python
+    import httpx
+    from httpx_retries import retry_request
+
+    with httpx.Client() as client:
+        response = retry_request(client, "GET", "https://example.com")
+    ```
+
+    The retry configuration can be customised, just like [RetryTransport][httpx_retries.RetryTransport]:
+
+    ```python
+    response = retry_request(client, "GET", "https://example.com", retry=Retry(total=5, backoff_factor=0.5))
+    ```
+
+    This helper buffers the full response body, so it is not suitable for streaming. Errors raised while
+    iterating a streaming response (`client.stream(...)`) cannot be retried.
+
+    Body-phase errors are a niche case; see
+    [Why wasn't my `ReadTimeout` retried?](faq.md#why-wasnt-my-readtimeout-retried) for when these helpers are
+    worth using and when to prefer [RetryTransport][httpx_retries.RetryTransport] instead.
+
+    Args:
+        client: The client used to build and send the request.
+        method: The HTTP method.
+        url: The URL to request.
+        retry: The retry configuration. A per-request `request.extensions["retry"]` takes precedence.
+        **kwargs: Additional arguments. `auth` and `follow_redirects` are forwarded to `client.send`; all others
+            (for example `params`, `headers`, `json`, `content`) are passed to `client.build_request`.
+
+    Returns:
+        The final response.
+    """
+    if _client_retries(client):
+        raise ValueError(
+            "retry_request runs the retry loop itself and must be used with a client that does not also retry. "
+            "The given client uses RetryTransport, which would retry every request twice. Use a plain "
+            "httpx.Client instead; retry_request already retries header-phase errors and retryable status codes."
+        )
+
+    send_kwargs = {key: kwargs.pop(key) for key in _SEND_KWARGS if key in kwargs}
+    request = client.build_request(method, url, **kwargs)
+    retry = request.extensions.setdefault("retry", retry or Retry())
+
+    def send(request: httpx.Request) -> httpx.Response:
+        return client.send(request, **send_kwargs)
+
+    if not retry.is_retryable_method(request.method):
+        return send(request)
+
+    if retry.validate_response is not None and inspect.iscoroutinefunction(retry.validate_response):
+        raise TypeError("validate_response must be a sync function when using a sync client")
+
+    return _retry_operation(request, send, retry)
+
+
+async def aretry_request(
+    client: httpx.AsyncClient,
+    method: str,
+    url: httpx.URL | str,
+    *,
+    retry: Retry | None = None,
+    **kwargs: Any,
+) -> httpx.Response:
+    """
+    Send a request asynchronously with retries, including errors raised while reading the response body.
+
+    This is the async counterpart to [retry_request][httpx_retries.retry_request]. Body-phase errors are a niche
+    case; see [Why wasn't my `ReadTimeout` retried?](faq.md#why-wasnt-my-readtimeout-retried) for when these
+    helpers are worth using and when to prefer [RetryTransport][httpx_retries.RetryTransport] instead.
+
+    ```python
+    import httpx
+    from httpx_retries import aretry_request
+
+    async with httpx.AsyncClient() as client:
+        response = await aretry_request(client, "GET", "https://example.com")
+    ```
+
+    Args:
+        client: The client used to build and send the request.
+        method: The HTTP method.
+        url: The URL to request.
+        retry: The retry configuration. A per-request `request.extensions["retry"]` takes precedence.
+        **kwargs: Additional arguments. `auth` and `follow_redirects` are forwarded to `client.send`; all others
+            (for example `params`, `headers`, `json`, `content`) are passed to `client.build_request`.
+
+    Returns:
+        The final response.
+    """
+    if _client_retries(client):
+        raise ValueError(
+            "aretry_request runs the retry loop itself and must be used with a client that does not also retry. "
+            "The given client uses RetryTransport, which would retry every request twice. Use a plain "
+            "httpx.AsyncClient instead; aretry_request already retries header-phase errors and retryable status "
+            "codes."
+        )
+
+    send_kwargs = {key: kwargs.pop(key) for key in _SEND_KWARGS if key in kwargs}
+    request = client.build_request(method, url, **kwargs)
+    retry = request.extensions.setdefault("retry", retry or Retry())
+
+    async def send(request: httpx.Request) -> httpx.Response:
+        return await client.send(request, **send_kwargs)
+
+    if not retry.is_retryable_method(request.method):
+        return await send(request)
+
+    return await _retry_operation_async(request, send, retry)