fix(url): fall back to Latin-1 when a response has no declared charset#255
Merged
Conversation
fetch() decoded every response body as strict UTF-8 when the server sent no charset, so a single non-UTF-8 byte (e.g. 0xb0, ° in ISO-8859-1, as emitted by some sensor firmware) aborted the whole check with a UnicodeDecodeError. Fall back to Latin-1 in that case, which maps every byte 1:1 and never fails. A charset the server explicitly declared still fails on mismatch, so genuine encoding errors are not masked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
url.fetch()decodes every response body as strict UTF-8 when the server sends no charset (response_charsetisNone→ default'UTF-8'). A single non-UTF-8 byte then aborts the whole check with aUnicodeDecodeError.Real-world trigger: Comet System Web Sensors serve the degree sign
°as the raw ISO-8859-1 byte0xb0in theirvalues.jsonand declare no charset, socometsystemfails with:This is a regression from the pre-
httpxengine (lib 3.x), which decoded tolerantly.Fix
When the server declared no charset and the UTF-8 assumption fails, fall back to Latin-1, which maps every byte
0x00–0xFFone-to-one and can never raise.0xb0becomes°, which is what the firmware means.A charset the server explicitly declared still fails on mismatch, so genuine encoding errors are not masked (the existing
test_decode_error_returns_failureguards this).Why Latin-1 and not
surrogateescapeThe plugins decode a body, build a message, and print it to stdout (re-encoding to UTF-8). Tested against the byte values seen across the monitoring-plugins decode issues (
0xb0,0x81,0xc2,0xfc):25.3 \xb0C25.3 °COK\udcb0UnicodeEncodeError>m\x81llerM\xfcllerMüllerOKsurrogateescapeonly relocates the crash from fetch to output. Latin-1 maps to real Unicode scalars that survive both the JSON round-trip and the stdout re-encode. This matches how the lib already fixed the Windows console decode crashes (shell.pydecodes with the actual OEM code page rather than papering over a wrong-codec decode with an error handler).Tests
test_undeclared_charset_falls_back_to_latin1reproduces the cometsystem case (bodyb'25.3 \xb0C', no charset →'25.3 °C').test_decode_error_returns_failurestays valid: a declared charset with invalid bytes still fails.Full
tests/urlsuite green (90 tests),ruff check/ruff formatclean.Fixes the
cometsystemdecode failure reported downstream in Linuxfabrik/monitoring-plugins.