Skip to content

fix(url): fall back to Latin-1 when a response has no declared charset#255

Merged
markuslf merged 1 commit into
mainfrom
fix/url-tolerate-undeclared-charset
Jul 3, 2026
Merged

fix(url): fall back to Latin-1 when a response has no declared charset#255
markuslf merged 1 commit into
mainfrom
fix/url-tolerate-undeclared-charset

Conversation

@NavidSassan

Copy link
Copy Markdown
Member

Problem

url.fetch() decodes every response body as strict UTF-8 when the server sends no charset (response_charset is None → default 'UTF-8'). A single non-UTF-8 byte then aborts the whole check with a UnicodeDecodeError.

Real-world trigger: Comet System Web Sensors serve the degree sign ° as the raw ISO-8859-1 byte 0xb0 in their values.json and declare no charset, so cometsystem fails with:

'utf-8' codec can't decode byte 0xb0 in position 164: invalid start byte while fetching http://.../values.json

This is a regression from the pre-httpx engine (lib 3.x), which decoded tolerantly.

Fix

When the server declared no charset and the UTF-8 assumption fails, fall back to Latin-1, which maps every byte 0x000xFF one-to-one and can never raise. 0xb0 becomes °, which is what the firmware means.

A charset the server explicitly declared still fails on mismatch, so genuine encoding errors are not masked (the existing test_decode_error_returns_failure guards this).

Why Latin-1 and not surrogateescape

The plugins decode a body, build a message, and print it to stdout (re-encoding to UTF-8). Tested against the byte values seen across the monitoring-plugins decode issues (0xb0, 0x81, 0xc2, 0xfc):

bytes latin-1 → output surrogateescape → output
25.3 \xb0C 25.3 °C OK \udcb0 UnicodeEncodeError
>m\x81ller OK UnicodeEncodeError
M\xfcller Müller OK UnicodeEncodeError

surrogateescape only relocates the crash from fetch to output. Latin-1 maps to real Unicode scalars that survive both the JSON round-trip and the stdout re-encode. This matches how the lib already fixed the Windows console decode crashes (shell.py decodes with the actual OEM code page rather than papering over a wrong-codec decode with an error handler).

Tests

  • New test_undeclared_charset_falls_back_to_latin1 reproduces the cometsystem case (body b'25.3 \xb0C', no charset → '25.3 °C').
  • Existing test_decode_error_returns_failure stays valid: a declared charset with invalid bytes still fails.

Full tests/url suite green (90 tests), ruff check/ruff format clean.

Fixes the cometsystem decode failure reported downstream in Linuxfabrik/monitoring-plugins.

fetch() decoded every response body as strict UTF-8 when the server sent
no charset, so a single non-UTF-8 byte (e.g. 0xb0, ° in ISO-8859-1, as
emitted by some sensor firmware) aborted the whole check with a
UnicodeDecodeError. Fall back to Latin-1 in that case, which maps every
byte 1:1 and never fails. A charset the server explicitly declared still
fails on mismatch, so genuine encoding errors are not masked.
@NavidSassan NavidSassan requested a review from markuslf July 3, 2026 09:41
@markuslf markuslf merged commit 7659855 into main Jul 3, 2026
6 checks passed
@NavidSassan NavidSassan deleted the fix/url-tolerate-undeclared-charset branch July 3, 2026 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants