Skip to content

fix: unbreak TinyGo builds, and remember Apple's back-off across capabilities - #17

Open
deluan wants to merge 2 commits into
mainfrom
fix/tinygo-reflect-panic
Open

deluan wants to merge 2 commits into
mainfrom
fix/tinygo-reflect-panic

Conversation

@deluan

@deluan deluan commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

Two changes. The first is a hotfix: #16 shipped a plugin that crashes on every call. The second is the follow-up that #16 should have included.

1. errors.As traps under TinyGo

isThrottled used errors.As, which calls reflectlite.AssignableTo. TinyGo only partly implements that and traps on unreachable. Simply making it reachable broke everything, because encoding/json calls the same function from newTypeEncoder while decoding the request struct. nd_get_artist_images died inside json.Unmarshal before reaching Apple:

plugin call failed: failed to initialize runtime: wasm error: unreachable
	main.runtime._panic(i32,i32)
	main.(*internal/reflectlite.RawType).AssignableTo(i32,i32) i32
	main.encoding/json.newTypeEncoder(i32,i32,i32,i32)
	...
	main.encoding/json.Unmarshal(i32,i32,i32,i32,i32,i32)

Five in a row opened Navidrome's breaker, which then skipped apple-music for the rest of the run. The plugin issued zero requests, so no throttling behaviour was exercised at all. A type assertion is enough: httpGet returns the error unwrapped and both call sites test it immediately.

I verified this against the real TinyGo artifacts with a wazero harness that stubs the six extism:host/user imports and calls nd_get_artist_images on a built .ndp:

build result
pre-#16 (0.2.0-60a6100) decodes the request, reaches httpGet
#16 merged (0.2.0-80840df) traps in AssignableTo
this PR (0.2.0-PR17-02e2152) decodes the request, reaches httpGet

go test never caught it because it builds native Go, where errors.As is fine. All 139 specs passed on the broken build. Nothing in this repo exercises the TinyGo artifact, which is how it reached a production server.

2. Remember the back-off across every capability

Navidrome keeps two throttle stores that never talk to each other. Artwork uses the circuit breaker in core/artwork/gate.go; biography, similar artists, top songs and artist URL use the cooldown map in core/agents. Neither tells the other, so a 429 the artwork path just absorbed does nothing to stop the metadata path calling the same API seconds later.

The plugin sits below both, so it records the deadline in the cache host service and refuses every request until it passes, before any network call.

One shared key, because Navidrome's breaker keys on the agent name (core/artwork/agent_images.go:76, gate(a.Name, ...)). Finer granularity inside the plugin cannot keep one code path alive while another is parked, so it would be state for nothing.

It stores the deadline rather than the delay, so the wait it reports shrinks as it elapses instead of asking for the original hour every time. The cache TTL expires the key on its own.

This needs a new cache permission in the manifest, so upgrading users will see a prompt.

Why both hosts set the cooldown

I first gated this to itunes.apple.com, assuming a 403 from music.apple.com meant "not available in this storefront". That was wrong, and measuring it disproved it:

case status
bogus artist id 404
unused id (/artist/-/1) 404
non-numeric id 404
bogus storefront (/zz/) 200, falls back
tiny storefront (/va/) 200

Missing content is a 404, so there is no "unavailable" 403 to confuse with a block. The two hosts are also not separate services: both answer Server: daiquiri/5, the same backend family as iTunes' x-daiquiri-instance. So a 403 from either means blocked, and either may set the cooldown.

Separately measured, for the record: music.apple.com has no observable rate limit. 105/105 HTTP 200 ramping 30 to 120 per minute, all reaching origin (69 MISS, 36 TCP_MISS, zero cache hits, 105 distinct artist IDs), no Retry-After or any rate-limit header, and no latency degradation under load. All throttling observed to date comes from itunes.apple.com/search.

What this does not fix

The rate itself. Navidrome's DevArtworkExternalMaxRPS defaults to 2 rps, so 120/min per agent, against roughly 30/min that iTunes sustains. A bulk artwork reprocess will still trip 429s. Live on 2026-09-03 the token worked exactly as intended, and we still earned a 403 twenty seconds later:

16:15:57  429 -> breaker opened, probeAfter=3s   (Apple's own Retry-After)
16:16:07  breaker closed after 3 clean answers
16:16:17  403 -> breaker opened, probeAfter=1h
16:17+    0 iTunes calls, 168 skipped at the gate

Six bad responses, against 36 and accelerating before any of this existed. But a per-agent rate cap on the Navidrome side is the piece that prevents the block rather than reacting to it, and it is not in this PR.

Testing

go test -race ./..., 145 specs.

New coverage: the cooldown is set on a 429 with the delay Apple named and on a 403 for an hour; a parked plugin refuses requests without touching the network; the reported wait shrinks as the deadline approaches and is zero once it passes.

The meaningful check for the TinyGo half is the harness table above, run against the artifact this PR's CI produces, since native go test cannot see that class of failure.

`isThrottled` used `errors.As`, which calls `reflectlite.AssignableTo`.
TinyGo only partly implements that and traps on `unreachable`. Making it
reachable was enough to break every plugin call: `encoding/json` calls the
same function from `newTypeEncoder` while decoding the request struct, so
`nd_get_artist_images` died before it ever reached Apple.

Verified against the real TinyGo artifacts with a wazero harness that calls
`nd_get_artist_images` on a built .ndp. The pre-#16 build decodes the request
and gets as far as `httpGet`; the #16 build traps in `json.Unmarshal` ->
`cachedTypeFields` -> `AssignableTo`, matching what prod logged.

A type assertion is enough here: `httpGet` returns the error unwrapped and
both call sites test it directly, so there is no wrapping to see through.

The unit tests never caught this because `go test` runs native Go, where
`errors.As` is fine. Nothing in the repo exercises the TinyGo build.
@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Download the plugin for this PR: apple-music.zip

Built from f878931 on 2026-09-03T16:58:35Z

@deluan
deluan marked this pull request as ready for review September 3, 2026 16:09
Navidrome keeps two throttle stores that never talk to each other: artwork
uses the circuit breaker in core/artwork/gate.go, while biography, similar
artists, top songs and artist URL use the cooldown map in core/agents. A 429
the artwork path just absorbed does nothing to stop the metadata path calling
the same API seconds later.

The plugin sits below both, so record the deadline in the cache host service
and refuse every request until it passes, before any network call. One key,
because Navidrome's breaker is per-agent anyway: finer granularity here cannot
keep one code path alive while another is parked.

Store the deadline rather than the delay, so the wait we report shrinks as it
elapses instead of asking for the original hour every time. The cache TTL then
expires the key on its own.

Both hosts set it. Measured today: music.apple.com returns 404 for a missing
artist, bogus id or bogus storefront, so a 403 from it is not "no such content"
-- and it answers from the same daiquiri origin that blocks itunes.apple.com,
behind both Fastly and Akamai.

Needs a new cache permission in the manifest, so upgrading users get a prompt.
@deluan deluan changed the title fix: avoid errors.As, which panics under TinyGo fix: unbreak TinyGo builds, and remember Apple's back-off across capabilities Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant