Conversation
`isThrottled` used `errors.As`, which calls `reflectlite.AssignableTo`. TinyGo only partly implements that and traps on `unreachable`. Making it reachable was enough to break every plugin call: `encoding/json` calls the same function from `newTypeEncoder` while decoding the request struct, so `nd_get_artist_images` died before it ever reached Apple. Verified against the real TinyGo artifacts with a wazero harness that calls `nd_get_artist_images` on a built .ndp. The pre-#16 build decodes the request and gets as far as `httpGet`; the #16 build traps in `json.Unmarshal` -> `cachedTypeFields` -> `AssignableTo`, matching what prod logged. A type assertion is enough here: `httpGet` returns the error unwrapped and both call sites test it directly, so there is no wrapping to see through. The unit tests never caught this because `go test` runs native Go, where `errors.As` is fine. Nothing in the repo exercises the TinyGo build.
Contributor
|
Download the plugin for this PR: apple-music.zip Built from f878931 on 2026-09-03T16:58:35Z |
deluan
marked this pull request as ready for review
September 3, 2026 16:09
Navidrome keeps two throttle stores that never talk to each other: artwork uses the circuit breaker in core/artwork/gate.go, while biography, similar artists, top songs and artist URL use the cooldown map in core/agents. A 429 the artwork path just absorbed does nothing to stop the metadata path calling the same API seconds later. The plugin sits below both, so record the deadline in the cache host service and refuse every request until it passes, before any network call. One key, because Navidrome's breaker is per-agent anyway: finer granularity here cannot keep one code path alive while another is parked. Store the deadline rather than the delay, so the wait we report shrinks as it elapses instead of asking for the original hour every time. The cache TTL then expires the key on its own. Both hosts set it. Measured today: music.apple.com returns 404 for a missing artist, bogus id or bogus storefront, so a 403 from it is not "no such content" -- and it answers from the same daiquiri origin that blocks itunes.apple.com, behind both Fastly and Akamai. Needs a new cache permission in the manifest, so upgrading users get a prompt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes. The first is a hotfix: #16 shipped a plugin that crashes on every call. The second is the follow-up that #16 should have included.
1.
errors.Astraps under TinyGoisThrottledusederrors.As, which callsreflectlite.AssignableTo. TinyGo only partly implements that and traps onunreachable. Simply making it reachable broke everything, becauseencoding/jsoncalls the same function fromnewTypeEncoderwhile decoding the request struct.nd_get_artist_imagesdied insidejson.Unmarshalbefore reaching Apple:Five in a row opened Navidrome's breaker, which then skipped apple-music for the rest of the run. The plugin issued zero requests, so no throttling behaviour was exercised at all. A type assertion is enough:
httpGetreturns the error unwrapped and both call sites test it immediately.I verified this against the real TinyGo artifacts with a wazero harness that stubs the six
extism:host/userimports and callsnd_get_artist_imageson a built.ndp:0.2.0-60a6100)httpGet0.2.0-80840df)AssignableTo0.2.0-PR17-02e2152)httpGetgo testnever caught it because it builds native Go, whereerrors.Asis fine. All 139 specs passed on the broken build. Nothing in this repo exercises the TinyGo artifact, which is how it reached a production server.2. Remember the back-off across every capability
Navidrome keeps two throttle stores that never talk to each other. Artwork uses the circuit breaker in
core/artwork/gate.go; biography, similar artists, top songs and artist URL use the cooldown map incore/agents. Neither tells the other, so a 429 the artwork path just absorbed does nothing to stop the metadata path calling the same API seconds later.The plugin sits below both, so it records the deadline in the cache host service and refuses every request until it passes, before any network call.
One shared key, because Navidrome's breaker keys on the agent name (
core/artwork/agent_images.go:76,gate(a.Name, ...)). Finer granularity inside the plugin cannot keep one code path alive while another is parked, so it would be state for nothing.It stores the deadline rather than the delay, so the wait it reports shrinks as it elapses instead of asking for the original hour every time. The cache TTL expires the key on its own.
This needs a new
cachepermission in the manifest, so upgrading users will see a prompt.Why both hosts set the cooldown
I first gated this to
itunes.apple.com, assuming a 403 frommusic.apple.commeant "not available in this storefront". That was wrong, and measuring it disproved it:/artist/-/1)/zz/)/va/)Missing content is a 404, so there is no "unavailable" 403 to confuse with a block. The two hosts are also not separate services: both answer
Server: daiquiri/5, the same backend family as iTunes'x-daiquiri-instance. So a 403 from either means blocked, and either may set the cooldown.Separately measured, for the record:
music.apple.comhas no observable rate limit. 105/105 HTTP 200 ramping 30 to 120 per minute, all reaching origin (69MISS, 36TCP_MISS, zero cache hits, 105 distinct artist IDs), noRetry-Afteror any rate-limit header, and no latency degradation under load. All throttling observed to date comes fromitunes.apple.com/search.What this does not fix
The rate itself. Navidrome's
DevArtworkExternalMaxRPSdefaults to 2 rps, so 120/min per agent, against roughly 30/min that iTunes sustains. A bulk artwork reprocess will still trip 429s. Live on 2026-09-03 the token worked exactly as intended, and we still earned a 403 twenty seconds later:Six bad responses, against 36 and accelerating before any of this existed. But a per-agent rate cap on the Navidrome side is the piece that prevents the block rather than reacting to it, and it is not in this PR.
Testing
go test -race ./..., 145 specs.New coverage: the cooldown is set on a 429 with the delay Apple named and on a 403 for an hour; a parked plugin refuses requests without touching the network; the reported wait shrinks as the deadline approaches and is zero once it passes.
The meaningful check for the TinyGo half is the harness table above, run against the artifact this PR's CI produces, since native
go testcannot see that class of failure.