Snuba API Memory Leak

[lfritz-seweurodrive]

Environment

Latest

Steps to Reproduce

1. Host Sentry via Helm Chart
2. Snuba memory usage continues to rise

This issue is also affecting several others attempting to self-host Sentry, see here:
sentry-kubernetes/charts#1113

Expected Result

What you thought would happen:
The memory usage not to climb without end

Actual Result

It climbs until it reaches its resource memory limit and crashes.

Additional infobits

In a different issue for the taskworker, the suggested solution was setting a single parameter.
Perhaps there's also a parameter used for the snuba API.

[linear-code]

EAP-470

[TheNormalnij]

The same memory leak happens on docker compose for Sentry 26.2.1

[grindlife]

I am experiencing a similar memory leak issue in Sentry self-hosted 26.5.2.
The memory usage of snuba-api keeps increasing by roughly 20MB per hour after accessing/viewing issues on the web UI.

[brianmunk]

Same thing, we are doing daily reboots with a cronjob due to the 'snuba-api' selfhosted.
Can confirm excessive memory consumption over time starting from 26.2.1 to atleast 26.5.1 and we suspect this to be the issue.
We also just access issues via the web UI.

[phacops]

Thanks for the reports — this is a real, long-standing snuba-api RSS climb on self-hosted (Helm and docker-compose), not just one bad deploy.

What we know

• Multiple operators see snuba-api grow after opening issues in the UI (~20MB/h in 26.5.2) until the cgroup/memory limit, then crash. Helm thread: Why snuba-api pod’s memory usage rate continues to rise?? sentry-kubernetes/charts#1113 (back to chart 20 / Snuba 23.6.1).
• 26.2.1–26.5.2 still use the clickhouse-driver native pool (snuba/clickhouse/native.py), not clickhouse-connect. The later HTTP client cache / query_limit=0 comments in snuba/clickhouse/connect.py are not what those versions are running.
• Viewing issues hits SNQL on the events path (/events/snql and friends). That is expected; it is not EAP protobuf RPC.

What is not an unbounded leak

I walked the API process on origin/master and on the 26.5.2 native path:

• Native pool is a fixed LifoQueue (CLICKHOUSE_MAX_POOL_SIZE = 25). Connections stay warm; they do not grow without bound.
• Query result cache is Redis (snuba/web/db_query.py, snuba/state/cache/redis/backend.py), not an in-process map.
• HTTPWriteBatch(buffer_size=0) is the consumer INSERT path (snuba/clickhouse/http.py), not issue-view reads.
• Current connect _clients dict is keyed by timeout/retry/query_limit and drops stale option snapshots. Shared urllib3 managers are process-wide and sized.
• Flask app object, allocation policies, and querylog recording do not keep per-request result sets.

So there is no small, reviewable “delete this unbounded dict” fix I can prove from the code.

Why it became more visible after 26.x

#7566 replaced uWSGI with Granian. Self-hosted compose still sets UWSGI_MAX_REQUESTS=10000, which Granian ignores. #7735 added recycle knobs, but they default to off:

• snuba/settings/settings_self_hosted.py: SNUBA_API_WORKERS_MAX_RSS / SNUBA_API_WORKERS_LIFETIME → None
• snuba/cli/api.py + snuba/utils/server.py: passed through to Granian workers_max_rss / workers_lifetime

SaaS recycles workers. Self-hosted does not, so allocator fragmentation and anything the native client retains on a pooled connection (Client.last_query after each execute) never get a process restart. That matches a slow, query-driven RSS slope rather than a runaway cache.

Helm charts that never set UWSGI_MAX_REQUESTS would have shown the same climb even before Granian (and did).

Workaround (not a root-cause fix)

Recycle the API worker before OOM, same idea as the old uWSGI max-requests:

# docker-compose / helm env on snuba-api
SNUBA_API_WORKERS_MAX_RSS=800   # MiB; pick under your pod/container limit
# optional extra belt
SNUBA_API_WORKERS_LIFETIME=3600

That is read from settings_self_hosted.py and handed to Granian. It will flap the worker; it will not hide a true leak if we later find one.

What we still need to find the actual retainer

If someone can catch a climbing process before it is killed:

1. tracemalloc snapshot delta after N issue-view SNQL requests (or py-spy dump / memray attach).
2. Confirm whether RSS falls after gc.collect() (fragmentation) or stays (retained objects / C heap).
3. Check len(pool.pool.queue) / live clickhouse_driver.Client count — should be ≤ 25 per node/settings key.
4. Note Snuba tag, container RSS vs Python heaps, and whether growth happens with only issue views (no consumers in the same process — they should already be separate).

Until we have a heap dump that names a retainer, I am leaving this open rather than shipping a guessed cache cap. Defaulting SNUBA_API_WORKERS_MAX_RSS in self-hosted/helm is the product follow-up if we want the workaround on by default.