v2.9.1: events are inserted but receive stale timestamps after upgrade

[Perrschy]

Description

After upgrading a self-hosted Rybbit instance from 2.8.0 to the current latest backend image (the container reports rybbit-backend@2.9.1), pageviews are accepted and inserted into ClickHouse, but all newly inserted rows receive old timestamps.

The dashboard therefore appears frozen at the upgrade time although the events table keeps growing.

Environment

• Self-hosted Docker Compose deployment
• Backend reports: rybbit-backend@2.9.1
• ClickHouse: 26.3.17.4
• Database: analytics
• Backend uses 4 cluster workers
• Tracking endpoint: POST /api/track
• Events are forwarded server-to-server from a first-party collector

Evidence

At the start of the investigation:

SELECT count(), max(timestamp) FROM analytics.events

8913097    2026-09-15 14:57:22

Later, after many successful incoming pageviews:

SELECT count() AS rows, max(timestamp) AS newest_event FROM events

8915273    2026-09-15 14:57:22

So 2,176 rows were added, but max(timestamp) did not advance.

The backend's pageviewQueue was temporarily instrumented. For every received pageview it logged both:

Pageview queue batch started
Pageview queue insert starting

No Error processing pageview queue errors occurred.

Both containers have the same correct current UTC time:

backend:    Tue Sep 15 14:06:26 UTC 2026
clickhouse: Tue Sep 15 02:06:26 PM UTC 2026

ClickHouse itself reported approximately 16:03 in Europe/Berlin, while the newest event remained at 14:57:22 Europe/Berlin.

I also tested disabling ClickHouse async inserts explicitly:

<async_insert>0</async_insert>

The setting was active at runtime, but the symptom remained unchanged.

## Expected behavior

Each accepted POST /api/track request should create an events row with the current server receive time, and the dashboard should show current traffic.

## Actual behavior

Rows are inserted, but their timestamps are stale/backdated. Dashboard and time-based queries stop progressing even though row count increases.

## Additional note

The forwarding collector had to truncate an excessively long referrer because 2.9.1 now correctly rejects referrers over 2048 characters. That validation issue was fixed separately and is not the cause of the stale timestamps.

[Perrschy]

Root cause found: this was not an async-insert or queue failure.

The backend and ClickHouse container clocks are both correct and synchronized (UTC). New rows were being inserted successfully: the events count increased by 2,176 while max(timestamp) remained stuck at 2026-09-15 14:57:22.

Our ClickHouse server timezone is Europe/Berlin. In pageviewQueue.ts, Rybbit 2.9.1 does:

const receivedAt = DateTime.fromISO(pv.timestamp).toUTC();

and then writes receivedAt as an unzoned yyyy-MM-dd HH:mm:ss string to ClickHouse. ClickHouse interprets that string in its configured Europe/Berlin timezone. During CEST this shifts newly inserted events two hours into the past.

As a temporary local workaround, changing this to the ClickHouse timezone fixes the timestamps immediately:

const receivedAt = DateTime.fromISO(pv.timestamp).setZone("Europe/Berlin");

This seems to be a timezone-handling regression/incompatibility: the backend should either format values in ClickHouse’s configured timezone or send timezone-aware/typed timestamps, rather than converting to UTC and inserting an unzoned string.