fix(p2p/sensor): keep writing when ClickHouse is down at startup

[minhd-vu]

Description

connectClickHouse treated a failed startup ping as fatal. NewClickHouse then returned a no-op instance that discarded every row for the lifetime of the process and never reconnected — the sensor peered, tracked the head, and looked entirely healthy while writing nothing.

Nothing about that was necessary. The recovery path already existed:

• clickhouse.Open performs no I/O. The pool dials lazily on first use and replaces broken connections.
• newInsertBatcher already retries every flush chMaxFlushAttempts times and logs each failure with its table and row count.

The hard failure was the only thing preventing us from ever reaching that machinery. So the ping becomes diagnostic — logged at warn — and the connection is returned regardless.

c.conn is still assigned exactly once before any goroutine reads it, so this needs no synchronisation and introduces no reconnect goroutine.

What prompted it

A sensor fleet and its ClickHouse VM were replaced in a single Terraform apply. The sensors pinged at 18:35:37; ClickHouse finished binding 9000 at 18:35:49 — twelve seconds later. All seven sensors latched "unavailable" and discarded roughly 2.5M rows over ~30 minutes, until every host was restarted by hand. system.query_log showed zero INSERTs for the whole window.

Restarting the containers fixed it immediately, which is what confirmed the connection logic, not the environment, was the problem.

Behaviour change worth reviewing

A genuinely bad DSN or password no longer fails fast. I don't think that is a regression — the pre-existing comment in this file records an auth failure where "two sensors ran for an hour writing nothing", so the old path degraded to the same silent no-op. It now surfaces as a per-flush warning naming the table and error, which says more than the single startup line that used to scroll past. But it does mean a misconfigured DSN presents as repeated flush warnings rather than one fatal error, and reviewers may prefer a different trade-off.

I also updated the startUnavailableWarning comment, which claimed "a sensor whose database was down at boot needs a restart". That is no longer true — the path now only runs for DSN/config failures.

Jira / Linear Tickets

• DVT-000

Testing

• go build ./p2p/...
• go vet ./p2p/database/
• go test ./p2p/database/... — passes
• Not covered by any test, and the passing suite does not demonstrate the fix. The behaviour that matters is: start a sensor with ClickHouse down, bring ClickHouse up, confirm INSERTs begin without a restart. Worth adding before this merges — happy to write it if you want it in this PR.
• Verify a deliberately bad password still produces visibly failing flushes rather than silence

🤖 Generated with Claude Code