fix(events): re-validate live socket keys against the database

CHANGELOG.md

@@ -16,6 +16,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Restoring a data archive into PostgreSQL from a gateway that does not run in UTC no longer shifts every timestamp by the host offset, and no longer shifts it again on each further restore ([#1624](https://github.com/rmyndharis/OpenWA/issues/1624)).
 - Retention sweeps on PostgreSQL delete the rows their window names instead of taking up to the host's UTC offset of younger rows with them, and the `today` message counts cover the host's local day.
 - Session leases on PostgreSQL compare as instants across nodes in different time zones and across a daylight-saving change.
+- Live WebSocket sockets are re-validated against the API-key table once a minute, so a key deleted, revoked, expired or narrowed on another node or by a direct database write drops its sockets there too, and a socket that connected while its key was being revoked no longer keeps that authorization for the life of the connection ([#1625](https://github.com/rmyndharis/OpenWA/issues/1625)).
+- A WebSocket subscribe whose socket is evicted while it is in flight no longer registers its rooms after the disconnect.
 
 ### Upgrade notes (behavior changes)
 
@@ -25,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - A table that has had an archive from a **SQLite** gateway restored into it holds those rows in correct UTC while the app wrote its own in local time. The two are indistinguishable within the column, so a blanket `UPDATE` would move the rows that are already right; correct such a column row by row against a known archive, or leave it as it is.
 - An archive a **PostgreSQL** gateway outside UTC exported before this release is shifted the other way, and restoring it moved the whole table, `DEFAULT now()` columns included: the export read every value back through the local-time parser and the import bound the resulting ISO text into a column that drops the zone, so each restore took the table one offset backward. No row in such a table is correct, and the conversion above would move those rows a further offset the wrong way. Where the table holds nothing but restored rows, apply the inverse once per restore taken: `UPDATE sessions SET "createdAt" = ("createdAt" AT TIME ZONE 'UTC') AT TIME ZONE 'Asia/Jakarta';`. Where it also holds rows written after the restore, no blanket conversion is safe.
 - Re-export after upgrading for an archive whose stamps are the instants they claim; restoring a pre-release archive carries its shift in as it is.
+- A WebSocket client can now be disconnected with an `UNAUTHORIZED` frame up to a minute after its key changed, where before only the node that processed the change disconnected it; reconnect and resubscribe on that frame. A rename, and the key's usage counters, evict nobody.
 
 ## [0.23.5] - 2026-09-15
 

docs/06-api-specification.md

@@ -6785,7 +6785,8 @@ A subscribe request whose `events` array contains no recognized name (after filt
 - **`sessionId: "*"`** subscribes to every session; **`events: ["*"]`** subscribes to every subscribable event. They combine (e.g. `"*"` + `["*"]` = every event of every session).
 - The API key is **re-validated on every `subscribe`** (not just at connect), so a key revoked or expired mid-connection is caught — the server replies `UNAUTHORIZED` and disconnects.
 - **Per-key session scope is enforced** against the fresh key: a key restricted via `allowedSessions` may NOT subscribe to `"*"` and may NOT subscribe to a session outside its allowlist — either is rejected with `FORBIDDEN_SESSION`. An unrestricted key (no `allowedSessions`) may subscribe to anything, including `"*"`.
-- **`session.qr` requires the OPERATOR role**, matching `GET /api/sessions/{sessionId}/qr`. A VIEWER key may still subscribe to it, by name or through a wildcard, but the QR is never delivered to its sockets; every other event is. The role is read from the key re-validated on each `subscribe`, so a key narrowed to VIEWER stops receiving the QR from its next `subscribe` (a role change also disconnects the key's sockets).
+- **Live sockets are re-validated against the database once a minute**, with no client activity required. A socket carries the key as it stood when it connected, and rooms joined earlier are never revisited, so that snapshot is what the sweep compares against the current row, along with any later `subscribe` whose key no longer matched it (what that `subscribe` granted outlives the change, so putting the row back does not spare the socket). It closes the key's sockets with an `UNAUTHORIZED` frame naming the cause: `API key has been deleted`, `API key has been revoked`, `API key has expired`, or `API key authorization changed; please reconnect` when `role`, `allowedIps`, `allowedSessions` or `expiresAt` moved. A change made through this API still evicts synchronously in the same request; the sweep is what catches a change made on another node, written straight to the database, or committed in the instant a socket was connecting. A rename, and the usage counters the gateway itself writes, evict nobody. Treat these frames as "reconnect and resubscribe", not as fatal.
+- **`session.qr` requires the OPERATOR role**, matching `GET /api/sessions/{sessionId}/qr`. A VIEWER key may still subscribe to it, by name or through a wildcard, but the QR is never delivered to its sockets; every other event is. The role is read from the key re-validated on each `subscribe`, so a key narrowed to VIEWER stops receiving the QR from its next `subscribe`. Between subscribes the QR gate rests on the same snapshot as every other event: a key demoted right after a `subscribe` keeps receiving the QR until its sockets are evicted, which is immediate on the node processing the change and within the sweep's minute anywhere else.
 
 ### Example (socket.io-client)
 

docs/10-devops-infrastructure.md

@@ -270,9 +270,9 @@ volumes:
 > **Keep `replicas: 1`.** Session ownership gained claim/lease fencing (`nodeId` owner +
 > `leaseExpiresAt`), which bounds any two-engine overlap on one session to roughly one heartbeat
 > interval instead of eliminating it — and docs/13 still says DO NOT run its multi-replica examples
-> yet: process-local key eviction, WebSocket rate-limit buckets, the unfenced liveness watchdog,
-> bulk-batch state and MCP locality all remain per-process. Follow
-> [13 - Horizontal Scaling Guide](./13-horizontal-scaling.md) for the full list and the design
+> yet: WebSocket key eviction on a peer node lags by up to a minute, and WebSocket rate-limit
+> buckets, the unfenced liveness watchdog, bulk-batch state and MCP locality all remain per-process.
+> Follow [13 - Horizontal Scaling Guide](./13-horizontal-scaling.md) for the full list and the design
 > sketch. What multi-node eventually buys is engine capacity, not shared engine state: live engine
 > handles live in exactly one process's `EngineRegistry` (`src/engine/engine-registry.service.ts`),
 > and the hard requirements include a stable `NODE_ID` across restarts, NTP-synced clocks (lease skew

docs/13-horizontal-scaling.md

@@ -99,14 +99,25 @@
 > the same flag the throttler and cache already use). The gateway broadcasts to rooms; a Redis
 > pub/sub adapter attached to Socket.IO relays those broadcasts to every replica, so a client
 > connected to node A receives an event raised on node B. Scope honestly: this distributes event
-> **fan-out only**. Mid-connection key eviction (`socketsByKeyId`) is still process-local — a key
-> revoked on node A tears down only A's sockets — as are the per-key WS rate-limit buckets (counted
-> per replica) and the engine registry. Without `REDIS_ENABLED` the adapter is inert and delivery
-> is single-node, exactly as before.
+> **fan-out only**. The per-key WS rate-limit buckets (counted per replica) and the engine registry
+> are still process-local. Without `REDIS_ENABLED` the adapter is inert and delivery is single-node,
+> exactly as before.
 >
-> **What does not exist yet, and is why one replica is still the answer.** The cross-replica gaps
-> just named (key eviction, WS rate-limit state) remain process-local. Not every lifecycle path is
-> fenced: the liveness watchdog and reconnect timers still act on whatever is in the local
+> **Mid-connection key eviction converges on a timer, not a broadcast.** The node that processes a
+> revoke, delete, or narrowing tears down that key's sockets synchronously, in the same request.
+> Nothing is published to peers; instead every node re-validates the keys behind its own live
+> sockets against the database once a minute (`EventsGateway.sweepApiKeyAuthorization`, one batched
+> read of the key ids currently holding sockets) and evicts on a row that is gone, inactive, expired,
+> or whose role, `allowedIps`, `allowedSessions` or expiry no longer matches the snapshot the socket
+> authenticated with. So a peer node's sockets close within a minute of the change. Before, a revoke,
+> delete or expiry there waited for the client's next subscribe, and a narrowing was never caught at
+> all: it leaves the key valid, so only the new subscribe is rejected while every room joined earlier
+> stays joined. That minute is the current worst case for a socket streaming events its key has just
+> lost; a key's REST calls are rejected immediately everywhere, since REST reads the row per request.
+>
+> **What does not exist yet, and is why one replica is still the answer.** The cross-replica gap just
+> named (WS rate-limit state) remains process-local. Not every lifecycle path is fenced: the
+> liveness watchdog and reconnect timers still act on whatever is in the local
 > registry. `BulkMessageService` keeps its live batch state in process, so a takeover cannot resume
 > a batch — only fail it. MCP/agent tool invocations execute on the node that received them rather
 > than being forwarded.

src/modules/auth/api-key-authorization.ts

@@ -0,0 +1,48 @@
+import type { ApiKey } from './entities/api-key.entity';
+
+/**
+ * Collapse an `allowedSessions` list to the two shapes the enforcement sites actually distinguish.
+ *
+ * The column is `simple-array`: TypeORM joins on write and splits on read, so `['']` is stored as
+ * `''` and read back as `[]`. Every site treats a zero-length list as "every session", so a write
+ * that looked like a scoping landed as a widening. The DTO validator now refuses such an entry at
+ * the boundary, and this is the second half: whatever reaches storage is either a non-empty list of
+ * real ids, or NULL.
+ *
+ * NULL rather than `[]` on purpose. Both already exist in the table for the same intent, and the
+ * published contract says an unscoped key omits the field, which only NULL produces.
+ */
+export function normalizeScopeList(list: string[] | null | undefined): string[] | null {
+  if (list == null) return null;
+  const cleaned = list.map(entry => entry.trim()).filter(entry => entry.length > 0);
+  return cleaned.length > 0 ? cleaned : null;
+}
+
+/** An `expiresAt` in milliseconds, or null when it is unset or unparseable (a Date from the driver, a string from a snapshot). */
+export function apiKeyExpiryTime(value: Date | string | null | undefined): number | null {
+  if (!value) return null;
+  const time = value instanceof Date ? value.getTime() : new Date(value).getTime();
+  return Number.isFinite(time) ? time : null;
+}
+
+/** The columns that decide what a key may do. Everything else on the row is descriptive or advisory. */
+export type ApiKeyAuthorization = Pick<ApiKey, 'role' | 'allowedIps' | 'allowedSessions' | 'expiresAt'>;
+
+/**
+ * A stable string for what a key is AUTHORIZED to do: role, IP allowlist, session scope, expiry.
+ * Two rows sharing a fingerprint authorize identically, so any other column moving must leave it
+ * untouched, in particular `lastUsedAt`/`usageCount`/`updatedAt`, which the usage tracker rewrites
+ * for every key in active use. A fingerprint that moved with them would disconnect every live
+ * WebSocket client on the next windowed statistics write.
+ *
+ * Membership, not order: both allowlists are enforced with `.includes()`, so a reorder authorizes
+ * exactly the same and is sorted away here. `''`, `[]` and NULL all mean "unscoped" at every
+ * enforcement site and normalize to the same value, so a legacy row does not read as a change.
+ */
+export function apiKeyAuthorizationFingerprint(key: ApiKeyAuthorization): string {
+  const scope = (list: string[] | null | undefined): string[] | null => {
+    const normalized = normalizeScopeList(list);
+    return normalized ? [...normalized].sort() : null;
+  };
+  return JSON.stringify([key.role, scope(key.allowedIps), scope(key.allowedSessions), apiKeyExpiryTime(key.expiresAt)]);
+}

src/modules/auth/auth.service.ts

@@ -8,7 +8,7 @@ import {
 } from '@nestjs/common';
 import { ModuleRef } from '@nestjs/core';
 import { InjectRepository } from '@nestjs/typeorm';
-import { Repository, UpdateQueryBuilder, DeleteQueryBuilder, type QueryDeepPartialEntity } from 'typeorm';
+import { In, Repository, UpdateQueryBuilder, DeleteQueryBuilder, type QueryDeepPartialEntity } from 'typeorm';
 import { randomBytes } from 'crypto';
 import { ipMatches } from '../../common/utils/ip';
 import { hashApiKey } from './api-key-hash';
@@ -17,6 +17,7 @@ import { CreateApiKeyDto, UpdateApiKeyDto } from './dto';
 import { createLogger } from '../../common/services/logger.service';
 import { readBootstrapKey, removeBootstrapKey, writeBootstrapKey } from './bootstrap-key-file';
 import { ApiKeyUsageTracker } from './api-key-usage-tracker.service';
+import { apiKeyAuthorizationFingerprint, normalizeScopeList } from './api-key-authorization';
 import { EventsGateway, type ApiKeyEvictionReason } from '../events/events.gateway';
 
 /**
@@ -49,24 +50,6 @@ export function bannerKeyLine(displayKey: string, isNewKey: boolean): string {
   return `${displayKey.slice(0, 8)}… (full key in data/.api-key or the dashboard)`;
 }
 
-/**
- * Collapse an `allowedSessions` list to the two shapes the enforcement sites actually distinguish.
- *
- * The column is `simple-array`: TypeORM joins on write and splits on read, so `['']` is stored as
- * `''` and read back as `[]`. Every site treats a zero-length list as "every session", so a write
- * that looked like a scoping landed as a widening. The DTO validator now refuses such an entry at
- * the boundary, and this is the second half: whatever reaches storage is either a non-empty list of
- * real ids, or NULL.
- *
- * NULL rather than `[]` on purpose. Both already exist in the table for the same intent, and the
- * published contract says an unscoped key omits the field, which only NULL produces.
- */
-function normalizeScopeList(list: string[] | null | undefined): string[] | null {
-  if (list == null) return null;
-  const cleaned = list.map(entry => entry.trim()).filter(entry => entry.length > 0);
-  return cleaned.length > 0 ? cleaned : null;
-}
-
 @Injectable()
 export class AuthService implements OnModuleInit, OnModuleDestroy {
   private readonly logger = createLogger('AuthService');
@@ -276,21 +259,10 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
       saved = await this.findOne(id);
     }
 
-    // Compare membership, not order: a pure reorder of allowedIps/allowedSessions is a no-op for the
-    // .includes()-based enforcement, so sort before stringify to avoid a spurious eviction on a reorder.
-    // Normalize before comparing: a legacy row stored as '' reads back as [], which means the same
-    // as NULL at every enforcement site, so treating them as different would evict live sockets for a
-    // write that changed nothing.
-    const ordered = (v: string[] | null) => {
-      const normalized = normalizeScopeList(v);
-      return normalized ? [...normalized].sort() : null;
-    };
-    const authzChanged =
-      saved.role !== before.role ||
-      saved.expiresAt?.getTime() !== before.expiresAt?.getTime() ||
-      JSON.stringify(ordered(saved.allowedIps)) !== JSON.stringify(ordered(before.allowedIps)) ||
-      JSON.stringify(ordered(saved.allowedSessions)) !== JSON.stringify(ordered(before.allowedSessions));
-    if (authzChanged) {
+    // One fingerprint definition, two callers: this immediate eviction and the gateway's periodic
+    // re-validation sweep. Sharing it keeps the two from disagreeing about what an authorization
+    // change is (membership over order, '' and NULL alike, usage statistics ignored).
+    if (apiKeyAuthorizationFingerprint(saved) !== apiKeyAuthorizationFingerprint(before)) {
       this.evictActiveSockets(id, 'authorization_changed');
     }
     return saved;
@@ -447,6 +419,17 @@ export class AuthService implements OnModuleInit, OnModuleDestroy {
     }
   }
 
+  /**
+   * The current rows for a set of key ids, in one statement. Feeds the WebSocket gateway's periodic
+   * re-validation of the keys behind its live sockets: an id whose row is gone simply comes back
+   * absent, which the gateway reads as deleted. Usage statistics are deliberately not recorded here,
+   * so a passive socket does not look like traffic.
+   */
+  async findAuthorizationStates(ids: string[]): Promise<ApiKey[]> {
+    if (ids.length === 0) return [];
+    return this.apiKeyRepository.findBy({ id: In(ids) });
+  }
+
   async validateApiKey(rawKey: string, clientIp?: string, sessionId?: string): Promise<ApiKey> {
     // Trim before hashing so every surface agrees on what the credential is. HTTP already strips
     // surrounding whitespace from header values, so a pasted key with a stray space/newline