Skip to content

Telegram channel permanently goes down when the polling loop exits — no auto-reconnect (Discord gateway has one) #409

Description

@gokcanakalin

Summary

When the Telegram polling loop (grammY bot.start()) exits for any reason — including a transient
network error on the getUpdates long-poll — TelegramChannel releases its lockfile, transitions to
state down, and never tries to reconnect. The agent process keeps running and looks healthy; every
subsequent Telegram message is silently ignored until the user fully restarts atomic-agent. grammY
itself does not auto-reconnect, and no supervision exists around it.

Notably, the Discord channel in the same codebase implements auto-reconnect with exponential
backoff
(500 ms → 60 s cap, plus jitter, heartbeat watchdog, session resume). The Telegram channel
has none of this, which makes this look like an oversight rather than a design choice.

Root cause (decompiled from the v0.6.0 binary)

TelegramChannel.handlePollingStopped:

handlePollingStopped(e, r) {
    if (this.stopRequested || this.bot !== e) return;
    let n = r === void 0 ? "polling stopped unexpectedly" : Ku(r);
    this.deps.logger.warn("telegram: polling loop ended", { reason: n });
    this.bot = null;
    try { this.lock.release() } catch {}   // telegram.lock deleted
    this.transition("down", n);            // terminal — no retry, ever
}

Bot factory wiring (grammY's start() promise settles when polling exits, including on network errors):

start(h, g) { n.start({onStart: h}).then(() => g?.()).catch(m => g?.(m)) }

By contrast, the Discord gateway in the same binary:

function backoffMs(attempt) {
  return Math.floor(Math.random() * Math.min(60_000, 1000 * 2 ** Math.min(attempt, 6))) + 500;
}
// on disconnect (non-fatal): attempt++, warn("discord: gateway disconnected, retrying"), sleep(backoffMs(attempt))

There are also no config keys to enable retry behavior: telegram.autoRetry /
telegram.reconnectDelay do not exist in the schema (verified: zero occurrences in the v0.6.0 binary).

Observed (captured live, 2026-09-12, passive read-only watch)

Time Observation
19:05:58 turn finished, reply sent (session trace)
19:06–19:12 channel object still "up" (lock held) but no TCP connection to the Telegram DC — zombie state; a new turn was still ingested and ran in this window
~19:12:20 telegram.lock deleted → handlePollingStopped → state down (terminal)
19:12:59 next turn finished with reply — sent into the dead channel; no message ingested afterwards

Same signature captured twice earlier the same day (channel dead with process alive at 17:27; whole
daemon silently dead at ~15:52–16:24), and matching historical incidents recorded locally since
2026-08-31 ("lock released, state down, no auto-retry").

Additional pain point: the loop-exit reason is only rendered in the TUI, which repaints over it —
nothing reaches stderr or a log file, so post-mortem diagnosis is impossible unless someone is watching.

Reproduction

  1. Enable + pair the Telegram channel; confirm telegram.lock exists and a TCP connection to a
    Telegram DC (149.154.0.0/16, 91.108.0.0/16, or 2001:67c:4e8::/48) is established.
  2. Simulate a transient network blip:
    sudo iptables  -A OUTPUT -d 149.154.160.0/20 -p tcp --dport 443 -j DROP
    sudo ip6tables -A OUTPUT -d 2001:67c:4e8::/48 -p tcp --dport 443 -j DROP
    
  3. Wait 60–120 s, then remove both rules.
  4. Observe: telegram.lock gone, no reconnection attempt, agent process still running, all Telegram
    messages unanswered until a full restart.

Expected

Transient polling failures should reconnect with backoff — like the Discord gateway already does.

Suggested fixes

  1. Supervise the polling loop: on unexpected stop (not user-requested), re-run start() with
    exponential backoff + jitter (the Discord gateway's backoffMs pattern can be reused verbatim).
  2. Add telegram.autoRetry (default true) and telegram.reconnectDelayMs config keys.
  3. Persist channel state transitions + loop-exit reasons to a log file (or stderr) for post-mortems.

Workaround

External watchdog: restart atomic-agent whenever the process is gone or telegram.lock is missing
(the lockfile's presence while state is up makes this a reliable health signal at steady state).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions