Skip to content

server: drop stale notifications queued while a peer session was down - #3578

Closed
notsrch wants to merge 1 commit into
osrg:masterfrom
notsrch:fix-3561-bfd-stale-reset
Closed

server: drop stale notifications queued while a peer session was down#3578
notsrch wants to merge 1 commit into
osrg:masterfrom
notsrch:fix-3561-bfd-stale-reset

Conversation

@notsrch

@notsrch notsrch commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #3561.

What was wrong

newFSM creates fsm.notification once per peer, and only the ESTABLISHED loop reads it, so the
channel outlives every session of the peer. ResetPeer and ShutdownPeer enqueue into it without
checking the session state. A NOTIFICATION queued while the session is down waits in the channel,
and the next session reads it the moment it reaches ESTABLISHED and tears itself down. BFD produces
exactly this sequence: its detect timer expires seconds after the BGP session already ended and
queues an administrative reset. Against a peer that destroys its BFD session when BGP goes down,
this repeats on every reconnect and the peering never recovers (#3561 has the full trace).

What this changes

handleFSMMessage now drops whatever is still queued in fsm.notification on every state
transition, next to the existing outgoingCh drain.

This is the issue's Option 1 in spirit, with one correction to its wording. Draining when the FSM
leaves ESTABLISHED would not break the loop: in the reported cycle, BFD queues the reset about
three seconds after teardown, while the peer is IDLE. The drain has to run between that enqueue
and the next delivery point, so it runs on every transition instead.

Why this placement is race-free: the transition callback runs under the server's read lock, and the
producers (ResetPeer, ShutdownPeer, including BFD's calls) run under the write lock. An enqueue
therefore lands either before the transition into ESTABLISHED — and is dropped, correctly, because
the reset targeted a session that no longer exists — or after it, when the established loop is live
and delivers it. A BFD failure detected moments after Peer Up still resets the session.

Draining at the top of established() instead would race the producers: a legitimate reset issued
right after Peer Up could be destroyed, and BFD would not repeat it (expiry() stops itself once
its state is DOWN). That would suppress a real BFD failure, which is the one behavior the fix must
protect.

The drain also covers a rarer variant of the same bug: the receive goroutine queues NOTIFICATIONs
for parse errors, and one left behind when the session dies concurrently was likewise delivered to
the next session. The teardown joins that goroutine before the state function returns, so the next
transition's drain disposes of it deterministically.

Intended behavior change: a hard ResetPeer or ShutdownPeer against a peer whose session is down
is now a no-op, instead of a deferred kill of the next session.

Checks from the issue

  • The loop ends: TestResetPeerWhileDownDoesNotResetNextSession reproduces the exact BFD call
    (ResetPeer with Soft: false while the peer is down) and fails on master — the next session
    dies with a NOTIFICATION on its connection moments after establishing. With the fix it stays
    ESTABLISHED and the channel is empty.
  • A real BFD failure on an established session still tears BGP down promptly:
    TestResetPeerEstablishedSendsNotification asserts the session leaves ESTABLISHED and a
    cease/administrative-reset NOTIFICATION reaches the wire. It passes with and without the fix.
  • A BFD session that never comes up still leaves BGP alone: unchanged; the expiry() guard and the
    existing BFD tests cover it.

go test -race ./pkg/server/ passes. Option 3 from the issue (reset only on a real Up → Down BFD
transition) is left for a follow-up PR as suggested there.

Assisted-by: Claude Fable 5 noreply@anthropic.com

newFSM creates fsm.notification once per peer and only the ESTABLISHED
loop reads it, so the channel outlives every session of the peer.
ResetPeer and ShutdownPeer enqueue into it without checking the session
state. A NOTIFICATION queued while the session is down waits in the
channel, and the next session reads it the moment it reaches
ESTABLISHED and tears itself down. BFD produces exactly this sequence:
its detect timer expires seconds after the BGP session already ended
and queues an administrative reset. Against a peer that destroys its
BFD session when BGP goes down, FRR with a dynamic neighbor for
example, every reconnect dies immediately and the peering never
recovers.

Drop whatever is still queued on every state transition, next to the
outgoingCh drain. The transition callback runs under the server's read
lock and the producers run under the write lock, so an enqueue lands
either before the transition into ESTABLISHED, where dropping is right
because the reset targeted a session that no longer exists, or after
it, when the established loop is live and delivers it. A BFD failure
on an established session still resets it promptly. The drain also
covers the NOTIFICATIONs the receive goroutine queues for parse
errors: teardown joins that goroutine before the state function
returns, so a message it left behind is dropped on the next
transition.

A hard ResetPeer or ShutdownPeer against a peer whose session is down
is now a no-op instead of a deferred kill of the next session.

Assisted-by: Claude Fable 5 <noreply@anthropic.com>
@fujita

fujita commented Sep 1, 2026

Copy link
Copy Markdown
Member

Fixed the lint error and pushed. Thanks!

@fujita fujita closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BFD: a stale ADMINISTRATIVE_RESET kills the next BGP session, causing a permanent flap loop

2 participants