server: drop stale notifications queued while a peer session was down - #3578
Closed
notsrch wants to merge 1 commit into
Closed
server: drop stale notifications queued while a peer session was down#3578notsrch wants to merge 1 commit into
notsrch wants to merge 1 commit into
Conversation
newFSM creates fsm.notification once per peer and only the ESTABLISHED loop reads it, so the channel outlives every session of the peer. ResetPeer and ShutdownPeer enqueue into it without checking the session state. A NOTIFICATION queued while the session is down waits in the channel, and the next session reads it the moment it reaches ESTABLISHED and tears itself down. BFD produces exactly this sequence: its detect timer expires seconds after the BGP session already ended and queues an administrative reset. Against a peer that destroys its BFD session when BGP goes down, FRR with a dynamic neighbor for example, every reconnect dies immediately and the peering never recovers. Drop whatever is still queued on every state transition, next to the outgoingCh drain. The transition callback runs under the server's read lock and the producers run under the write lock, so an enqueue lands either before the transition into ESTABLISHED, where dropping is right because the reset targeted a session that no longer exists, or after it, when the established loop is live and delivers it. A BFD failure on an established session still resets it promptly. The drain also covers the NOTIFICATIONs the receive goroutine queues for parse errors: teardown joins that goroutine before the state function returns, so a message it left behind is dropped on the next transition. A hard ResetPeer or ShutdownPeer against a peer whose session is down is now a no-op instead of a deferred kill of the next session. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Member
|
Fixed the lint error and pushed. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3561.
What was wrong
newFSMcreatesfsm.notificationonce per peer, and only the ESTABLISHED loop reads it, so thechannel outlives every session of the peer.
ResetPeerandShutdownPeerenqueue into it withoutchecking the session state. A NOTIFICATION queued while the session is down waits in the channel,
and the next session reads it the moment it reaches ESTABLISHED and tears itself down. BFD produces
exactly this sequence: its detect timer expires seconds after the BGP session already ended and
queues an administrative reset. Against a peer that destroys its BFD session when BGP goes down,
this repeats on every reconnect and the peering never recovers (#3561 has the full trace).
What this changes
handleFSMMessagenow drops whatever is still queued infsm.notificationon every statetransition, next to the existing
outgoingChdrain.This is the issue's Option 1 in spirit, with one correction to its wording. Draining when the FSM
leaves ESTABLISHED would not break the loop: in the reported cycle, BFD queues the reset about
three seconds after teardown, while the peer is IDLE. The drain has to run between that enqueue
and the next delivery point, so it runs on every transition instead.
Why this placement is race-free: the transition callback runs under the server's read lock, and the
producers (
ResetPeer,ShutdownPeer, including BFD's calls) run under the write lock. An enqueuetherefore lands either before the transition into ESTABLISHED — and is dropped, correctly, because
the reset targeted a session that no longer exists — or after it, when the established loop is live
and delivers it. A BFD failure detected moments after Peer Up still resets the session.
Draining at the top of
established()instead would race the producers: a legitimate reset issuedright after Peer Up could be destroyed, and BFD would not repeat it (
expiry()stops itself onceits state is DOWN). That would suppress a real BFD failure, which is the one behavior the fix must
protect.
The drain also covers a rarer variant of the same bug: the receive goroutine queues NOTIFICATIONs
for parse errors, and one left behind when the session dies concurrently was likewise delivered to
the next session. The teardown joins that goroutine before the state function returns, so the next
transition's drain disposes of it deterministically.
Intended behavior change: a hard
ResetPeerorShutdownPeeragainst a peer whose session is downis now a no-op, instead of a deferred kill of the next session.
Checks from the issue
TestResetPeerWhileDownDoesNotResetNextSessionreproduces the exact BFD call(
ResetPeerwithSoft: falsewhile the peer is down) and fails on master — the next sessiondies with a NOTIFICATION on its connection moments after establishing. With the fix it stays
ESTABLISHED and the channel is empty.
TestResetPeerEstablishedSendsNotificationasserts the session leaves ESTABLISHED and acease/administrative-reset NOTIFICATION reaches the wire. It passes with and without the fix.
expiry()guard and theexisting BFD tests cover it.
go test -race ./pkg/server/passes. Option 3 from the issue (reset only on a real Up → Down BFDtransition) is left for a follow-up PR as suggested there.
Assisted-by: Claude Fable 5 noreply@anthropic.com