[WPB-28375] Move sync-user-group background jobs from RabbitMQ to Arbiter - #5490
Open
blackheaven wants to merge 2 commits into
Open
[WPB-28375] Move sync-user-group background jobs from RabbitMQ to Arbiter#5490blackheaven wants to merge 2 commits into
blackheaven wants to merge 2 commits into
Conversation
zebot
added
the
ok-to-test
blackheaven
force-pushed
the
gdifolco/WPB-28375-background-jobs-arbiter
branch
from
August 28, 2026 08:57
6ac3844 to
159cd2b
Compare
…iter Add a 'user-groups' queue to the Arbiter job registry with a UserGroupsJobPayload sum (sync_user_group / sync_user_group_and_channel). brig now publishes sync jobs through the shared Postgres pool (per-group serialization, 3 attempts) and runs the registry migrations at startup. background-worker consumes the queue in a third Arbiter worker pool, reusing the existing dispatch pipeline; chained syncs are published back onto Arbiter and failures are retried via throwRetryable. The RabbitMQ jobs path is removed: jobs consumer and registry modules, RabbitMQ publisher interpreter, the BackgroundJob envelope and its dead serialization, the amqpJobsPublisherChannel in brig and background-worker, the backgroundJobs config block, and the corresponding charts, docs, and queue assertions in Test.Events. Backend-notification push and dead-user-notification watching remain on RabbitMQ.
blackheaven
force-pushed
the
gdifolco/WPB-28375-background-jobs-arbiter
branch
from
August 28, 2026 09:20
159cd2b to
377e193
Compare
akshaymankar
requested changes
Aug 31, 2026
| @@ -0,0 +1,3 @@ | |||
| Move the sync-user-group family of background jobs from the RabbitMQ `background-jobs` queue to the Arbiter Postgres job queue (new `user-groups` queue/table). brig now enqueues sync jobs via the shared Postgres pool with per-user-group serialization and 3 attempts; background-worker consumes them in a third Arbiter worker pool. The RabbitMQ jobs consumer, publisher channel, and `backgroundJobs` configuration are removed; backend-notification push and dead-user-notification watching remain on RabbitMQ. | |||
|
|
|||
| Rollout note for operators: deploy only after the RabbitMQ `background-jobs` queue has drained to zero. Jobs still in that queue when the last old background-worker pod terminates are dropped; sync jobs are idempotent reconciliations re-triggered by the next SCIM write, so bounded loss is acceptable. | |||
Member
There was a problem hiding this comment.
We need better instructions here. How can operators achieve this? When they do helm upgrade, everything gets upgraded.
Perhaps we can leave the part of code that processes the RabbitMQ queues and in the next release remove that code too, while marking this a release which should not be skipped for a safe transition.
Contributor
Author
There was a problem hiding this comment.
Done in 853d14f: the RabbitMQ background-jobs consumer is restored in this release (publisher stays on Arbiter, so the queue only drains), and the changelog now gives concrete operator steps: upgrade normally, wait until background-jobs is empty (rabbitmqctl list_queues name messages or the management API messages field), then upgrade to the next release which removes the consumer. This release is marked as must-not-be-skipped.
…al release Review feedback on the Arbiter move: a helm upgrade replaces brig and background-worker atomically, so operators cannot drain the RabbitMQ background-jobs queue before upgrading. Restore the pre-PR consumer (wire format, RabbitMQ publisher interpreter, background-worker consumer path, backgroundJobs config plumbing, charts/docs) so leftover jobs are drained instead of dropped; brig keeps publishing to Arbiter only. The consumer is removed in the next release; this release must not be skipped. Changelog spells out the drain-and-verify rollout steps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://wearezeta.atlassian.net/browse/WPB-28375
Checklist
changelog.d