fix: preserve email retries until attempts exhaust - #464
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are correctness/reliability issues in the updated worker path (phishing error persistence, event sentAt accuracy, and potential dropped emails on worker crash between SENDING and SES submission) that should be addressed before merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the API email worker send path to make BullMQ retries meaningful and safe by preserving retryable failures as PENDING, checkpointing SES acceptance into job data for Postgres-outage recovery, and ensuring post-send failures don’t lead to duplicate sends.
Changes:
- Add
acceptedBySestoSendEmailJobDatato persist SES acceptance in BullMQ job data for recovery. - Update the email worker to (a) retry only explicit SES throttle/service failures, (b) checkpoint SES acceptance before DB finalization, and (c) treat phishing failures as terminal.
- Add integration-style worker tests that exercise retry success/exhaustion, checkpoint recovery, ambiguous transport failure behavior, and post-send failure behavior.
File summaries
| File | Description |
|---|---|
| packages/types/src/jobs/email.ts | Extends job data type with optional SES acceptance checkpoint payload. |
| apps/api/src/jobs/email-processor.ts | Implements retry/terminal logic, SES acceptance checkpointing, and recovery semantics in the worker. |
| apps/api/src/jobs/tests/email-processor.test.ts | Adds queue/worker-level tests covering retries, recovery, and terminal failure behavior. |
Review details
Suppressed comments (1)
apps/api/src/jobs/email-processor.ts:380
email.sentevents are emitted withsentAt: new Date()which can differ from the SES acceptance timestamp, and in recovery cases (job retry finalizing a checkpoint) will be incorrect. Use the persisted SES acceptance time to keep downstream workflows/billing consistent.
messageId: acceptedBySes.messageId,
emailId: email.id,
templateId: email.templateId,
campaignId: email.campaignId,
sourceType: email.sourceType,
- Files reviewed: 3/3 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if (email.status === EmailStatus.SENDING && !recoveredAcceptance) { | ||
| const message = 'Previous attempt ended without an SES acceptance checkpoint; not retried to avoid a duplicate'; | ||
| await prisma.email.update({ | ||
| where: {id: emailId}, | ||
| data: {status: EmailStatus.FAILED, error: message}, |
| }, | ||
| }); | ||
|
|
||
| throw new UnrecoverableError(`Project ${email.projectId} has been disabled due to a policy violation`); |
What changed
The email worker currently writes
FAILEDafter the first exception, so BullMQ retries find a non-PENDINGrow and exit without reaching SES again.This patch:
PENDINGuntil the configured attempts are exhaustedVerification
yarn turbo build --filter=api(5/5 tasks)