Skip to content

fix: preserve email retries until attempts exhaust - #464

Open
vladbisceanu wants to merge 1 commit into
useplunk:nextfrom
vladbisceanu:upstream/136-email-retries
Open

fix: preserve email retries until attempts exhaust#464
vladbisceanu wants to merge 1 commit into
useplunk:nextfrom
vladbisceanu:upstream/136-email-retries

Conversation

@vladbisceanu

Copy link
Copy Markdown
Contributor

What changed

The email worker currently writes FAILED after the first exception, so BullMQ retries find a non-PENDING row and exit without reaching SES again.

This patch:

  • keeps explicit retryable SES failures PENDING until the configured attempts are exhausted
  • retries only SES throttling/service responses; ambiguous transport failures stop without risking a duplicate
  • checkpoints successful SES acceptance on the BullMQ job before database finalization, so a Postgres outage can recover without another send
  • makes phishing failures terminal
  • preserves accepted messages when later billing, event, or campaign bookkeeping fails

Verification

  • real BullMQ worker coverage for retry success, terminal exhaustion, database recovery, dual Redis/Postgres failure, ambiguous SES transport loss, and post-send failure (17/17 focused tests)
  • focused ESLint on the three changed files
  • yarn turbo build --filter=api (5/5 tasks)
  • two local review passes; all P1/P2 findings resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are correctness/reliability issues in the updated worker path (phishing error persistence, event sentAt accuracy, and potential dropped emails on worker crash between SENDING and SES submission) that should be addressed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the API email worker send path to make BullMQ retries meaningful and safe by preserving retryable failures as PENDING, checkpointing SES acceptance into job data for Postgres-outage recovery, and ensuring post-send failures don’t lead to duplicate sends.

Changes:

  • Add acceptedBySes to SendEmailJobData to persist SES acceptance in BullMQ job data for recovery.
  • Update the email worker to (a) retry only explicit SES throttle/service failures, (b) checkpoint SES acceptance before DB finalization, and (c) treat phishing failures as terminal.
  • Add integration-style worker tests that exercise retry success/exhaustion, checkpoint recovery, ambiguous transport failure behavior, and post-send failure behavior.
File summaries
File Description
packages/types/src/jobs/email.ts Extends job data type with optional SES acceptance checkpoint payload.
apps/api/src/jobs/email-processor.ts Implements retry/terminal logic, SES acceptance checkpointing, and recovery semantics in the worker.
apps/api/src/jobs/tests/email-processor.test.ts Adds queue/worker-level tests covering retries, recovery, and terminal failure behavior.
Review details

Suppressed comments (1)

apps/api/src/jobs/email-processor.ts:380

  • email.sent events are emitted with sentAt: new Date() which can differ from the SES acceptance timestamp, and in recovery cases (job retry finalizing a checkpoint) will be incorrect. Use the persisted SES acceptance time to keep downstream workflows/billing consistent.
          messageId: acceptedBySes.messageId,
          emailId: email.id,
          templateId: email.templateId,
          campaignId: email.campaignId,
          sourceType: email.sourceType,
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +157 to +161
if (email.status === EmailStatus.SENDING && !recoveredAcceptance) {
const message = 'Previous attempt ended without an SES acceptance checkpoint; not retried to avoid a duplicate';
await prisma.email.update({
where: {id: emailId},
data: {status: EmailStatus.FAILED, error: message},
},
});

throw new UnrecoverableError(`Project ${email.projectId} has been disabled due to a policy violation`);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants