A production-style asynchronous task processing system built with Java 17, Spring Boot, RabbitMQ, MySQL, Docker, and Maven.
The system accepts background jobs through REST APIs, persists task state in MySQL, publishes jobs to RabbitMQ, and processes them asynchronously using scalable worker services.
Synchronous APIs are not ideal for slow or failure-prone operations such as sending emails, generating reports, or delivering notifications.
This project solves that problem using a producer-worker architecture:
- The Producer Service accepts and validates task requests.
- MySQL stores task state, retry information, and audit logs.
- RabbitMQ acts as the message broker and buffers background work.
- Worker Services consume and process tasks asynchronously.
- Docker Compose runs the complete distributed system locally.
The project also demonstrates production-oriented backend patterns including:
- Manual message acknowledgements
- Retry mechanisms with exponential backoff
- Dead-letter queues
- Horizontal worker scaling
- Idempotent task submission
- API rate limiting
- Priority-based message processing
- Health checks
- Audit logging
┌─────────────────┐
│ Client │
│ REST / Postman │
└────────┬────────┘
│
▼
┌──────────────────────┐
│ Producer Service │
│ Spring Boot │
└──────────┬───────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
┌───────────────┐ ┌─────────────────┐
│ MySQL │ │ RabbitMQ │
│ Task State + │ │ Exchange/Queues │
│ Audit Logs │ └────────┬────────┘
└───────────────┘ │
▼
┌─────────────────────┐
│ Worker Service │
│ │
│ ┌─────┐ ┌─────┐ │
│ │ W1 │ │ W2 │ ... │
│ └─────┘ └─────┘ │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Retry / DLQ Flow │
└─────────────────────┘
| Technology | Purpose |
|---|---|
| Java 17 | Backend development |
| Spring Boot | REST APIs and application framework |
| Spring Data JPA | Database persistence |
| Hibernate | ORM |
| RabbitMQ | Message broker |
| MySQL | Task state and audit persistence |
| Docker | Containerization |
| Docker Compose | Local distributed environment |
| Maven | Build and dependency management |
- REST-based task submission
- EMAIL, REPORT, and NOTIFICATION task types
- Asynchronous task processing
- RabbitMQ-based message routing
- MySQL task persistence
- Multiple worker instances
- Manual message acknowledgements
- Task status tracking
- Audit logging
- Retry mechanism
- Exponential backoff
- Dead-letter queue handling
- Failure tracking
- Manual ACK/NACK flow
- Persistent task state
- Idempotency keys
- Fixed-window API rate limiting
- RabbitMQ message priorities
- Health check endpoints
- Dockerized deployment
- Horizontal worker scaling
The project was implemented incrementally:
- Phase 1 — Producer Service
- Phase 2 — Worker Service
- Phase 3 — Multi-worker Load Balancing
- Phase 4 — Retry & Dead-Letter Queue
- Phase 5 — Dockerized Deployment
- Phase 6 — Idempotency, Rate Limiting & Priority Queues
The Producer Service provides the REST entry point for task submission.
POST /api/tasks- Request DTO validation
- EMAIL, REPORT, and NOTIFICATION jobs
- UUID-based task creation
- MySQL persistence
QUEUEDtask status- RabbitMQ publishing
- Direct exchange routing
Client
│
▼
REST Controller
│
▼
Task Service
│
├──────────────► MySQL
│ │
│ └── Task = QUEUED
│
└──────────────► RabbitMQ
│
▼
Queue
Instead of keeping the HTTP request open while the task executes, the API quickly accepts the job and returns a response.
This provides:
- Lower API latency
- Better scalability
- Decoupling between API and workers
- Persistent task tracking
The controller remains thin while the service layer contains the business logic. RabbitMQ provides asynchronous decoupling between request handling and background processing.
The Worker Service processes tasks asynchronously from RabbitMQ.
@RabbitListenerconsumers- EMAIL processor
- REPORT processor
- NOTIFICATION processor
- Manual acknowledgements
- Success/failure handling
- Task status updates
- Audit logs
RabbitMQ
│
▼
Worker
│
▼
PROCESSING
│
▼
Task Processor
/ \
/ \
Success Failure
│ │
▼ ▼
ACK Retry/DLQ
Slow or failure-prone work is removed from the synchronous HTTP request path.
Multiple workers can process tasks concurrently, improving throughput and resilience.
Manual acknowledgements make message processing explicit. A successful task is ACKed, while failed tasks can enter the retry or dead-letter flow.
This phase demonstrates horizontal worker scaling.
RabbitMQ distributes messages from the same queue across multiple consumers.
docker compose up --build -ddocker compose up --scale task-worker=3 -dchmod +x scripts/phase3-load-balancing-demo.sh
./scripts/phase3-load-balancing-demo.shdocker compose logs -f task-workerEach worker logs its own workerId.
You should see tasks being processed by different worker containers.
With:
prefetch-count: 1RabbitMQ limits each worker to one unacknowledged message at a time, helping achieve fairer dispatch.
Failed tasks are not immediately requeued.
Instead, retryable failures are published to a dedicated retry exchange and retry queue.
The retry queue uses message TTL + dead-lettering to delay the next processing attempt.
┌──────────────┐
│ Worker │
└──────┬───────┘
│
Failure
│
▼
┌───────────────┐
│ Retry Exchange│
└───────┬───────┘
│
▼
┌───────────────┐
│ Retry Queue │
│ TTL │
└───────┬───────┘
│
Expired
│
▼
┌───────────────┐
│ Main Exchange │
└───────┬───────┘
│
▼
Main Queue
If the maximum retry count is exhausted:
Worker
│
▼
Failure
│
▼
Retries Remaining?
│
┌┴─────────┐
│ │
Yes No
│ │
▼ ▼
Retry DLQ
│
▼
DEAD_LETTER
Immediate requeue can create a hot retry loop where the same failing task repeatedly consumes worker resources.
Delayed retries:
- Reduce pressure on workers
- Give dependencies time to recover
- Avoid tight retry loops
- Preserve failure history
- Provide predictable retry behavior
Submit tasks using the built-in failure scenarios and monitor:
docker compose logs -f task-workerLook for events such as:
FAILED
RETRY_SCHEDULED
DEAD_LETTERED
RabbitMQ Management UI can also be used to inspect retry queues and DLQ depth.
The complete system runs through Docker Compose.
Docker Compose
│
├── MySQL
├── RabbitMQ
├── Producer Service
└── Worker Service(s)
chmod +x scripts/phase5-up.sh
./scripts/phase5-up.shBy default, the script starts three worker replicas.
To change the number:
WORKER_REPLICAS=5 ./scripts/phase5-up.shdocker compose psdocker compose logs -f task-producerdocker compose logs -f task-workerdocker compose down- Starts MySQL
- Initializes the database schema
- Starts RabbitMQ with Management UI
- Builds producer and worker images
- Waits for healthy dependencies
- Runs multiple worker containers
- Provides a reproducible local environment
The final phase adds production-style controls around task submission.
Each task can contain an idempotencyKey.
If the same key is submitted again, the existing task is returned instead of creating another task.
{
"type": "EMAIL",
"payload": "{\"to\":\"user@example.com\",\"subject\":\"Welcome\"}",
"idempotencyKey": "welcome-email-user-123"
}Clients may retry requests because of:
- Network failures
- Request timeouts
- Frontend retries
- Mobile connectivity issues
Without idempotency, the same operation could be executed multiple times.
For example:
Client
│
├── Request ───────► Producer
│
│ Network timeout
│
└── Retry ─────────► Producer
The idempotency key allows the backend to recognize both requests as the same logical operation.
The Producer applies a fixed-window rate limit based on client IP.
When the limit is exceeded, the API returns:
429 Too Many Requests{
"timestamp": "2026-08-22T12:00:00",
"status": 429,
"error": "Rate Limit Exceeded",
"message": "Too many requests. Retry after 60 seconds.",
"retryAfterSeconds": 60
}Rate limiting prevents a single client from generating excessive traffic and protects the producer from accidental bursts.
RabbitMQ message priorities allow urgent tasks to be processed before lower-priority tasks that are waiting in the queue.
The producer sets the message priority using the task's priority field.
{
"type": "REPORT",
"priority": 8,
"maxRetries": 3
}A higher priority value means the task receives higher processing preference while waiting in the queue.
curl -X POST http://localhost:8080/api/tasks \
-H "Content-Type: application/json" \
-d '{
"type": "EMAIL",
"payload": "{\"to\":\"user@example.com\",\"subject\":\"Welcome\"}",
"priority": 8,
"maxRetries": 3,
"idempotencyKey": "welcome-email-user-123"
}'curl http://localhost:8080/api/tasks/{taskId}Import the following files into Postman:
postman/
├── DistributedTaskQueueSystem.postman_collection.json
└── DistributedTaskQueueSystem.postman_environment.json
The collection includes:
- EMAIL task submission
- REPORT task submission
- NOTIFICATION task submission
- Idempotency duplicate request
- Worker health check
- Task log inspection
- Rate-limit burst demo
A typical end-to-end test looks like:
POST /api/tasks
│
▼
Task persisted in MySQL
│
▼
Published to RabbitMQ
│
▼
Worker consumes message
│
▼
Task → PROCESSING
│
▼
Processor executes task
│
┌──┴──┐
│ │
Success Failure
│ │
▼ ▼
ACK Retry
│
▼
Eventually
┌───┴───┐
│ │
Success DLQ
This project demonstrates practical backend engineering concepts including:
- REST API design
- DTO validation
- Layered architecture
- Spring Boot
- Spring Data JPA
- Hibernate
- MySQL persistence
- RabbitMQ exchanges and queues
- Routing keys
- Manual message acknowledgements
- Consumer concurrency
- Fair dispatch
- Retry with exponential backoff
- Dead-letter queues
- Idempotency
- Rate limiting
- Priority queues
- Audit logging
- Health checks
- Docker
- Docker Compose
- Horizontal scaling
- Asynchronous processing
- Event-driven architecture
| Capability | Implementation |
|---|---|
| API Layer | Spring Boot REST APIs |
| Persistence | MySQL + JPA/Hibernate |
| Messaging | RabbitMQ |
| Processing | Asynchronous workers |
| Scaling | Multiple worker containers |
| Reliability | Manual ACK + retry |
| Failure Handling | DLQ |
| Retry Strategy | TTL + dead-lettering |
| Duplicate Protection | Idempotency keys |
| API Protection | Rate limiting |
| Task Prioritization | RabbitMQ priority queues |
| Deployment | Docker Compose |
| Observability | Task status + audit logs |
GitHub:
https://github.com/anik-ug/distributed-task-queue-system