Ledger Core

A hands-on distributed-systems project that reproduces real money-movement failures and fixes them — the way a payment switch (GPay → NPCI → banks) actually has to. It starts as a single Spring Boot monolith and evolves, one reproduced failure at a time, into a microservices saga with after-the-fact reconciliation across independent banks that cannot be polled mid-transfer.

---

Why this project is different

Most wallet/ledger demos show the happy path. This one is built around the unhappy paths — the retries, crashes, partial failures, and "did the other bank actually credit?" ambiguity that make real payments hard. Every milestone is a small, self-contained experiment that first reproduces a failure as a reproducible, timestamped artifact, then a code change that flips it from FAIL to PASS. Nothing is asserted without a chaos test to back it.

The result is a system that mirrors the real interbank model:

flowchart LR
    %% Client
    C["Client App<br/>(GPay, etc.)"]

    %% Payment Switch
    AG["API Gateway<br/>(verify + route)"]
    TS["Transfer Service<br/>(the 'switch' / NPCI)<br/>Saga + Reconciliation"]

    %% Banks
    subgraph Banks["Independent Banks (black boxes)"]
        direction LR

        BA["Bank A<br/>account-service<br/><br/>Own DB + Own Ledger"]
        BB["Bank B<br/>account-service<br/><br/>Own DB + Own Ledger"]
    end

    %% Flow
    C -- JWT --> AG
    AG --> TS

    TS -- "reserve / confirm" --> BA
    TS -- "credit" --> BB

    %% Note
    N["No shared commit oracle.<br/>If a bank goes offline mid-transfer,<br/>the switch reconciles against each bank's<br/>ledger after the fact to settle the money."]

    TS -.-> N

    classDef note fill:#fff8dc,stroke:#b8860b,color:#333;
    class N note;

Loading

transfer-service is the switch: it runs the source legs (reserve → confirm-debit) on the source bank and the credit on the target bank, with no distributed transaction across them. Correctness comes from per-step idempotency, a persisted saga with compensation + recovery, and — the finale — reconciliation that asks each bank's ledger for a settlement window and rolls stranded transfers forward or back.

---

The journey — seven experiments

Each experiment lives in experiments/ with its own README, runnable run.ps1, and timestamped artifacts (a git SHA-stamped receipt, not a story). PASS = the system stayed consistent; a reproduced FAIL is itself the deliverable that motivates the next fix.

#	Experiment	Reproduces / proves	Verdict
001	Retry without idempotency	A retried transfer moves money N times — the API is not idempotent	FAIL (the bug)
002	Retry with idempotency	An Idempotency-Key makes the same transfer apply exactly once	PASS
003	Distributed transaction failure	Split into services, a mid-transfer outage leaves balances moved but no ledger record	FAIL (the gap)
004	Saga rollback via recovery	A persisted saga + recovery sweep compensates the stranded transfer — nothing stuck	PASS
005	Chaos + concurrency	Random crashes under concurrent load; timing-independent invariants still hold	PASS
006	The "deemed" limbo (two banks)	With no shared commit oracle, a failed external credit strands money the sweep cannot settle	FAIL (the seam)
007	Reconciliation & recovery	The switch asks each bank's ledger after the fact and rolls the transfer forward or back	PASS

The arc: idempotency (001→002) → saga + compensation/recovery (003→004) → reliability under chaos (005) → the limits of polling an external bank (006) → reconciliation closes the seam (007).

---

Concepts demonstrated

• Idempotency keys and per-step idempotency (processed_operations) so retries and recovery replays never double-apply.
• Saga orchestration with an explicit commit boundary — compensation before it (release), roll-forward after it (confirm / credit).
• Persisted saga state + recovery — a startup scan and a periodic sweep drive stranded transfers to a terminal state; optimistic @version stops a sweep racing a live request.
• The two-phase-ish money move: reserve → confirm-debit → credit, so money is in-flight, never duplicated.
• Reconciliation — after-the-fact settlement against each bank's ledger, with the load-bearing ideas made explicit: a grace/settlement cutoff (so the ledger's answer is final and you adjudicate rather than presume failure), the ledger as the source of truth ("beneficiary credit is truth"), and a watermark that advances over the contiguous reconciled prefix.
• Chaos & concurrency testing with k6, verified by timing-independent invariants (conservation, no leftover reserved, all transfers terminal, ledger agrees).

See backend/README.md for the full architecture and backend/ledger-api/README.md for the monolith it started from.

---

Architecture at a glance

Spring Boot 4.1 on Java 21, a Maven multi-module reactor, PostgreSQL per service, JWT auth, synchronous HTTP between services.

Module	Port	Owns	Responsibility
api-gateway	8080	—	Routing, JWT verification, X-User-Name propagation
account-service	8081	account_db	Identity, auth/tokens, balances, idempotent single-account primitives + its own ledger
ledger-service	8082	ledger_db	Double-entry record-keeping (standard stack)
transfer-service	8083	transfer_db	Saga orchestration + reconciliation — persisted state, compensation, recovery, watermark
common	—	—	Shared entities, error envelope, JWT utils, inter-service DTOs

In the two-bank experiments (006/007) the same account-service binary runs twice as banks A and B, each on its own database — identical code, no shared ledger, no live commit oracle.

---

Quick start

Option A — Docker (the whole stack, one command)

docker compose up --build

Brings up PostgreSQL (all databases auto-created) + the four services. Then, through the gateway (:8080) and account-service (:8081):

# register two users + log in
curl -s -XPOST localhost:8080/api/v1/auth/register -H 'Content-Type: application/json' \
  -d '{"username":"alice","password":"s3cret","email":"alice@x.io"}'
curl -s -XPOST localhost:8080/api/v1/auth/register -H 'Content-Type: application/json' \
  -d '{"username":"bob","password":"s3cret","email":"bob@x.io"}'
TOKEN=$(curl -s -XPOST localhost:8080/api/v1/auth/login -H 'Content-Type: application/json' \
  -d '{"username":"alice","password":"s3cret"}' | jq -r .token)

# fund alice (dev endpoint — no production deposit endpoint exists), then transfer
curl -s -XPOST localhost:8081/api/v1/debug/accounts/alice/fund \
  -H 'Content-Type: application/json' -d '{"amount":1000}'
curl -s -XPOST localhost:8080/api/v1/transactions/transfer -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"targetAccountName":"bob","amount":"25.00"}'
curl -s localhost:8080/api/v1/transactions -H "Authorization: Bearer $TOKEN"

Option B — run locally (JDK 21 + Maven + PostgreSQL)

See backend/README.md §6 — createdb the three databases, then mvn -pl <service> spring-boot:run per service.

Run an experiment (the interesting part)

Experiments need the dev profile and, for 006/007, a two-bank topology with its own scripts:

cd experiments/007-reconciliation-recovery
./start-topology.ps1      # account-service ×2 (banks A/B) + transfer (recon on) + gateway
./status-topology.ps1     # wait for Running + Ready
./run.ps1                 # reproduce + settle; writes artifacts/verification.txt (expect PASS)
./stop-topology.ps1

Prereqs for experiments: k6 on PATH; some earlier verifiers use Node.js 18+.

---

Repository layout

ledger-core/
├── backend/               # Maven reactor: common, account-service, ledger-service,
│   │                      #   transfer-service, api-gateway  (+ ledger-api: the original monolith)
│   ├── README.md          # architecture, the saga, recovery, reconciliation — the deep dive
│   └── Dockerfile         # shared multi-stage build (per-module via --build-arg MODULE)
├── experiments/           # 001–007: each a reproduced failure → fix, with runnable scripts + artifacts
├── tools/k6/              # shared k6 scenarios, utils, and invariant verifiers
├── infra/postgres/        # first-boot DB creation for docker compose
├── frontend/              # (optional) companion UI
└── docker-compose.yml     # the standard stack, one command

---

Tech stack

Java 21 · Spring Boot 4.1 (Web MVC, Data JPA, Validation, Security) · Spring Cloud Gateway · PostgreSQL + Flyway (schema per service) · JWT (jjwt, HS256) · Lombok · k6 for chaos / load · JUnit 5 + Mockito + AssertJ for unit tests, integration tests against a real PostgreSQL.

---

A note on scope

This is a learning project — deliberately built to feel the failure modes distributed payments face and to earn each fix with a reproduced experiment, rather than to be a production payment switch. The honest limits are called out too: what's still deliberately deferred (an Outbox so a ledger write can't be silently lost; Kafka/event-driven delivery) is documented in backend/README.md.