Microservices communication patterns on a real shop: retries, Kafka, the outbox and a saga

Diagrams of microservice communication look clean: an arrow for a REST call, an arrow into a broker, done. The trouble lives in the cases the arrows skip: the answer that comes back too late, the retry that repeats a side effect, the process that dies between two writes.

This post uses a small shop where each of those cases can be triggered on purpose and then fixed. Every output below comes from a real run. The whole project is on GitHub, so you can repeat any of it: huseyinbabal/microservices-communication-patterns.

The shop

Versions: Spring Boot 4.1.1 on Java 25, Apache Kafka 4.3.2 in KRaft mode, Debezium Connect 3.7.0.Final, PostgreSQL 17 with wal_level=logical, Resilience4j for retry and circuit breaker. All four services share one Postgres container to keep the lab small, but each one reads and writes only its own schema.

The order service has one switch, SHOP_MODE, with three values (sync, dualwrite, outbox), plus flags for resilience, the idempotency key and a deliberate crash. Two services can be made slow: PAYMENT_DELAY_MS and EMAIL_DELAY_MS.

git clone https://github.com/huseyinbabal/microservices-communication-patterns
cd microservices-communication-patterns/lab
docker compose up -d --build
./connector.sh
export PGHOST=localhost PGPORT=15432 PGUSER=postgres PGPASSWORD=postgres PGDATABASE=shop

Every scenario starts from ./reset.sh and a restart of the four services with the right variables, for example:

SHOP_MODE=sync EMAIL_DELAY_MS=30000 docker compose up -d --force-recreate order payment inventory email

Results at a glance

1. Synchronous calls: the slowest service sets the pace

The first version of checkout is the one most code starts with: four calls in a row.

long place(PlaceOrder o) {
    long id = orders.insert(o, "PENDING");
    clients.charge(id, o.amountCents());
    clients.reserve(id, o.item(), o.qty());
    clients.sendEmail(id);
    orders.setStatus(id, "CONFIRMED");
    return id;
}order-service/.../SyncCheckout.java

With the email service set to answer after 30 seconds, the whole order takes 30 seconds, even though payment and inventory finished in milliseconds:

curl -s localhost:18080/orders \
  -H 'Content-Type: application/json' \
  -w '\ntime: %{time_total}s\n' \
  -d '{
    "item": "keyboard",
    "qty": 1,
    "amountCents": 4999
  }'

{"status":"CONFIRMED","orderId":63}
time: 30.273103soutput

Then docker compose stop email and the same request again. The customer gets a 500 in 34 ms. Payment did its part before email failed, and nothing undoes it:

psql -c "
SELECT o.id, o.status, c.status AS payment
FROM orders.orders o
JOIN payments.charges c ON c.order_id = o.id
ORDER BY o.id;"

 id |  status   | payment
----+-----------+---------
 63 | CONFIRMED | CHARGED
 64 | PENDING   | CHARGEDoutput

Two separate problems show up here. Latency adds up across the chain, and a failure late in the chain leaves the earlier steps half done.

2. Timeout and retry, and the triple charge

The usual first fix is a read timeout and a retry. In this lab both come from SHOP_RESILIENCE=true:

private final Retry retry = Retry.of("payment", RetryConfig.custom()
        .maxAttempts(3)
        .waitDuration(Duration.ofMillis(500))
        .build());

private RestClient client(String url) {
    var factory = new JdkClientHttpRequestFactory(HttpClient.newHttpClient());
    if (resilience) factory.setReadTimeout(Duration.ofSeconds(2));
    return RestClient.builder().baseUrl(url).requestFactory(factory).build();
}order-service/.../Clients.java

Now make payment slow in the worst possible way: it writes the charge first and answers after 3 seconds (PAYMENT_DELAY_MS=3000). The order service gives up after 2 seconds and tries again, three times in total:

{"timestamp":"2026-10-21T19:53:12.956Z","status":500,"error":"Internal Server Error","path":"/orders"}
time: 7.079114soutput

The 7.08 seconds are exactly the retry policy: three attempts of 2 s each, plus two waits of 0.5 s. The customer still gets an error. The payment log tells the rest:

docker compose logs --no-log-prefix payment \
  | grep -o 'charged order.*'
psql -c "
SELECT order_id, count(*) AS charges, sum(amount_cents) AS cents
FROM payments.charges
GROUP BY order_id;"

charged order 65 (4999 cents) -> charge 72  key=null
charged order 65 (4999 cents) -> charge 73  key=null
charged order 65 (4999 cents) -> charge 74  key=null

 order_id | charges | cents
----------+---------+-------
       65 |       3 | 14997output

A timeout tells the caller nothing about whether the work happened. Retrying an operation that is not safe to repeat turns one slow answer into three charges.

3. The idempotency key

The fix sits on both sides. The caller sends a key that stays the same across retries, and the payment service stores it under a unique constraint:

payment.post().uri("/payments")
       .headers(h -> { if (idempotencyKey) h.set("Idempotency-Key", "order-" + orderId); })order-service/.../Clients.java

Optional<Long> created = db.sql("""
        INSERT INTO payments.charges(order_id, amount_cents, idempotency_key, status) VALUES (?, ?, ?, 'CHARGED')
        ON CONFLICT (idempotency_key) DO NOTHING RETURNING id""")
    .params(orderId, amountCents, idempotencyKey).query(Long.class).optional();
return created.orElseGet(() -> db.sql("SELECT id FROM payments.charges WHERE idempotency_key = ?")
        .param(idempotencyKey).query(Long.class).single());payment-service/.../Charges.java

Same scenario with SHOP_IDEMPOTENCY_KEY=true:

charged order 66 (4999 cents) -> charge 75  key=order-66
charged order 66 (4999 cents) -> charge 75  key=order-66
charged order 66 (4999 cents) -> charge 75  key=order-66

 order_id | charges | cents
----------+---------+-------
       66 |       1 |  4999output

Three requests, one charge id. Note what the key does not fix: payment still answers after 3 seconds, so all three attempts still time out and the order still fails. The key makes the retry safe. It does not make the slow service fast.

ON CONFLICT DO NOTHING on a unique column is what makes this safe under concurrency too. A “check if the key exists, then insert” in two statements would let two parallel retries both pass the check.

4. Circuit breaker

When a dependency is down for good, a timeout still costs every request the full timeout. A circuit breaker counts failures and, past a threshold, stops calling for a while:

private final CircuitBreaker emailBreaker = CircuitBreaker.of("email", CircuitBreakerConfig.custom()
        .slidingWindowSize(4)
        .failureRateThreshold(50)
        .waitDurationInOpenState(Duration.ofSeconds(20))
        .build());order-service/.../Clients.java

With email answering after 30 seconds, six orders in a row:

for i in 1 2 3 4 5 6; do
  curl -s -o /dev/null localhost:18080/orders \
    -H 'Content-Type: application/json' \
    -w "order $i: %{time_total}s\n" \
    -d '{"item": "keyboard", "qty": 1, "amountCents": 4999}'
done
docker compose logs --no-log-prefix order \
  | grep -o 'email for order.*'

order 1: 2.222774s
order 2: 2.016465s
order 3: 2.015658s
order 4: 2.025158s
order 5: 0.012299s
order 6: 0.009348s

email for order 67 skipped: ResourceAccessException (breaker CLOSED)
email for order 68 skipped: ResourceAccessException (breaker CLOSED)
email for order 69 skipped: ResourceAccessException (breaker CLOSED)
email for order 70 skipped: ResourceAccessException (breaker OPEN)
email for order 71 skipped: CallNotPermittedException (breaker OPEN)
email for order 72 skipped: CallNotPermittedException (breaker OPEN)output

The window holds 4 calls, so the breaker decides after the fourth failure. That is why order 70 still waited 2 seconds but was logged with the breaker already OPEN. From order 71 on, Resilience4j rejects the call without touching the network (CallNotPermittedException). After 20 seconds in the open state, the breaker moves to half-open and lets a limited number of test calls through; they decide whether it closes again or stays open.

The log also shows what this pattern costs: six orders went through and none of them got a confirmation email. The breaker protects the caller; it does not deliver the work.

5. Asynchronous messaging with Kafka

In outbox mode the order service makes no HTTP calls at all. It records the order and an OrderPlaced event (more on how in the next section), answers right away, and the other services react to events:

order ──OrderPlaced──> payment ──PaymentCompleted──> inventory ──InventoryReserved──> email
                                                                       └─────────────> order (CONFIRMED)

With the email service stopped:

{"orderId":73,"status":"PENDING"}
time: 0.040080s

$ sleep 5; curl -s localhost:18080/orders/73
{"orderId":73,"status":"CONFIRMED"}output

The order is confirmed while email is still down, because confirmation depends on inventory now, not on email. After docker compose start email, the email consumer reads the event it missed:

email sent for order 73output

The response says PENDING, and that is the honest answer at that moment. With events, the API has to expose intermediate states, and clients have to poll or be notified. That is the price of not waiting.

6. The dual write

The obvious way to publish that event is to save the order and then send to Kafka. The lab’s version even waits for the database commit before sending, which is better than most:

@Transactional
long place(PlaceOrder o) {
    long id = orders.insert(o, "PENDING");
    TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
        @Override public void afterCommit() {
            if (crashBeforePublish) Runtime.getRuntime().halt(1);   // row committed, event never sent
            var event = new ProducerRecord<>("order.events", String.valueOf(id), Events.orderPlaced(id, o));
            event.headers().add("eventType", "OrderPlaced".getBytes());
            kafka.send(event);
        }
    });
    return id;
}order-service/.../DualWriteCheckout.java

CRASH_BEFORE_PUBLISH=true kills the JVM between the commit and the send, which is what a deploy, an OOM kill or a node failure would do:

http 000
order: Exited (1) 4 seconds ago

 id |   item   | status
----+----------+---------
 74 | keyboard | PENDINGoutput

And Kafka has nothing for order 74:

docker compose exec kafka \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server kafka:9092 --topic order.events \
  --from-beginning --timeout-ms 5000 \
  --property print.key=true 2>/dev/null \
  | grep -c '^74\b'

0output

Swapping the order (send first, then commit) only moves the problem: then Kafka can have an event for an order that was rolled back. Two systems without a shared transaction cannot be updated atomically by the application.

7. The outbox and change data capture

The outbox pattern stops writing to two systems. The event becomes a row in the same database, in the same transaction as the order:

@Transactional
long place(PlaceOrder o) {
    long id = orders.insert(o, "PENDING");
    db.sql("INSERT INTO orders.outbox(id, aggregatetype, aggregateid, type, payload) VALUES (?, 'order', ?, 'OrderPlaced', ?::jsonb)")
      .params(UUID.randomUUID(), String.valueOf(id), Events.orderPlaced(id, o))
      .update();
    return id;
}order-service/.../OutboxCheckout.java

There is no Kafka call in this code. Something else has to move the row to Kafka, and here that is Debezium, reading Postgres’s write-ahead log. The chain has five links:

So the row (aggregatetype=order, aggregateid=75, type=OrderPlaced, payload={...}) arrives in topic order.events, with key 75, header eventType: OrderPlaced, and the JSON payload as the value. Using the order id as the key matters: all events of one order land in the same partition, so consumers see them in order.

To show the decoupling, stop Debezium and place an order:

{"status":"PENDING","orderId":75}
time: 0.062091s

 id | status  | outbox_event
----+---------+--------------
 75 | PENDING | OrderPlacedoutput

The order is accepted and its event is safe in the outbox table. docker compose start connect, and Debezium resumes from the slot where it stopped:

{"status":"CONFIRMED","orderId":75}output

Two things to know before running this in production. A replication slot that nobody reads keeps WAL on disk forever, so a stopped connector slowly fills the database disk; monitor slot lag. And Debezium delivers at least once: after a crash it can publish the same row again. Consumers have to handle duplicates, which is why the payment consumer in this project charges with the key order-<id> from section 3.

8. A saga with a compensating step

Events solve waiting, but one business action still spans three databases, and there is no transaction across them. A saga splits the action into local steps, each in its own transaction, and gives every step an undo:

In a choreography, no service is in charge. Each one listens for the event before it and publishes its own result through its own outbox:

@Transactional
@KafkaListener(topics = {"order.events", "inventory.events"})
void on(ConsumerRecord<String, String> event) {
    String type = new String(event.headers().lastHeader("eventType").value());
    ...
    switch (type) {
        case "OrderPlaced" -> {
            charges.charge(orderId, e.get("amountCents").asLong(), "order-" + orderId);
            publish(orderId, "PaymentCompleted", event.value());
        }
        case "InventoryRejected" -> {
            charges.refund(orderId);                      // the compensating step
            publish(orderId, "PaymentRefunded", event.value());
        }
    }
}payment-service/.../PaymentEvents.java

The lab keeps monitors at zero stock. Ordering one:

{"orderId":76,"status":"PENDING"}

$ sleep 8; curl -s localhost:18080/orders/76
{"orderId":76,"status":"CANCELLED"}

 order_id | amount_cents |  status
----------+--------------+----------
       76 |        19999 | REFUNDEDoutput

And the trail, one outbox per service:

psql -c "
SELECT 'order' AS service, type FROM orders.outbox WHERE aggregateid = '76'
UNION ALL
SELECT 'payment', type FROM payments.outbox WHERE aggregateid = '76'
UNION ALL
SELECT 'inventory', type FROM inventory.outbox WHERE aggregateid = '76';"

  service  |       type
-----------+-------------------
 order     | OrderPlaced
 payment   | PaymentCompleted
 payment   | PaymentRefunded
 inventory | InventoryRejectedoutput

The customer was charged and then refunded. A saga does not hide intermediate states the way a database transaction does; for a few seconds the money really was taken. Whether that is acceptable is a business decision, and it shapes how the steps are ordered (reserving stock before charging would avoid this particular refund).

Which one, when

The saga here is a choreography, and its weak spot shows in the UNION ALL above: to see the whole flow you have to collect it from three services. When flows get longer or have branches, an orchestrator (one service that tells each step what to do and keeps the state) is the other way to run a saga, at the cost of a component that knows about everyone.

For how Kafka stores these events (partitions, segments, consumer groups), see Kafka: batches, segments and consumers.

Clean up

docker compose down -v

Sources

• Project on GitHub: huseyinbabal/microservices-communication-patterns
• Debezium: Outbox Event Router
• Debezium: PostgreSQL connector (replication slots, pgoutput)
• PostgreSQL: logical decoding and replication slots
• Resilience4j: CircuitBreaker and Retry
• Apache Kafka documentation