Resilient Backend Service
A scalable backend service designed for production: clean APIs, resilient failure handling, and the operational tooling to run it reliably at scale.
- Role
- Backend engineering
- Timeline
- 2024
- Stack
- Java · Spring Boot · Docker · Kubernetes · PostgreSQL
The problem
The service sat in the middle of a call graph it did not control: upstream clients expected it to stay responsive, downstream dependencies were free to be slow or unavailable. Early versions treated every downstream call as if it would succeed, so a single slow dependency consumed the request threads and turned one degraded system into a full outage. The goal was a service that fails narrowly and predictably instead of all at once.
Constraints
- Downstream services had no availability guarantee and could not be changed.
- Horizontal scaling meant no instance could hold state that mattered.
- Clients retry, so every mutating endpoint had to tolerate duplicate delivery.
- Operability was a requirement, not a follow-up — a service you can't diagnose isn't finished.
Architecture
- 01
API layer
Versioned REST endpoints with validation at the boundary and a single consistent error contract, so clients can distinguish "your request was wrong" from "try again shortly" without parsing prose.
- 02
Resilience
Every outbound dependency sits behind a timeout, a bounded retry with backoff and jitter, and a circuit breaker with a defined fallback. Bulkheads keep one saturated dependency from exhausting the pool the whole service shares.
- 03
Data & idempotency
PostgreSQL with schema migrations kept in version control. Mutating endpoints accept an idempotency key so a client retry after a timeout resolves to the original result instead of duplicating work.
- 04
Runtime
Containerised and deployed to Kubernetes with liveness and readiness probes that mean different things — readiness drops the instance out of rotation while it recovers, liveness restarts it only when it's genuinely stuck. Structured logs carry a correlation ID end to end.
Decisions & tradeoffs
Circuit breakers with explicit fallbacks on every dependency.
why Failing fast with a defined degraded response keeps the service answering while a dependency recovers. Without it, latency propagates upward and the whole call graph goes down together.
cost Every dependency needs a considered answer to "what do we return when this is down?" — and some fallbacks are genuinely worse than an error.
Idempotency keys on mutating endpoints.
why In a distributed system a timeout tells you nothing about whether the work happened. Making retries safe is the only way clients can retry correctly.
cost Key storage, an expiry policy, and more complexity in the write path — paid on every request to protect against the uncommon one.
Stateless instances, state pushed to the database.
why Any instance can serve any request, so scaling is a replica count and losing a pod is uneventful.
cost More load on the datastore, and caching has to be handled deliberately rather than accidentally in process memory.
Outcomes
[verify]
p99 latency
Steady-state p99, and the same figure with a dependency degraded.
[verify]
Throughput
Sustained requests per second per instance under load test.
[verify]
Degraded-mode behaviour
What the service does when a downstream is fully unavailable — the number worth quoting in an interview.
What I'd do differently
- Timeouts are the foundation. A retry or circuit breaker on top of an unbounded call is decoration — the default "wait forever" is what actually causes the outage.
- Retries without jitter turn a blip into a thundering herd and make recovery slower than the original fault.
- Readiness and liveness probes are not interchangeable; conflating them turns a recoverable slowdown into a restart loop.
next case study
Kubernetes Observability Platform