Distributed systems rarely fail one component at a time. Under heavy load, a single slow dependency can trigger a chain reaction that spreads across services — a pattern known as a failure cascade. This walkthrough compares two common ways for services to communicate, a synchronous remote procedure call (RPC) model and an asynchronous event-driven model, and shows in code how each behaves when downstream services slow down, overload or throw transient errors.
The comparison simulates downstream services with variable latency, overload conditions and intermittent failures, then drives both architectures with bursty traffic. By watching metrics such as tail latency, retries, failures and dead-letter queues, it becomes clear how tightly coupled RPC calls amplify failures and how event-driven designs trade immediate consistency for resilience. The focus throughout is on practical mechanisms — retries, exponential backoff, circuit breakers, bulkheads and queues — that engineers use to contain cascading failures in production. The full code is available on GitHub.
Shared utilities and metrics
The first step defines the common utilities and data structures used throughout: timing helpers, percentage calculations, and a metrics container that tracks latency, retries, failures and tail behaviour. This provides a consistent basis for measuring and comparing the two execution models.
import asyncio, random, time, math, statistics
from dataclasses import dataclass, field
from collections import deque
def now_ms():
return time.perf_counter() * 1000.0
def pctl(xs, p):
if not xs:
return None
xs2 = sorted(xs)
k = (len(xs2) - 1) * p
f = math.floor(k)
c = math.ceil(k)
if f == c:
return xs2(int(k))
return xs2(f) + (xs2(c) - xs2(f)) * (k - f)
@dataclass
class Stats:
latencies_ms: list = field(default_factory=list)
ok: int = 0
fail: int = 0
dropped: int = 0
retries: int = 0
timeouts: int = 0
cb_open: int = 0
dlq: int = 0
def summary(self, name):
l = self.latencies_ms
return {
"name": name,
"ok": self.ok,
"fail": self.fail,
"dropped": self.dropped,
"retries": self.retries,
"timeouts": self.timeouts,
"cb_open": self.cb_open,
"dlq": self.dlq,
"lat_p50_ms": round(pctl(l, 0.50), 2) if l else None,
"lat_p95_ms": round(pctl(l, 0.95), 2) if l else None,
"lat_p99_ms": round(pctl(l, 0.99), 2) if l else None,
"lat_mean_ms": round(statistics.mean(l), 2) if l else None,
}Modelling failure and resilience primitives
Next, the code models failure behaviour and the resilience primitives that shape stability: overload-sensitive latencies and failures, alongside circuit breakers, bulkheads and exponential backoff to control cascading effects. These building blocks make it possible to experiment with resilient versus fragile configurations.
@dataclass
class FailureModel:
base_latency_ms: float = 8.0
jitter_ms: float = 6.0
fail_prob: float = 0.05
overload_fail_prob: float = 0.40
overload_latency_ms: float = 50.0
def sample(self, load_factor: float):
base = self.base_latency_ms + random.random() * self.jitter_ms
if load_factor > 1.0:
base += (load_factor - 1.0) * self.overload_latency_ms
fail_p = min(0.95, self.fail_prob + (load_factor - 1.0) * self.overload_fail_prob)
else:
fail_p = self.fail_prob
return base, (random.random() < fail_p)
class CircuitBreaker:
def __init__(self, fail_threshold=8, window=20, open_ms=500):
self.fail_threshold = fail_threshold
self.window = window
self.open_ms = open_ms
self.events = deque(maxlen=window)
self.open_until_ms = 0.0
def allow(self):
return now_ms() >= self.open_until_ms
def record(self, ok: bool):
self.events.append(not ok)
if len(self.events) >= self.window and sum(self.events) >= self.fail_threshold:
self.open_until_ms = now_ms() + self.open_ms
class Bulkhead:
def __init__(self, limit):
self.sem = asyncio.Semaphore(limit)
async def __aenter__(self):
await self.sem.acquire()
async def __aexit__(self, exc_type, exc, tb):
self.sem.release()
def exp_backoff(attempt, base_ms=20, cap_ms=400):
return random.random() * min(cap_ms, base_ms * (2 ** (attempt - 1)))The synchronous RPC path
The synchronous RPC implementation shows how timeouts, retries and in-flight load feed directly into latency and failure propagation. It illustrates how tight coupling in RPC turns transient downstream problems into wider outages under heavy traffic.
class DownstreamService:
def __init__(self, fm: FailureModel, capacity_rps=250):
self.fm = fm
self.capacity_rps = capacity_rps
self._inflight = 0
async def handle(self, payload: dict):
self._inflight += 1
try:
load_factor = max(0.5, self._inflight / (self.capacity_rps / 10))
lat, should_fail = self.fm.sample(load_factor)
await asyncio.sleep(lat / 1000.0)
if should_fail:
raise RuntimeError("downstream_error")
return {"status": "ok"}
finally:
self._inflight -= 1
async def rpc_call(
svc,
req,
stats,
timeout_ms=120,
max_retries=0,
cb=None,
bulkhead=None,
):
t0 = now_ms()
if cb and not cb.allow():
stats.cb_open += 1
stats.fail += 1
return False
attempt = 0
while True:
attempt += 1
try:
if bulkhead:
async with bulkhead:
await asyncio.wait_for(svc.handle(req), timeout=timeout_ms / 1000.0)
else:
await asyncio.wait_for(svc.handle(req), timeout=timeout_ms / 1000.0)
stats.latencies_ms.append(now_ms() - t0)
stats.ok += 1
if cb: cb.record(True)
return True
except asyncio.TimeoutError:
stats.timeouts += 1
except Exception:
pass
stats.fail += 1
if cb: cb.record(False)
if attempt <= max_retries:
stats.retries += 1
await asyncio.sleep(exp_backoff(attempt) / 1000.0)
continue
return FalseThe asynchronous event-driven path
The event-driven pipeline uses queues and background consumers. Events are processed independently of request submission, retry logic is applied, and undeliverable messages are routed to a dead-letter queue. This decoupling improves flexibility, but introduces new operational considerations of its own.
@dataclass
class Event:
id: int
tries: int = 0
class EventBus:
def __init__(self, max_queue=5000):
self.q = asyncio.Queue(maxsize=max_queue)
async def publish(self, e: Event):
try:
self.q.put_nowait(e)
return True
except asyncio.QueueFull:
return False
async def event_consumer(
bus,
svc,
stats,
stop,
max_retries=0,
dlq=None,
bulkhead=None,
timeout_ms=200,
):
while not stop.is_set() or not bus.q.empty():
try:
e = await asyncio.wait_for(bus.q.get(), timeout=0.2)
except asyncio.TimeoutError:
continue
t0 = now_ms()
e.tries += 1
try:
if bulkhead:
async with bulkhead:
await asyncio.wait_for(svc.handle({"id": e.id}), timeout=timeout_ms / 1000.0)
else:
await asyncio.wait_for(svc.handle({"id": e.id}), timeout=timeout_ms / 1000.0)
stats.ok += 1
stats.latencies_ms.append(now_ms() - t0)
except Exception:
stats.fail += 1
if e.tries <= max_retries:
stats.retries += 1
await asyncio.sleep(exp_backoff(e.tries) / 1000.0)
await bus.publish(e)
else:
stats.dlq += 1
if dlq is not None:
dlq.append(e)
finally:
bus.q.task_done()Running the experiment
Finally, both architectures are run under heavy workloads and the results compared. The experiment collects metrics, identifies end consumers explicitly, and ties together latency, throughput and failure behaviour in a consistent, system-level comparison.
async def generate_requests(total=2000, burst=350, gap_ms=80):
reqs = ()
rid = 0
while rid < total:
n = min(burst, total - rid)
for _ in range(n):
reqs.append(rid)
rid += 1
await asyncio.sleep(gap_ms / 1000.0)
return reqs
async def main():
random.seed(7)
fm = FailureModel()
svc = DownstreamService(fm)
ids = await generate_requests()
rpc_stats = Stats()
cb = CircuitBreaker()
bulk = Bulkhead(40)
await asyncio.gather(*(
rpc_call(svc, {"id": i}, rpc_stats, max_retries=3, cb=cb, bulkhead=bulk)
for i in ids
))
bus = EventBus()
ev_stats = Stats()
stop = asyncio.Event()
dlq = ()
consumers = (
asyncio.create_task(event_consumer(bus, svc, ev_stats, stop, max_retries=3, dlq=dlq))
for _ in range(16)
)
for i in ids:
await bus.publish(Event(i))
await bus.q.join()
stop.set()
for c in consumers:
c.cancel()
print(rpc_stats.summary("RPC"))
print(ev_stats.summary("EventDriven"))
print("DLQ size:", len(dlq))
await main()What the comparison shows
The trade-off between the two models is clear. RPC delivers low latency when dependencies are healthy but grows fragile under saturation, where retries and timeouts quickly escalate into system-wide failures. The event-driven approach isolates producers from consumers, absorbs bursts through buffering and localises failures — but it demands careful handling of retries, backpressure and dead-letter queues to avoid hidden overloads and unbounded queues. Resilience, in other words, comes not from picking one architecture but from pairing the right communication model with disciplined failure-handling patterns and capacity-aware design.
Limitations and what to watch
This is a simulation intended to build intuition, not a benchmark of any specific production system. The latency, error and overload behaviours are modelled with simplified assumptions, so absolute numbers will differ from real infrastructure, where network conditions, service dependencies and hardware all play a role. The two models are also not mutually exclusive: most real systems combine synchronous and asynchronous communication, and the resilience primitives shown here — circuit breakers, bulkheads, backoff and dead-letter queues — apply to both. The value of the exercise lies in the relative behaviour it exposes under stress rather than in any single measured figure.
