I built Uber's backend. It crashed at 5,000 drivers, and these three fixes held it up

The core of a ride-hailing app is a location service. Drivers report where they are, riders want to see the cars around them move in real time. It sounds like a map and a loop, and for fifty drivers it is. This post follows that service from one pod to an autoscaled deployment and records where each version stops working, measured with k6 on a laptop.

The code, the Kubernetes manifests and the k6 script are on GitHub: huseyinbabal/uber-system-design.

Results at a glance

The exact thresholds depend on the machine. The shape of each failure does not.

The service

One Go binary, two WebSocket endpoints and a page that draws the map:

Polling would also work: the rider asks every second and usually gets back almost the same list. With one long-lived WebSocket per app the server pushes instead, and the connection setup happens once. The price is that every online driver and rider holds a connection, and that connection lives in exactly one pod.

The driver side reads positions and stores them:

var (
	mu      sync.RWMutex
	drivers = map[string]Location{} // every driver this pod knows about
)

func save(l Location) {
	mu.Lock()
	drivers[l.Driver] = l
	mu.Unlock()
}

// inside driverHandler: one loop per connected driver
for {
	_, msg, err := c.Read(ctx)
	if err != nil {
		return
	}
	var l Location
	if json.Unmarshal(msg, &l) != nil {
		continue
	}
	l.Driver, l.Pod, l.At = id, pod, time.Now().UnixMilli()
	save(l)
}

The rider side runs a one-second ticker and filters the map by distance:

func nearby(lat, lng, km float64) []Location {
	cutoff := time.Now().Add(-15 * time.Second).UnixMilli()
	out := []Location{}
	mu.RLock()
	defer mu.RUnlock()
	for _, d := range drivers {
		if d.At > cutoff && distanceKm(lat, lng, d.Lat, d.Lng) <= km {
			out = append(out, d)
		}
	}
	return out
}

distanceKm is the haversine formula. Note the loop: every rider, every second, walks every driver. That is fine for now and is the subject of part 2.

The lab

HAProxy needs one setting for WebSockets: a tunnel timeout long enough that idle connections are not cut.

controller:
  config:
    timeout-tunnel: "1h"

Set it up:

git clone https://github.com/huseyinbabal/uber-system-design
cd uber-system-design
kind create cluster --config k8s/kind.yaml
helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm install haproxy haproxytech/kubernetes-ingress -n haproxy --create-namespace -f k8s/haproxy-values.yaml --wait
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm install metrics-server metrics-server/metrics-server -n kube-system --set 'args={--kubelet-insecure-tls}' --wait
docker build -t location-service:v1 app
kind load docker-image location-service:v1 --name uber
kubectl apply -f k8s/app.yaml

The map is at http://localhost:8088. The k6 script takes the peak and the ramp time from the environment:

MAX=50 RAMP=10s k6 run k6/drivers.js

It records three custom metrics: rider_drivers_seen (how many drivers each push contained), rider_update_gap (time between two pushes, 1 s when healthy) and ws_connect_failed.

1. One pod and a Friday night

Raise the open-file limit first, otherwise k6 runs out of sockets before the server does:

ulimit -n 65000
MAX=7000 RAMP=3m k6 run k6/drivers.js

Memory grows with every connection: each one has its goroutines, read buffers and an entry in the map. Somewhere around 5,000 drivers the pod reaches its 256 MiB limit and the kernel kills it. k9s shows it:

NAME                               READY STATUS           RESTARTS CPU MEM %MEM/L
location-service-7fc64cd764-8946f  0/1   OOMKilled         1 191 188     73output

Then it gets worse. When the pod restarts, every driver whose connection dropped reconnects at the same moment, the new pod fills up in seconds and dies again. After a few rounds Kubernetes backs off:

NAME                               READY STATUS           RESTARTS CPU MEM %MEM/L
location-service-7fc64cd764-8946f  0/1   CrashLoopBackOff        3   0   0      0output

k6 counts the damage:

ws_connect_failed.....: 88.06% 174941 out of 198651output

A real client would reconnect with exponential backoff and jitter so that a restart does not become a stampede. The server side still needs more than one process, because one pod can only hold so many connections in its own memory.

2. Three pods behind HAProxy

kubectl scale deploy/location-service --replicas=3

The balancing algorithm matters more for WebSockets than for short HTTP requests. A connection stays on its pod for as long as the driver is online, so the algorithm only ever decides where new connections go:

The Ingress asks HAProxy for least connections with one annotation:

metadata:
  annotations:
    haproxy.org/load-balance: leastconn

The connections spread evenly. The map does not: with the same 3,000 drivers it shows about a third of them. Each pod only knows the drivers connected to itself, and a rider sees whatever their own pod has in memory.

MAX=3000 RAMP=1m k6 run k6/drivers.js

rider_drivers_seen....: avg=115.909118 p(50)=125   p(95)=152    max=168
rider_update_gap......: avg=1s         p(50)=1s    p(95)=1s     max=1.03s
ws_connect_failed.....: 0.00%  0 out of 3300output

On one pod the same run gave about 330. Nothing failed, every push arrived on time, and the answer was still wrong. That is the kind of bug that load balancing on its own introduces.

3. Redis pub/sub

The pods have to share positions. The smallest change is a broadcast over one Redis channel named locations. Whichever pod holds the driver’s connection sends the position there; all pods, the sender too, listen on it and write each message into their own map.

// driverHandler: publish instead of save
if rdb != nil {
	b, _ := json.Marshal(l)
	rdb.Publish(ctx, "locations", b)
} else {
	save(l)
}

func subscribe() {
	for msg := range rdb.Subscribe(context.Background(), "locations").Channel() {
		var l Location
		if json.Unmarshal([]byte(msg.Payload), &l) == nil {
			save(l)
		}
	}
}

kubectl apply -f k8s/redis.yaml
kubectl set env deploy/location-service REDIS_ADDR=redis:6379
MAX=3000 RAMP=1m k6 run k6/drivers.js

The riders see about 340 drivers again, as many as with a single pod, and the map shows dots in three colours mixed together. The connections stay split, so the memory does too:

NAME                               READY STATUS   RESTARTS CPU MEM %MEM/L
location-service-6c76868786-ftjsr  1/1   Running         0  86  57     22
location-service-6c76868786-ljssc  1/1   Running         0  92  57     22
location-service-6c76868786-rsz6j  1/1   Running         0  88  57     22
redis-75c7d7d78c-jfdsn             1/1   Running         0  29   7    n/aoutput

Two things to know about this design. Redis pub/sub is fire and forget: a pod that is restarting misses the messages sent meanwhile, which is acceptable here because every driver sends a fresh position 4 seconds later. And every pod still stores every driver, so the map grows with the city, not with the pod count.

4. Autoscaling

A fixed replica count is a guess about traffic. Ride demand swings hard over a day, so any fixed number is either wasted capacity in the quiet hours or too little at the peak. A Horizontal Pod Autoscaler on memory follows the connections, since memory is what grows with them:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: location-service
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: location-service }
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: memory
        target: { type: Utilization, averageUtilization: 50 }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0

averageUtilization: 50 is measured against the request (256 MiB), so a pod counts as busy at 128 MiB. stabilizationWindowSeconds: 0 lets it scale up as soon as the average crosses the line.

kubectl scale deploy/location-service --replicas=1
kubectl apply -f k8s/hpa.yaml
MAX=10000 RAMP=4m k6 run k6/drivers.js
kubectl get hpa,pods -w

No pod was killed on the way to 10,000. Least connections matters here too: each new pod starts empty and HAProxy sends it the new connections first.

Two limits of this setup: existing connections never move, so a new pod only relieves the others as drivers come and go. And the HPA only changes the replica count; on kind the nodes are fixed. On a managed cluster, the Cluster Autoscaler (or Karpenter) is what provisions extra nodes when new pods stay Pending.

What breaks next

The architecture now is drivers and riders → HAProxy → N pods → Redis. The weak spot is nearby: every pod checks the distance from every rider to every driver in the city, once a second. With D drivers and R riders that is D × R distance calculations per second per pod, and both numbers grow together. A geo index fixes that, and it is where part 2 starts.

Clean up

kind delete cluster --name uber

Sources

• Project on GitHub: huseyinbabal/uber-system-design
• HAProxy Kubernetes Ingress Controller: annotations (load-balance)
• Redis: Pub/Sub
• Kubernetes: Horizontal Pod Autoscaling
• k6: WebSockets (k6/websockets)
• coder/websocket