Lately, I’ve been tinkering with a Postgres setup, mostly to see how much performance I could squeeze out of it. Nothing formal — just a curiosity-driven experiment to understand what’s possible with the right tuning.
As part of that, I started benchmarking. I wanted a sense of how the database was performing under load, so I tracked the average latency and the 95th percentile. The numbers looked great: average latency was sitting at 5ms, and the 95th percentile was around 10ms. That felt tight. I figured I’d done a good job and left it there.
A few days later, I came back to see if I could push it even further. This time, I added another metric: max latency. And that’s when everything changed.
Suddenly, I was staring at 200ms.
That number didn’t make sense at first. I had a 95th percentile of 10ms, but the worst-case requests were 20x slower. That’s not just noise — that’s a different class of performance altogether. What I realized was that my setup wasn’t consistently fast — it was fast most of the time, but occasionally really slow.
In a production system, this kind of behavior can be a silent killer. Most metrics dashboards will still look green, but users might intermittently hit painful delays. And in some use cases — like APIs where latency consistency matters more than raw throughput — I’d argue that a slower system with predictable latency is actually better than a fast system with random spikes.
I’ll share more about how I approached optimizing Postgres in a future article. But for now, this was a good reminder: averages (even percentiles) don’t tell the full story. Without looking at the full range — especially the outliers — you might walk away with a completely false sense of how your system is behaving.
If you’re benchmarking, don’t just look at what’s typical. Look at what’s possible.