Debugging production at 130 km/h

We were driving across Poland — I wasn’t at the wheel — when something broke in production, and nobody on the team could figure out why.

I took out my laptop, asked the guys to switch on my work computer, tunnelled through it to the server and started digging right there in the passenger seat, at 130 km/h.

The database couldn’t keep up with the load: one query was executed far too many times. As far as I remember, it was either a missing index or the classic N+1 — a loop that fires one query per item instead of one query for all of them.

The lesson: when production is slow, look at the database first, and count queries, not just their duration. A fast query run a thousand times is a slow page.