Post

😱 You see this distribution of response times from running 500 requests/sec against some microservice for 120 sec. What do you do to in order to investigate what's going on?

You measure for longer to see if the pattern repeats itself to see if there's any hint on when this happens. If possible, try to color it based on the container where it is running in?

It's just a single process, not running in a container. The pattern occurs always around the same time (20 sec in) after starting the load test.

What kind of program is it? Does it make requests? It kind of feels like there is an exponential backoff that doesn't trigger the first try.

It runs queries against a database. I suspected the connection pool, but the same behavior is exposed when just returning some dummy data from the service itself. Starting to wonder whether it's an artifact of the load generator (oha) and/or running this on macOS. Will try on a Linux box.

Try Hyperfoil 😉 Seriously, we've tested it against other load drivers and it's reporting is more accurate. New blog post coming in the not too distant future 🙂

Ha, yeah, turns out it was on the load gen side indeed. It was running out of connections. Using HTTP2 mitigates it (via pipelining), which I got reminded of reading the docs of... Hyperfoil ;) Is the project alive? Saw the last post is from a few years back.

Load gen as bottle eck, nice classic! Wasn't this what Gatling was originally meant to solve (vs jmeter which didn't do that well)?