Post

😱 You see this distribution of response times from running 500 requests/sec against some microservice for 120 sec. What do you do to in order to investigate what's going on?

You measure for longer to see if the pattern repeats itself to see if there's any hint on when this happens. If possible, try to color it based on the container where it is running in?

It's just a single process, not running in a container. The pattern occurs always around the same time (20 sec in) after starting the load test.

What kind of program is it? Does it make requests? It kind of feels like there is an exponential backoff that doesn't trigger the first try.

It runs queries against a database. I suspected the connection pool, but the same behavior is exposed when just returning some dummy data from the service itself. Starting to wonder whether it's an artifact of the load generator (oha) and/or running this on macOS. Will try on a Linux box.

Try Hyperfoil 😉 Seriously, we've tested it against other load drivers and it's reporting is more accurate. New blog post coming in the not too distant future 🙂

With oha, any idea what is the cause (reaching maximum number of open files, oha bug or something else)?

Yeah, I was running into connection limits, which you can sidestep by using keep-alive or HTTP2 (if that fits the spec for your test). But I'm seeing some random noise with oha, which make tail latencies not very trustworthy.

Did you specify --latency-correction and -q ?

@stalep.bsky.social, in case you have a moment, posted a question on Hyperfoil here: github.com/Hyperfoil/Hy... 🙏.

I see Franz did not take time off though :)

Haha, yeah, indeed. Are you gonna be at JavaZone, btw.?

Load gen as bottle eck, nice classic! Wasn't this what Gatling was originally meant to solve (vs jmeter which didn't do that well)?