Post

😱 You see this distribution of response times from running 500 requests/sec against some microservice for 120 sec. What do you do to in order to investigate what's going on?

0 reposts7 likes

It runs queries against a database. I suspected the connection pool, but the same behavior is exposed when just returning some dummy data from the service itself. Starting to wonder whether it's an artifact of the load generator (oha) and/or running this on macOS. Will try on a Linux box.

Inject requestids into a function name (generate a pass through dynamically one way or another) and attach a sampling profiler. Log slow requests and then find them in the profile.

First, analyze using a JFR execution profile to find out which method is taking more time. If that doesn't yield results, use OTel instrumentation (if running on a JVM, use the Java agent) and analyze it using the response trace ID that is taking a longer time. I am using clickhouse+hyperdx for Otel

I would open the traces and look at the outliers. I've built rather detailed traces for our SaaS using standard java and bytebuddy for all BL and DAO methods; *some traces* would match and be sent to datadog for this period in our case. If still in doubt, I would look at the JFR runtime profiles

Turn on GC logs and see whether GC causes the outliers? Since the outliers seem to get worse over time, maybe there's a memory leak. So I'd also dump the heap towards the end and inspect it.