Post

Let's talk about #Trace #sampling, how it allows you to harness the power of distributed tracing without breaking the bank, how it adds inaccuracies, and how you fix that. 🧵

There are two commonplace ways of doing sampling: probabilistic head sampling and tail sampling. With probabilistic head sampling, it is like you throw a die for every trace, and keep it only if the die's face is low enough. With tail sampling, you keep *interesting* traces.

Head and tail sampling reduce the accuracy of your data, over or underestimating error rates and latencies of your services. (Btw, creating animations with LLMs is pretty addictive.)

What can you do to reduce inaccuracies? You additionally record rate, errors and durations (RED) metrics for your services on the full data before sampling. With #OpenTelemetry, you can do trace sampling and RED metrics collection inside the same Collector.

What other tricks do you have to improve the accuracy of sampling?

I've found that if you're needing to get ground truth on how "usable" your sampling is, you can use a zstd dictionary plus a custom collector processor to build up a cheap "offline" store of traces. That data set lets look at if your "online" metrics have a distribution that meets the actual data.

So effectively you would look at the distribution of matches and figure out if your probabilistic head sampling is random enough? Each span is indexed separately by which attributes?