Post

Michele Mancioppi
@michele.dash0.com

Let's talk about #Trace #sampling, how it allows you to harness the power of distributed tracing without breaking the bank, how it adds inaccuracies, and how you fix that. 🧵

Michele Mancioppi
@michele.dash0.com

There are two commonplace ways of doing sampling: probabilistic head sampling and tail sampling. With probabilistic head sampling, it is like you throw a die for every trace, and keep it only if the die's face is low enough. With tail sampling, you keep *interesting* traces.

Michele Mancioppi
@michele.dash0.com

Head and tail sampling reduce the accuracy of your data, over or underestimating error rates and latencies of your services. (Btw, creating animations with LLMs is pretty addictive.)

0 reposts0 likes

Michele Mancioppi
@michele.dash0.com

What can you do to reduce inaccuracies? You additionally record rate, errors and durations (RED) metrics for your services on the full data before sampling. With #OpenTelemetry, you can do trace sampling and RED metrics collection inside the same Collector.

MartinDotNet
@martindotnet.bsky.social

You can also record sample rate on you traces and logs, then extrapolate at runtime. At low scales it doesn't work, at large scales we've see it works incredibly well. Couple that with adaptive tail sampling and you have the best of both worlds.

Michele Mancioppi
@michele.dash0.com

The two-tiered routing setup is the same. I will have a thread on BlueSky about this coming out in a bit. This is the same content on LinkedIn: www.linkedin.com/posts/michel....

linkedin.comTail Sampling with OpenTelemetry: What You Need to Know | Michele Mancioppi posted on the topic | LinkedInLet's talk about *tail sampling*, what it is, why it helps, and how you do it with #OpenTelemetry. (If you like this post, please leave a like and a comment. It massively helps it reach others who ma...
Green (Not A Bot)
@355e3b.bsky.social

I've found that if you're needing to get ground truth on how "usable" your sampling is, you can use a zstd dictionary plus a custom collector processor to build up a cheap "offline" store of traces. That data set lets look at if your "online" metrics have a distribution that meets the actual data.