Post

Rian van der Merwe
@rianvdm.com

This was a rough incident for our team, but if you want to know more about our data pipeline, you might enjoy this post...

Cloudflare@cloudflare.social

On November 14, 2024, Cloudflare experienced a Cloudflare Logs outage. About 55% of the logs we normally send to customers were not sent and were lost. Here's what happened and what we’re doing to prevent recurrences: blog.cloudflare.com/cloudflare-i...

blog.cloudflare.comCloudflare incident on November 14, 2024, resulting in lost logsOn November 14, 2024, Cloudflare experienced a Cloudflare Logs outage, impacting the majority of customers using these products. During the ~3.5 hours that these services were impacted, about 55% of t...

1 repost2 likes

Andy Nortrup
@nortrup.dev

As a fellow log slinger, I'm both awed by the scale and frankly terrified of fighting through an incident like that. Customers get quite upset when the logs stop flowing.

Rian van der Merwe
@rianvdm.com

Here’s a really great response from @norootcause.surfingcomplexity.com. (It was wild seeing this show up in my RSS feed—I’m a little starstruck by the fact that Lorin wrote about our incident.)

surfingcomplexity.blogQuick takes on the latest Cloudflare public incident write-upCloudflare consistently generates the highest quality public incident writeups of any tech company. Their latest is no exception: Cloudflare incident on November 14, 2024, resulting in lost logs. I…
Rian van der Merwe
@rianvdm.com

Lots of things in here that I fully agree with, like: “I believe that all complex systems failures result from the interaction of multiple contributors that all had to be present for the incident to occur and to be as severe as it was.”