Fred Hebert

Fred Hebert

@ferd.ca

Principal SRE @ honeycomb.io, Tech Book Author, Resilience in Software Foundation board member, Erlang Ecosystem Foundation co-founder, Resilience Engineering fan. SRE-not-sorry. blog: https://ferd.ca notes: https://ferd.ca/notes/

2.1K followers191 following583 posts

Pinned

People tend to have a mental model where a system is stable until disturbed, far more often than they have one where the system is balanced because it is constantly intervened with. The latter is a more useful approach to thinking about complex systems.

I keep seeing people mentioning how ai-generated code lets you spend more time thinking about what the app should do, but I think there’s a need to remember that there are large groups of engineers whose role hasn’t been redefined to include that level of control and trust.

Many should demonstrate judgment by copy/pasting tickets they have no say about, into prompts that generate code for them, and then approving/denying at a higher rate than before (or while hands off), to meet deadlines also out of their control. You get accountability but can’t do much about it.

Many workplaces have adopted a discourse of empowerment met by just increasing productivity pressures, and what you get is people being told their skills are still valued while practically being shown they aren’t. Asking engineers to reimagine their work isn’t something they can do in isolation.

Me? No, I would never financially pay for an open source ecosystem. I’d much rather pay indirectly through elaborate systemic consequences. That lets me still find people to be mad at, and that’s comforting. Also think of the content.

I need people using AI to spam my inbox about AI products to realize that patterns like: "hey, I saw your work about <whatever>. I like <whatever the point is> and I would like your opinion on <shit I'm selling>." are so numerous they're obvious and all I ask of them is to please shut up.

I'm seriously getting multiple of these a day now, it's ridiculous, and spam filters have not yet fully caught up.

I first put learnyousomeerlang.com online in 2009, while I was still working my first PHP job. Today, 17 years later, I’ve had to take it down for maintenance for the first time while awaiting a PHP security patch. It uses it for variable substitution in links and establishing cross-chapter links.

Disabling all certificate checking in my client code so nobody uses the bad intermediary certs to own me

An element in this tooling we built is the approach: if we're stuck solving puzzles, we can either try to make a puzzle-solving machine to make puzzle-solving faster, or we can re-represent the components of the puzzle such that they're less demanding, less of a puzzle overall. We chose the latter.

honeycomb.ioHow Honeycomb Private Cloud Drinks From the Fire HoseHow Honeycomb's Private Cloud team built automated drift reports and an AI triage skill to keep up with configuration changes from every other team.

In 1997, James Reason published 'Managing Organizational Accidents'. It was influential in popularizing his concepts around the Swiss Cheese model, his specific brand/recipe of safety (and just) culture, etc. and discussed ever since. But it also has this line in it just throwing incredible shade:

mornings are starting to be colder which means it is once again time to hover my hands above the toaster while it cooks toast like it’s a campfire

I think it’s now been more than a year since I last wrote software just for fun (with or without AI). It feels more like chores or work by now. It feels like it had been a long time coming, but it just became clearer recently.

Fred Hebert reposted

If I were a scientist my email signature would absolutely contain the words “more research is needed”

I just realized I hadn’t shared this one yet, but I wrote about how the team I’m on restructured its workflows to lean into the code review bottleneck rather than trying to eliminate it when code generation took over, and what that ended up doing: www.honeycomb.io/blog/embraci...

honeycomb.ioHow I Came to Embrace the Code Review BottleneckFaced with an endless stream of AI-generated code reviews, our team made the counterintuitive choice to lean into the bottleneck rather than reduce it.

Fix code review bottlenecks by doing like private torrent trackers and only allowing people to get their PR reviewed if they reviewed enough PRs beforehand to keep their ratio high enough.

The first rule of the papers-reading club is that you’re unfortunately already in the papers-reading club and I will send you links and summaries

Yet again hearing how some skilled work never truly mattered now that it's getting automated, while it absolutely did and was a point of professional pride for many. It's a significant aspect of 'deskilling', which is a known consequence of automation. Pretending otherwise is needlessly unkind.

Although there have been changes in cognition and systems theory, I wanted to bring up some stuff Rasmussen published in the 80s that was really elegant. I'll use 3 diagrams he published, covering the ideas of abstraction hierarchies and how people operate systems when troubleshooting them.

View full thread

The cool thing though is that he then maps a problem-solving path as a line walking that conceptual path within a "conceptual reasoning space" of operators that is event-independent. Neelam Naikar (doi.org/10.1016/j.ap...) re-renders diagrams Rasmussen had made in a few publications in the mid 80s:

This is the "abstraction-decomposition space". It maps a troubleshooting workflow from a system symptom ("no communication to tape or disk") via a diagonal walk to isolate a faulty component ("short-circuited transistor"). This is such an incredible visualization of so many complex elements!

Although there have been changes in cognition and systems theory, I wanted to bring up some stuff Rasmussen published in the 80s that was really elegant. I'll use 3 diagrams he published, covering the ideas of abstraction hierarchies and how people operate systems when troubleshooting them.

View full thread

Rasmussen maps these as a space across two dimensions: abstraction (purpose to physical) and decomposition (whole-system to individual components). Each component can have a physical set of properties, but also a high level purpose it is connected to; so do aggregated portions of a system.

He then describes how workers do their work by skipping around abstraction layers, going up and down the hierarchies based on both purposes (high-level goals and functions) and reasons (bottom-up signals) but also across various aggregations (whole systems vs. components).

Although there have been changes in cognition and systems theory, I wanted to bring up some stuff Rasmussen published in the 80s that was really elegant. I'll use 3 diagrams he published, covering the ideas of abstraction hierarchies and how people operate systems when troubleshooting them.

The first one is the abstraction hierarchy from Coping With Complexity (1981). This shows how requirements and design tends to go top-down based on goals that must be met by the system. However, signals of problems tend to go bottom-up, from a local issue showing up as disrupted functionality.

The 2nd one (same paper) shows aggregation, where a nuclear power plant can be described as a whole conceptual system (top), as functional sub-units with material elements (middle), or as granular units that get closer to the material world (bottom).

Wrote up a bunch of stuff about some patterns in system design, about the tension, contrast, and possibility of composing approaches of analytical decomposition to increase control, and of complexity-aware stances for emergence, and some pitfalls of either stance: ferd.ca/control-and-...

ferd.caControl and complexity: tension in systems designcomposing two broad approaches, one based on analytical decomposition that aims to maintain control over a system, and one based on a perspective of complex systems that resist analysis, and implicati...

Delegating pressure to the final individual, who is now tasked with continuously fixing the entire system’s misalignments through their personal choices.

I was a reviewer for this book so I can give you some spoilers about it: it rules, get a copy of it, start telling everyone about cumulative culture, unbuild the myths that keep your teams down, make shit be meaningful.

Cat Hicks@grimalkina.bsky.social

IT'S POST'S BOOK BIRTHDAY! 🥳 Want a guidebook to the human side of tech? Want to cut through the noise of despair and anger to craft more meaningful work? Want to get sharp metacognitive strategies for this new era? Get all that *and* Launch Squad bragging rights here: www.drcathicks.com#book

One of the ironies about AI agents in ops tasks is that it feels like there has never been as much interest in creating a forgiving environment with proper structural support than through promising to remove humans from it, finally forcing a less individualistic and blameful approach to design.

These Angine de Poitrine fans got the best flags I’ve seen in a while; that’s some amazing concert gear.

Even after years, one of the weirdest parts of gardening to me is needing to harden the seedlings before transplanting them. Like “yes hold on a minute I gotta take the plants out so they can play outdoors for a while, but they gotta be in before streetlights turn on” is a real and necessary thing.

The ongoing stream of software engineering pieces that mention that the future is in writing spec but never bother to define what a specification is or at what abstraction levels it should be is appalling; arguably, tickets are a spec, the code is a spec, and work between both is connecting dots.

The underlying argument as far as I can tell is that we might now operate at a higher level of abstraction in spec definition, but without stating which level it is or how we can switch across abstraction layers for various degrees of specificity, this whole thing is meaningless.

“[…] we reached for Recon, an amazing tool for diagnosing issues […] (the related Erlang in Anger is more-or-less required reading as all on-call engineers end up scouring its pages eventually)” Wild! I wrote these 10 years ago to help coworkers, and they’re still useful for real world issues now!

discord.comYou’ve Got (Too Much) Mail: Behind the Scenes of the 3/25/26 Voice OutageOn March 25th, voice and video on Discord suffered major degradation beginning at 12:13 PDT, lasting a little over three hours. Learn how the issue originated, how it affected systems across Discord, ...

Infinite love to my SRE coworkers, one of whom casually dropped this single line in a retro: "the industrial hourly deploy train and its consequences have been a disaster for society"

I wrote for @resilienceinsoftware.org on "Superficial Blamelessness", where under the label of "blamelessness", we avoid punishing people, yet still focus fixes and interventions based on the same individualistic framing rather than a broader systemic stance. resilienceinsoftware.org/news/11502437

resilienceinsoftware.orgSuperficial BlamelessnessIn 2012, John Allspaw (then CTO of Etsy) wrote a seminal blog post on the need for what he called Blameless Postmortems. Built off the notion of a “Just Culture” from the research of Sidney Dekker, he...