Post

Not to pick on Sam here, but this is a useful point to highlight some of the impossibility of trust & safety work. Meta actually used to have a program *exactly* like this, BUT it became a central feature of Frances Haugen's Facebook Files as an example of Meta being evil. I'll explain...

Content moderation at scale is impossible to do well. One part of that is that first pass moderation is often automated in some form, because the scale is way more than most people can wrap their heads around. Even if you could get it right 99% of the time, a lot of mistakes would be made.

Meta knew they got really bad press when their automated tripwires pulled down big name accounts. So they added in this secondary feature to kick enforcement actions to human review on those high profile accounts. But, when Frances Haugen leaked this to Jeff Horwitz, guess how it was covered?

Yup. It was framed as a special program to "exempt" a "secret elite" from enforcement, allowing them to "abuse" the privilege. And, yes, it appeared that Meta was sometimes too slow to do the human review. The Oversight Board eventually put out a whole report: www.oversightboard.com/news/5016549...

So, because of things like that, companies are less willing to do something like Sam's initial request to specifically monitor "big political accounts" to avoid things like having the No Kings Instagram taken down during its nationwide protest.