Post

"Pace the frontier" Makes sense the broligarchs would choose this settler colonial metaphor, since their actual proposal is just a weak sauce columbusing of regulatory authority

'embedded evaluators' my ass, how about a 3rd party auditor from an actually existing regulatory agency with the legal power to demand access to whatever they need, for as long as they need it, while they determine whether you broke the goddamn law

To understand how AI auditing actually works, & why we need 3rd party (fully independent) auditors, not just 2nd party (contractors) or 1st party (internal teams), please read the 2022 paper Who Audits the Auditors? I coauthored w/ @rajiinio.bsky.social & @poetofcode.bsky.social www.ajl.org/auditors

Our paper was published at the key conference in the field (FAccT), cited hundreds of times, and provides very clear recommendations for policymakers re: how to hold AI companies accountable. facctconference.org/static/pdfs_...

Here are the key recs: 1. Require audits for AI system owners and operators. AI auditors are in near-universal agreement (95%) that audits should be a required element of owning and operating AI systems.

2. Notify individuals when they are subject to algorithmic decision-making. Mandatory notification when individuals are subject to algorithmic decision-making is a widely supported regulatory intervention.

4. Require harm incident reporting and other mechanisms for real-world harm discovery. Many auditors say that, while their organization or their clients’ organizations have systems in place for users to report experiences of harm, they believe that those systems are ineffective.

5. Prioritize stakeholder involvement by affected communities. Less than half (41%) of auditors say that stakeholder involvement is currently a critical part of their audit process, and of those, just 30% say that they consider real-world harm to stakeholders (only two provide linked examples).

6. Formalize evaluation and consider accreditation. AI auditors express concern that their peers don't engage in rigorous audit[s.. &] companies perform cursory audits of their own algorithms if given the chance ... In other industries, accreditation is typically provided by an accreditation body.