Post

"Pace the frontier" Makes sense the broligarchs would choose this settler colonial metaphor, since their actual proposal is just a weak sauce columbusing of regulatory authority

'embedded evaluators' my ass, how about a 3rd party auditor from an actually existing regulatory agency with the legal power to demand access to whatever they need, for as long as they need it, while they determine whether you broke the goddamn law

To understand how AI auditing actually works, & why we need 3rd party (fully independent) auditors, not just 2nd party (contractors) or 1st party (internal teams), please read the 2022 paper Who Audits the Auditors? I coauthored w/ @rajiinio.bsky.social & @poetofcode.bsky.social www.ajl.org/auditors

Our paper was published at the key conference in the field (FAccT), cited hundreds of times, and provides very clear recommendations for policymakers re: how to hold AI companies accountable. facctconference.org/static/pdfs_...

Here are the key recs: 1. Require audits for AI system owners and operators. AI auditors are in near-universal agreement (95%) that audits should be a required element of owning and operating AI systems.

3. Mandate disclosure of key audit results to external stakeholders. The vast majority of AI auditors (82%) view disclosure of audit results as necessary [..] However, most do not yet disclose audit methods or findings themselves, typically citing client confidentiality concerns.

4. Require harm incident reporting and other mechanisms for real-world harm discovery. Many auditors say that, while their organization or their clients’ organizations have systems in place for users to report experiences of harm, they believe that those systems are ineffective.

5. Prioritize stakeholder involvement by affected communities. Less than half (41%) of auditors say that stakeholder involvement is currently a critical part of their audit process, and of those, just 30% say that they consider real-world harm to stakeholders (only two provide linked examples).