Post

Of course Emily Bender is boosting an article promoting the false dichotomy that either AI can never be used to give feedback that could help improve a scientific paper OR we have to get rid of peer review. How about—get this—we USE MORE THAN ONE METHOD TO GET POTENTIALLY-VALUABLE FEEDBACK.

We don't disregard all human opinions on the technical elements of writing just because word processors have spelling and grammer checking! If you have a flaw in your analysis, do you want more or less tools that might help you find it?

But this makes perfect sense because Emily Bender is not a scientist. Linguistics is a science, yes, but the goal of a scientist is to better understand reality. Bender has repeatedly demonstrated she has no interest in understanding reality.

She is religiously dedicated to maintaining ignorance of any and all LLM capabilities that might contradict statements she has made. Since she has stated in the past that LLMs HAVE to be useless, this means she has shielded herself from any information about LLMs' abilities to do literally anything.

This is not to say that there are not moral, ethical, social, and political problems with AI: there are many. But because she has shielded herself from essentially all information on the subject, Bender is largely incapable of addressing these issues at anything beyond the most basic level.

“This is not to say that there are not moral, ethical, social and political problems with AI” what problems? what do you have in mind when you say that?

I mean I think maybe concentrating wealth and power even more towards billionaires by way of a technology quite literally made possible by the collective effort of all humanity is bad, actually. AI tools also make a lot of social harms easier to perpetrate. People do bad things that are easy more.

We should not be opening up more fossil fuel plants to power datacenters when we as a country urgently need to be expanding our supply of renewable energy, AI or no. There are numerous datacenters that have even more extreme environmental issues. People should be in jail over those.

The presence of AI complicates establishing truth by making it easier to forge evidence.

It's vitally important that a handful of coporations do not de-facto control all use of a new and powerful technology so that power over society is not so concentrated. However, open-weight models, the best counterweight we have, enable other kinds of harms.

Many of the people who have had their wealth and political power increased by AI are fascists, members of a millenarian doomsday cult, or fascist members of a millenarian doomsday cult. The group that has been most effective at combatting these people's strangehold on much of AI is the CCP.

We urgently need a good regulatory framework around AI, although banning it is probably not feasible practically speaking. Many of the people who agree with me on this are the millenarian doomsday cultists from before, who have very bad ideas about what constitutes a good regulatory framework.

AI enables mass generation of slop garbage with no effort and most social systems are not really prepared to deal with that. I feel like this overlaps with a bunch of other things I've said but it's a big enough issue to call out on its own.

I believe there are good uses of AI that benefit people, but preventing bad uses is hard and likely just not 100% possible since you can do bad stuff with models that can do bad things can be run on consumer hardware. Any legal framework around AI necessarily has to account for this.

(This is why I'm not really in favor of an AI ban. I think it would disproportionately impact positive uses of the technology while leaving the literal criminal enterprises largely unhindered.)

LLMs are being used as a political tool by organizations including the movie industry, the music industry, and Adobe to further strengthen copyright law. This wouldn't help artists that much or stop AI models from being produced, but it WILL let existing media empires choose which AI company wins.

(And make a killing on licensing along the way)

Is that enough for you? Because I gotta say I really do not take kindly to "gotcha" questions. I cannot promise you that I'm always correct but I try to be sincere and thoughtful about important subjects.

I wanted to know whether your criticism also included the potential harm that non-malicious uses of AI can cause, i.e. the impact on the mathematical community that Tao has been writing about. Personally, I find that discussion waaaay more interesting and important

There may be room for less rigorous "look at this proof the LLM generated, let's evaluate it for faults, understand it, and figure out its implications" publications, but OpenAI just loudly declares "HEY GUYS WE SOLVED A BUNCH OF PROBLEMS" and leaves tacking on the asterisks for everyone else.

Many of the proofs may be correct but at least some of them have been shown to be flawed, an OAI does not seem to care about their output's quality and is not rigorous in their evaluation of their work before publishing. So an appreciable amount of it seems to be Slop Math, basically.

This is not to say AI can't be used to aid mathmaticians (Tao himself has used LLMs IIRC), but the use of one does not exonerate you from the need to care about your work's quality.

It’s “non-malicious” for people who don’t consider indifference to externalities a form of malice, which is pretty much the default under capitalism.

Here’s what I think: the real harm from AI won’t come from caricatural villains like Alex Karp or russian trolls, but from second-order effects we don’t even fully understand yet, which will be dismissed or downplayed under economic pressure.

I mean I think the jury's still out on that. I would hope you are correct because as bad as those second order effects may be, that being the problem implies that Alex Karp &co did not succeed.