Post

Hard to believe how bad this piece is; each answer contains at least one straightforward falsehood or fallacy

“superintelligence is science fiction” This says nothing: LLMs were science fiction 8 years ago. en.wikipedia.org/wiki/Appeal_...

“Does the existential threat from AI reflect a real concern that you observe on a daily basis among young researchers?... Not among researchers.” Wrong: 51% of researchers assign >10% to human extinction or similarly bad AI outcomes. aiimpacts.org/wp-content/u...

"at these companies… Many of them aren’t really engineers; they come from the humanities… they don’t do engineering work…” Anthropic’s technical arm is 40% hardcore infra people; their median prior work experience is 12 years; <1% have a philosophy background www.htx.com/pt-pt/news/w...

The use of “young” as an ad hominem is also surprisingly miserable, for a teacher.

"Geoffrey Hinton has never created a company that offers real services for real people…" Besides being irrelevant, also false: this one became Google's new image search. techcrunch.com/2013/06/12/h...

“Hinton believes that we have figured out how the brain works” No, he thinks the opposite. analyticsindiamag.com/ai-news-upda...

“we can reproduce it in silicon…” No. Hinton used to push a neuromorphic approach to AGI, but switched in 2023 after coming to believe that backprop is superior to the brain’s algorithm(s). www.youtube.com/watch?v=-9cW...

“None of the steps in this argument is credible” If Jordan is using “brain” as a metaphor for neural nets, then a weak form of the above, data parallelism, happens every day. (Hinton does believe in swarm updates - not obvious or publicly known, but highly credible) www.reddit.com/r/agi/commen...

“OpenAI agents hacked [HF]… they were conducting rather stupid experiments, letting the agents do whatever they wanted and encouraging this behavior.” Wrong: 1) their eval sandbox had no internet access; 2) their prompt forbade HPIM's methods, 3) their grader didn’t (explicitly) reward it.

(I can agree, though, that it is stupid to make superhuman hackers in the first place, and stupid to run security as they have been.)

“Think about engineers in the past: in chemical engineering, we would never have had a situation where explosions were allowed to happen just to see how badly things could go wrong”. Actually they totally do. www.youtube.com/watch?v=1yrb...

"Yoshua is a little different; he takes action and does things that directly focus on safety." Lots of people take action. And almost no one has ever taken such effective action as Coxon did. shallowreview.ai/overview

"I also don’t think they have a very clear idea of their business model. What’s the product? OpenAI tells us it’s going to create a personal assistant." The product is already automated software production and research assistance and also as much mathematics as you can stand.

"they’re not...eager to send money to Anthropic or OpenAI...With Chinese open-weight models...entirely accessible...most companies are looking in that direction" No; only about 20% of them, last year. Enterprises buy Anthropic tokens at insanely increasing rate: +30% a month x.com/martin_casad...

"anyone who has worked in education knows that, most of the time, children don’t like sitting in front of a computer" Has this man met a child