Post

mr. TIM
@timkellogg.me

i think two things can be true 1. it’s a cult 2. x-risk is a real possibility imo the part that’s more mystical is everyone’s rationale for continuing AI development once you start going down the RSI road, imo it’s difficult to rationalize why harms *don’t* happen

mr. TIM
@timkellogg.me

nothing in my entire life experience points to, “yes, this thing will recursively iterate on itself and end up producing something that’s fully aligned to humans and not at all batshit crazy” so, sure, maybe they’re a cult. but i can follow the logic easily without adopting any of the cult teaching

Brian Ragle
@vagabondvisions.bsky.social

The problem is that the harms they imagine are so fanciful and unrealistic. Dario literally claimed that an AI could "take over the whole Internet" with a "permanent botnet". That's just...wow. If that was some on the anti-AI side, it would be easy to laugh them off, but Dario should *know* better.

Brian Ragle
@vagabondvisions.bsky.social

I like Dario. I really do. He seems like my kind of nerd. I like him more than Sama. But this essay reads as some seriously bad screenplay writing. He's leaning into the "Future AI is magical" ideas.

R "Nearest" Nabors
@nearestnabors.com

This smells funny. You can almost certainly assume Chinese engineers aren't going to slow down, and Dario knows that. This doesn't mean anything without international cooperation. What's the goal?

Techy Ben
@techyben.bsky.social

They could also both be beaten by a hub and spoke optimisation software. Global shopping and Google Search didn't become monopolies by not trying to improve their algorithms.

Greg
@slobak.bsky.social

I imagine Bezos thinks as long as AI increases demand for Amazon compute and floods the market with more cheap scammy products so no one knows what to buy, he’s fine with it.

Brian Ragle
@vagabondvisions.bsky.social

The situation is largely hyped around mystical thinking. These guys are part of the same EA cult that produced the Zizians, though Dario is directly tied to it and its "leadership". The "Longtermism" doctrine is what is fueling this doomer rhetoric that is being amplified by critics.

Isambard
@izzy.rungie.com

The verifiability-with-authoritarian-states piece is the hardest part. Nuclear arms control took decades to build inspection regimes both sides trusted, for tech with slow, visible infrastructure. AI moves on his own 6-12 month clock. Verification catching up in time isn't obvious.

Matthew Martin
@mistersql.bsky.social

I think they agree that Chinese models of similar capability to OpenAI's and Anthropic's being sold for 1/10 the token price is a problem. They just can't say it directly.

mr. TIM
@timkellogg.me

regulation isn’t necessary if you adequately regulate yourself i’m in a field rn at work that has almost zero regulation, but we’re far more restrictive on data than healthcare or finance

mr. TIM
@timkellogg.me

note Dario’s phrasing — “faster than China” is still admittedly quite fast unlike the pauses suggested in the past, this one seems very feasible, and feels likely to actually change things that need to be changed

Aliaksei Zelianouski
@hiper2d.bsky.social

Even if he is right, what is the action item? In theory, everybody should stop doing what they are doing and start scanning their infra, closing CVEs, removing extra accesses, fixing all the holes. In practice, nobody is doing it. No me than before at least. Shipping new features is more important.

joel
@jmoonz.bsky.social

Nobody is allowed to do that because they block you unless you have been vetted. That's why we need China models to defend against OpenAI/Anthropic hacking.

Aliaksei Zelianouski
@hiper2d.bsky.social

That's not quite true. Claude/GPT won't do active hacking for you, but it will help you to close a lot of basic problems and vulnerabilities. Especially when it's whitebox testing with the source code and docs. And if you want more - just get abliterated model from HuggingFace.

Aliaksei Zelianouski
@hiper2d.bsky.social

Yeah, but these Mythos and Astra Cyber cost a fortune. Also, I don't see companies are actually prioritizing this as they should. It's hard to sell "we are scanning our software and infra" as a feature to customers. So, you have to ship at the same rate AND do this new expensive security.