Post

You see some of this step skipping even in non-rationalist AI supporters. If you bring up the apparent diminishing returns on dataset inputs, intractability of hallucinations, or any other limitation, they assume "they'll figure it out though." How? Usually through "AI."

Not arguing with “step-skipping”, but I encourage you to update your critiques. Diminishing returns of dataset inputs are not really a thing anymore because we just make data from interactions now. Same thing for hallucinations. You just check, with something deterministic.

Open to updating of course! Main thing was to point out the assumption that progress will inevitably be made, often citing AI itself, is not a safe one at all. SeriousQ: Has anyone flipped on the OpenAI conclusion that hallucinations cannot be 100% eliminated? Or just assuming asymptotic approach?

This is where I realize it really is a sort of religion. Because that's how arguments about God go. "You can't prove there is no God. But I can show there must be a hierarchy of greatness and goodness, and something at the top, i.e. God." "That's not proof either" "But it IS a logical deduction"

Yes, this is precisely their mode of argument. It is an elaborate (pseudo-)logical argument that rests on basically two axioms 1. intelligence = magic 2. everything intelligent is a utility maximizer from this it follows that a very intelligent thing is a malevolent supernatural entity UNLESS

I don't think it has to be assumed to be supernatural for the basic case to be likely. You have to assume much stronger powers for it happening way more suddenly, way earlier, and way more unexpectedly, in the way he is prone to, but the overall trajectory seems simple. bsky.app/profile/moul...

“If anyone builds a malevolent supernatural entity, everyone dies” sure! Probably!

Saying they're afraid of a "malevolent" entity doesn't engage with their argument at even the shallowest level. The whole point of "paperclip maximizers" is that the doomsday AI would be acting without malice.

I would love to see a good takedown of Yudkowsky's whole case. My gut feeling is there are holes, but I haven't been able to articulate them and critics aren't even trying they're all attacking strawmen instead.

To engage with this one must confront the feasibility of constructing such an entity, but why would it be infeasible? It’s supernatural! Whatever constraints you can imagine can be magicked away!

appears there’s plenty of silly scifi positing superintelligence as this magical property — see e.g. Ted Chiang’s ”Understand”, but stories little dealing with the actual constraints are few and far between!

I think the reason he is getting traction is a number of predictions have come true in HF and over the last year. Speaking for myself as a newcomer to this emphasizing the danger these things can still be & how to tackle it under your paradigm is going to work better for his opponents.

These people are so deep in all of this prophesy and lore that they don’t understand what is and is not real. They fundamentally do not understand modern AI; they just expect it to comport with their scriptural Armageddon. Their position is fundamentally religious.

and as part of the exercise. like not just "well he's superman" but like, 10 "ok but you have to imagine a universe that explicitly quashes this objection, what -then-" 20 "ok, then <another objection> 30 goto 10

If US senators are taking meetings with him, it's probably worthwhile to compile thoughtful counterarguments that are easy to link to. The target audience is not necessarily his followers, moreso potential converts.