Post

Googling was an under-appreciated skill 20 years ago. Most people got passable results but those of us who were good at Googling knew the advantage we had. We’re at this stage with prompting.

💯 "Journalism in AI" would be a pretty good class too. Traits of a good journalist: Ask good questions, *be prepared* before you start asking questions, explain well, be skeptical, verify . . . every one of them is useful when working with generative AI.

Professional skills is SUCH a better term

Had this exact conversation with my college-age family members yesterday: That art history class your generation of student has been shitting on because it's a gen-ed and no one cares just got *infinitely* more valuable because of image generators. You screw yourself blowing it off.

How does the kid or even an adult novice know their question-asking is lacking though? Seems they need to be motivated and also realize the chatbot responses are not good enough and then iterate?

Yeah it's amazing how much the cost of asking a question has plummeted. I've always googled everything and even still, find myself asking LLMs even more questions

the BA crowd went from telling us that engineers should take English because communication and reading skills were important to telling us communication skills aren't real skills when you are communicating with something that isn't human

Did you check las @deeplearningai.bsky.social course? www.deeplearning.ai/short-course... prompting with reasoning models is a new branch of knowledge. Furthermore, the prompt of a chat is not the same as the tricks that have to be done to put an LLM behind a product ready for production.

tool use confuses the hell out of people until they sit down and see what happens when prompting the general LLM vs telling it to use a tool. Usually a pretty cool aha moment for them

Very true, but is not obvious, like being good at google and filtering information or debugging, is not something you understand when you lack it, only when you get decent/good at it

I can't wait until the day that everyone is just reading ChatGPT (or whatever is next) at itself in a boardroom.

Case in point -- every time I've seen Carl Bergstrom criticize the answer an AI tool gave him on this app, he's inevitably used a bad prompt, and he can be dismissive when he gets that feedback in replies.

“Other persons” and “operations and use” mean that as @lilianedwards.bsky.social pointed out to me here, it’s not just for your lawyers or your data engineers

Wow the AI summary on that page is pointless, it rewords the content to be the exact same length without adding anything useful at all!

My initial intuition when learning was also that it was just “typing text into a chatbot”. I didn’t believe there was skill to it until I tried to introduce others to small tasks like writing scripts and witnessed their early attempts, frustration and dismissal of the tool’s usefulness entirely

Like the internet was supposed to make us smarter, it also made many dumber. LLM's will make many more productive and others much less, because they wont agree that using it is a skill and will let the AI do all the work. They will be the "flat-earthers", because in 15 years, everyone will use it.

It's still new. Give it 10 years.

I think it's very different. Working with an LLM with e.g. full computer control is a totally different ballgame than one without it.

I am a bloated sack of meat after three days of christmas-esque eateries right now, incapable of writing; I'll see if I can grab a few gems, properly redact sensitive stuff & write it up properly later...

It's hard doing thorough QA when the potential inputs to your tool span every possible combination of words across all human languages!

Haha I a word... 9 months ;) so hot on the heels of the ai craze

this is a very important one IME. maintaining (and snooping on) my RAG slackbot for the last couple years has shown me people have a wide range of intuitions for what it “ought” to be able to help with. most expect too much

Btw. I appreciate your rebuttal here and don’t dispute you have some good points

They're still LLMs with roughly the same capabilities, so the most important lessons about what problems are a good fit for LLMs and what problems aren't tend to hold steady

My thoughts on this: LLMs are not testimonial sources. We split fact-checking responsibilities with testimonial sources: we can't do that here. It's a pattern matcher, and not all real patterns are real things. Appropriate problems are ones where this status is OK. Not an answer, but a start?

Another one is knowing in which domains you can reliably identify errors. There's a level of dispassionate self awareness that's really important when working with them.

genuine question - where would the eventual line between the ai equivalents of "being good at excel idiosyncrasies" and "being good at data science" be drawn?

I don't think students should take classes on how ChatGPT Code Interpreter differs from ChatGPT Canvas - that would be like teaching MS Word specific VB macros in "intro to computers" I do think there's a lot of space for helping teach students how to navigate these utterly weird new AI systems

I do think that generally, the more skillfully one uses GenAI, the more interesting are the artifacts that emerge downstream. And staying on top of what systems & capabilities are available takes some elbow grease. Even if you have an ai doing *that* for you, can’t imagine it being a “set & forget”

So the Venn diagram of "prompting skill" and "question asking skill" will gradually overlap more and more as models get better? That makes sense. Thanks, Simon 🙏

I think a shift might happen if we move away from auto-regressive token generation, but not before then. Given the training corpus of models, and the need to give coherent answers, once a model chooses a "bad" token, the rest of the response is cooked because it can't easily back away from it

Strongly agree. I’m an AI power user and have been right from the beginning and I find that it’s a constant, painful struggle. I have to force myself to use AI tools and figure out how to make the most of them and while the tools are getting better it isn’t really getting any easier.

Those kinds of fundamental research skills are so relevant to information work, and massively underrated as well

Learning to recognise AI slop is the same as learning to recognise SEO’d out the wazoo drivel. And foundation model builders have an incentive to make their models produce less slop. “The web” in aggregate doesn’t have any incentives to stop SEO garbage.

It´s only a challenge because they´re badly conceived "tools" if you have to struggle to use them...

I agree with you - what are the best resources, in your opinion?

AI has an extremely jagged capabilities frontier, and just navigating that is hard

Yeah it's interesting how much of working with modern AI tools benefits from an education in the humanities! Not just English: art history, philosophy, human psychology

I know some CS programs are starting to literally teach AI-code gen and assume students are doing it. Similar to how they accept students will google answers. But the material will get harder and written tests will still be used to make sure kids can still understand concepts themselves.

Tricks like RAG have taken a bite out of the reliability problem but still have gaps - and users need to understand more of what's going on under the hood in order to judge if the answer is trustworthy I prefer RAG systems that are transparent about what they put in the context, but those are rare

Asking good questions is clearly mandatory for getting good answers, from an LLM or a human. I wonder if it would be valuable as an educational practice to have a topic to research, but the student’s work output is actually the questions that they should ask — rather than the answers to them.

prediction: we're going to start seeing students getting expelled for AI use

💯As soon as higher ed frames itself as the keeper of memorized knowledge, it becomes an anachronism (well, it largely already has)