Nikhil Thorat

Nikhil Thorat

@nst.bsky.social

AI quality at Databricks. Past: Co-founder of Lilac AI, acquired by Databricks. Co-created TensorFlow.js and Know Your Data. Google Brain // PAIR // Responsible AI. nsthorat@ over there

429 followers44 following27 posts

View full thread

I’m not sure if y’all have used LLMs in serious software engineering, but we’re not even at milestone 0 robustly. LLMs still completely screw up even when slightly complex technical constraints are well formed, in extremely subtle and frustrating ways.

I hate to be that guy, but I’ll go out on a limb and say this prediction is complete bullshit. A more realistic take is: LLMs will aid humans at all of these stages, which makes them insanely valuable tools. And that’s why I’m excited about them. But replace is a bit of a grifter take, IMO.

View full thread

Milestone 2: an AI translation layer of product requirements from a human to large scale technical requirements. Milestone 3: AI coming up with product requirements altogether.

For humans, these are all pretty intermingled, and you need software engineers at all of these stages. I think it’s safe to assume you have to get to milestone 3 to “fully replace all software engineers”

Milestone 0: AI can translate very focused technical requirements into software. Milestone 1: AI can translate project-scale technical requirements into software. This means understanding how many large technical components fit together.

We just released the Databricks synthetic evals SDK! 🎉 We’ve found that synthesizing evals is a great way to hill climb your AI system before you’re able to get labels from domain experts. We’ve also recently release a new diff UI that lets you both qualitatively and quantitatively view results!

This query sold me on Perplexity Trying to understand if PCM will block sunlight for plants for a friends greenhouse. He wanted a fully natural way of keeping it warm in the winter. The answer was deep in academic papers that aren’t normally summarized by other search engines.

Something about the blue color on this app makes it feel so much friendlier than X. Not to mention I now get overtly racist anti-indian memes now over there…

My twitter notifications are mostly spam / bots that have bios trying to convince me to click a link to go on a date with them. Let’s try to make this app work.

Bluesky nailed the “omg I want an invite” even though features are identical. I don’t like when someone yells “8$” and makes me feel bad about it. Carrots not sticks!

You are up to date.