Post

This really sounds so great. I still want to know what "looking at" means here. Maybe @tornorbye.bsky.social can answer? Is it actually getting screenshots or a text based view tree? Is it polling for this info or is there something pushing to Gemini that the screen has updated?

The core capability comes from grabbing screenshots and using Gemini's multimodal capabilities.

I was in the room during the demo made by @tornorbye.bsky.social and even if technically, it is very impressible, I've still so many questions. Mainly: how do we make sure that the results are reproducible from one run to another, using the same model or a new one? And how to debug tests run by AI?

There's no guarantee that it's 100% reproducible. But I think LLMs do a pretty decent (and consistent!) job with "easy" questions. Hopefully the journey commands are generally easy to understand. Of course if you write a ambiguous instruction ("make sure the UI is pretty"), that's less predictable.

To debug these things, we're working on various proposals; you obviously won't be able to debug inside the LLM, but we plan to have a way to capture the UI actions so you can replay those, under a debugger.