Post

dame
@dame.is

i assume that the vast majority of consumer AI usage will be entirely local on-device in 5 years (if not sooner)

4 reposts83 likes

Dan
@handackett.bsky.social

Is this supposing that small models become "good enough"? I'm skeptical that this will be the case for big models, there are huge economies of scale and the oversaturated demand for relevant hardware isn't going anywhere for at least a few years

dame
@dame.is

smaller models already are good enough for so many everyday consumer needs, we’re getting the first taste of this with iOS 27

Dan
@handackett.bsky.social

I could see that, technically adept users who notice the benefits of extra firepower might be in a minority versus users for whom on-device privacy improvements are more important

Dan
@handackett.bsky.social

I think it might matter how much everyday use cases grow to be more long-horizon agentic, like if some non-technical user starts using post-Siri and start offloading more and more complex life-wrangling tasks on it as they get used to it, are they eventually going to run into walls?

Ezra🦋
@ezraboeth.com

I don't think this happens without a market crash or political action, but I hope it does! I'd specifically love to see a ban on cloud AI so there's a ton more pressure on engineering to figure out how to make on-device AI work better. there's no reason it can't work once devices are powerful enough

Ezra🦋
@ezraboeth.com

currently, the nature of GPU/CPU architectures mean it's most efficient to serve inference for a large numbers of users on a single cluster of machines, rather than using a single device like a macbook. that could change with enough time and money though. Apple has made strides in this area.

Ezra🦋
@ezraboeth.com

I just want to see us trying to push more and more intelligence into tinier and tinier boxes, rather than trying to build the biggest boxes we can and to hell with efficiency

Ezra🦋
@ezraboeth.com

bsky.app/profile/ezra...

Ezra🦋@ezraboeth.com

my preferred way of solving the AI craze is federally banning AI *in the cloud* (over a certain amount of resources in one location). AI is still fine but it has to be on end-user devices. only exception is scientific research approved by or funded by government in some way.

asura
@asura.dev

This was my prediction a few years ago and included stuff like the Huggingface SLM initiatives and smolagents I wanted to share compute on mesh llm And then I got yelled at, called a sovereign citizen, told it was dangerous, and banned from a discord so... here we are

hailey
@hailey.at

ios shortcuts with natural language is a great entry point and its only going to get more interesting from there

dame
@dame.is

have you seen/tried the bespoke safari extension creation with natural language? super impressive from my early testing

hailey
@hailey.at

haven't had much time recently unfortunately for stuff outside of work, but im going to have to start digging soon because a lot of that is now relevant to work so...

Joey
@joeyrockwood.fyi

Agreed. I also see appliances without complicated interfaces that allow you to operate via natural language. Oven that can do all the options just by telling it to. Instead of clicking many buttons to navigate to the specific type of cooking etc (as an example) Should be small enough to fit easy

dame
@dame.is

i would even go as far as to say that most consumer AI usage will be private, local AI models within 12-18 months

dame
@dame.is

for this i’d probably define it as usage of features/tools that primarily are AI enabled or more directly an AI “action” but i think this will even be true of “agents” or conversational AI (ChatGPT style) within that timeframe

dame
@dame.is

i assume this is also what Apple believes about consumer AI and is why they are approaching AI the way they are it’s also why Sam Altman is desperately trying to develop an AI hardware device cause he understands the iPhone *is* the AI hardware device so many people will be using instead of ChatGPT

emily s.
@emily.news

i think your timeline might be a little tight but i agree with you a lot on this. Apple is uniquely positioned to make this happen because of their fundamental hardware design

Citizen Platano 🇵🇷
@daniloc.xyz

the same week I got my 128 GB Mac Studio, Nvidia dropped nemotron 3 nano and I got 100tok/sec on it now you’re never gonna confuse its outputs for Fable but it was so shockingly competent for many tasks that I completely revised my near term expectations exactly as you have

daniel fosco
@handle.invalid

oh yeah, token spend for anything other than enterprise and software development will soon trend towards 0 the hardware limitations are temporary, models and harnesses will also be more and more optimized for it

Asa ⭔
@asa.engineer

I disagree; with current technology this would mean constantly thrashing RAM, the device getting hot to the touch, and waiting several seconds for normal activities. all fine if you know what you’re doing, but the ambient, always-aware, and useful AI will need a few more step changes in efficiency.

jake
@yetanotheruseless.com

Not sure about that. The most powerful AI will always be in the GPU farms, so I guess it depends how you measure it? Maybe the average user’s experience of it will be via local inference, yeah. I could see that. But by FLOPs or by num_requests or by num_tokens? Might still be cloud.

dame
@dame.is

The vast majority of people will never be using frontier models or relying upon super powerful cloud models. The average daily person is probably gonna be able to do 99% of what they need on device

jake
@yetanotheruseless.com

Ok I guess since you said “consumer”, maybe I agree. But people are lazy and like convenience, and BigTech likes a captive audience… so you never know!

dame
@dame.is

definitely not! i think of engineering as B2B, enterprise, etc i think many consumers will be coding without realizing they are coding, and much of that will be local — iOS 28 brings coding without coding so anyone can make bespoke browser extensions, automations, shortcuts with natural language