Maciej Gryka

Maciej Gryka

@maciej.gryka.net

Software generalist with a particular interest in ML and AI. Soft spot for #ElixirLang and #Python. I enjoy challenges of early-stage startups. Ex-CTO, founded a couple companies, PhD in computer vision. More at https://maciej.gryka.net/

410 followers212 following168 posts

View full thread

...we see most often at distil labs. Most of our deployments are heavily prefill-dominated, so our bottleneck is compute, not memory. Overall I like the approach of taking something well-understood (like queuing theory) and describing how to apply it to some recent problem.

...it takes "with KV Cache Memory Constraints" very seriously! Basically, the paper only looks at decode-bound scenarios, where the prefill/decode token count ratio is at most 2:1. That's kind of reasonable in general, but unfortunately for me it doesn't help much with the kinds of setups...

I've recently been looking at multi-LoRA serving setups, it's a cool idea. Basically, if you have multiple adapters on top of the same model, you can serve them form the same instance, sharing all of the base-model weights. Conceptually that's pretty simple, but there's a lot of depth in...

arxiv.orgarxiv.org
View full thread

...different adapters. There is a catch, though, especially if you're interested in small models and low latency. Even with all the optimizations, the whole thing still adds about 2ms per token. This is not a lot, but...

...if you're using low-parameter-count models and care about latency, this relatively small overhead can still make a significant difference. It'd be a little surprising if no progress was made here since 2023 (when the Punica paper was released). Any pointers?

I've recently been looking at multi-LoRA serving setups, it's a cool idea. Basically, if you have multiple adapters on top of the same model, you can serve them form the same instance, sharing all of the base-model weights. Conceptually that's pretty simple, but there's a lot of depth in...

arxiv.orgarxiv.org

Maciej Gryka reposted

You don’t win an argument about definitions. You agree on definitions so that you can even have an argument

Maciej Gryka reposted

Replying to @skamille.themanagerswrath.com

we are quick to credit weird quirks for companies' success when really their success is usually some form of inventing a money printer and monopolizing it effectively and thus being allowed to indulge in weird quirks

Maciej Gryka reposted

Paging @masnick.com to the Streisand effect courtesy phone. An “engine problem“ is a complete non-issue and it was corrected in 24h! But Go2Sky sued @avherald.com! Now I’m reading about their 2016 incident they wanted to hide, and I’d never, ever fly an airline with this approach to transparency.

This post is unavailable.

Maciej Gryka reposted

Well I'm sad to say it's my last week at Rainforest QA - a bunch of us got let go so if anyone is looking for incredible remote frontend (React) or backend engineers (Ruby / Rails / Python) DM me and I'll be happy to introduce you.

Woah I can really feel my decision-making ability drain in a new way now in the evenings. When most general-coding trivia is taken away, constantly facing actually challenging decisions is something else.

So it turns out, if you've got an agentic workflow build out of specific subtasks, we can probably optimize it for you and drop your LLM bill by 10x. Small, distilled models often match and sometimes outperform much larger and more expensive models on well-defined tasks.

When my agent is doing stuff and I accidentally interrupt it by pressing Esc with the terminal focused, I feel the need to apologize. Which is even dumber than usual, it doesn't perceive time!

Maciej Gryka reposted

This is excellent. I keep telling folks that the superpower of SOTA models is not that they can just generate endless amounts of code in whatever language you want, but also that they can act as a personalized engineering/coding teacher that can help you explain concepts in a way that YOU get.

Cat Hicks@grimalkina.bsky.social

Key to efficient learning is realizing how we ACTUALLY learn, not just what FEELS like learning. I wrote a Claude Skill for some friends to help them think about this and they've liked it -- see Principles for some directions you could explore github.com/DrCatHicks/l...

Maciej Gryka reposted

Replying to @pfrazee.com

If you don’t see the utility personally, I can get that. If I wasn’t coding I probably wouldn’t either. If you’re worried about its downsides, join the club. A lot of us are too. But no matter what your take is, take it seriously. This shit isn’t crypto.

We recently ran a project to help Knowunity cut their LLM bill by half. It’s pretty cool to beat a frontier LLM (Gemini) on all dimensions, which matter: cost, speed, and accuracy by trading off generalization, which is unnecessary here. More details www.distillabs.ai/blog/how-kno...

distillabs.aiHow Knowunity used distil labs to cut their LLM bill by 50%We show how Knowunity use the distil labs platform to fine-tune and deploy their own models, significantly reducing their LLM costs.

Also! Turns out performance testing these deployments against high traffic scenarios is not super easy… unless you whip up a quick #ElixirLang script

Elixir's operational simplicity (having e.g. a single repo & piece of infra, compared to more moving parts in other stacks) might either be a very big advantage or not matter at all in the new coding-agent world. I'm like 75% on the side of it turning out to be important, but who knows!

Fun demo we shipped this week: AI slop detector running locally in a browser extension! github.com/distil-labs/... The utility of the tool itself is questionable :) But it was super fun to build! I was hoping to get something working in a browser for a while, and this shows how :)

github.comGitHub - distil-labs/distil-ai-slop-detector: Detect AI-generated text locally in your browserDetect AI-generated text locally in your browser. Contribute to distil-labs/distil-ai-slop-detector development by creating an account on GitHub.

Maciej Gryka reposted

Replying to @apenwarr.ca

And here you go! Definitely doesn't fix the entire state of LLM protocols... yet. But we're on our way. bsky.app/profile/apen...

apenwarr@apenwarr.ca

Coding agents crossed a line this year: they’re useful enough that people are going to use them at work, whether the tooling is ready or not. So... here's some tooling! tailscale.com/blog/apertur...

tailscale figuring out how to make something sensible out of the current state of all the llm apis wen

apenwarr@apenwarr.ca

I have looked at the actual byte streams LLMs use to communicate with clients and GOOD LORD WHAT THE HECK that is not how any of these protocols are supposed to be used

Playing with @fly.io 's sprites by creating Claude skills to set up sprites for LiveView development. Adding Tidewave was super easy, I guess I should try LiveDebugger next. Many disposable Phoenix apps, here we come!