Lucas Beyer (bl16)

26 Jan 2025

I noticed that I'm not using bsky much anymore. Not sure why, vibes. Anyways, someone noticing that DeepSeek refuses to answer *anything* about Xi Jinping, even the question whether he exists at all, triggered me writing a short snippet on safety fine-tuning: lb.eyer.be/s/safety-sft...

2 Jan 2025

First candidate for banger of the year appeared, only 2 days in:

Lucas Beyer (bl16) reposted

Nathan Lambert

@natolambert.bsky.social

20 Dec 2024

OpenAI skips o2, previews o3 scores, and they're truly crazy. Huge progress on the few benchmarks we think are truly hard today. Including ARC AGI. Rip to people who say any of "progress is done," "scale is done," or "llms cant reason" 2024 was awesome. I love my job.

18 Dec 2024

A post by @cloneofsimo on Twitter made me write up some lore about residuals, ResNets, and Transformers. And I couldn't resist sliding in the usual cautionary tale about small/mid-scale != large-scale. Blogpost: lb.eyer.be/s/residuals....

cloneofsimo.bsky.social@cloneofsimo.bsky.social

Sup cats

10 Dec 2024

Good morning Vancouver! Things are different here: this guy is alone, chonky, and not scared at all, I was more scared of him towards the end lol. Also look at that … industrialization

Lucas Beyer (bl16)@giffmana.ai

Good morning! On my way to NeurIPS, slightly sad to leave this beautiful place and my family for the week, but also excited to meet many new and old friends at NeurIPS!

youtube.com[M2L 2024] Transformers - Lucas BeyerYouTube video by Mediterranean Machine Learning (M2L) summer school

9 Dec 2024

Good morning! On my way to NeurIPS, slightly sad to leave this beautiful place and my family for the week, but also excited to meet many new and old friends at NeurIPS!

Lucas Beyer (bl16) reposted

Sai Kumar Dwivedi

@saidwivedi.in

8 Dec 2024

One of the best tutorials for understanding Transformers! 📽️ Watch here: www.youtube.com/watch?v=bMXq... Big thanks to @giffmana.ai for this excellent content! 🙌

Lucas Beyer (bl16) reposted

Ibrahim Alabdulmohsin

@ibomohsin.bsky.social

7 Dec 2024

Attending #NeurIPS2024? If you're interested in multimodal systems, building inclusive & culturally aware models, and how fractals relate to LLMs, we've 3 posters for you. I look forward to presenting them on behalf of our GDM team @ Zurich & collaborators. Details below (1/4)

Lucas Beyer (bl16) reposted

This aged well

The fourth nice thing we* have for you this week: PaliGemma 2. It’s also a perfect transition: this v2 was carried a lot more by @andreaspsteiner.bsky.social André and Michael than by us. Crazy new sota tasks! Interesting res vs LLM size study! Better OCR! Less hallucination!

Andreas Steiner@andreaspsteiner.bsky.social

🚀🚀PaliGemma 2 is our updated and improved PaliGemma release using the Gemma 2 models and providing new pre-trained checkpoints for the full cross product of {224px,448px,896px} resolutions and {3B,10B,28B} model sizes. 1/7

5 Dec 2024

*As mentioned: Andreas, @asusanopinto.bsky.social and @mtschannen.bsky.social did a heroic effort here and deserve all the credit for these models and the interesting report. I merely supported, advised, and handed things over. Keep a close eye on these guys!

Lucas Beyer (bl16) reposted

Marvin Schmitt

@marvin-schmitt.com

5 Dec 2024

OpenAI is coming to Switzerland, and the founding team of the new Zurich office is nothing short of stellar🇨🇭⭐️ What a fantastic win for the European AI landscape 🇪🇺 Congrats to @giffmana.ai @kolesnikov.ch @xzhai.bsky.social for that move, and to OpenAI for making these hires!

Lucas Beyer (bl16)@giffmana.ai

So, now that our move to OpenAI became public, @kolesnikov.ch @xzhai.bsky.social and I are drowning in notifications. I read everything, but may not reply. Excited about this new journey! 🚀 Quick FAQ thread...

@francois.fleuret.org hey, can you buy me a few of tomorrow's Le Temps, if the news about us is printed in it? I'll pay you in beers at neurips.

Alexander Kolesnikov@handle.invalid

Ok, it is yesterdays news already, but good night sleep is important. After 7 amazing years at Google Brain/DM, I am joining OpenAI. Together with @xzhai.bsky.social and @giffmana.ai, we will establish OpenAI Zurich office. Proud of our past work and looking forward to the future.

7. Why did you quit/change, is GDM toast? → No, GDM is still a fantastic place full of smart people. I really mean it, HR didn't and couldn't ask me to say this :)

8. Why not Anthro/Meta/xAI/...? → It's a complex, multi-faceted issue. Really though, all these places have something good going for them! (And it's something different for each.)

Alexander Kolesnikov@handle.invalid

5. Here's my CV, are you hiring? → Maybe. Keep your eyes open. No harm in asking of course, but I can't answer you yet.

6. What will you work on? → buddy, did you really think I'll tell you?

Alexander Kolesnikov@handle.invalid

3. What about big_vision?? → It's in great hands! @andreaspsteiner.bsky.social and @asusanopinto.bsky.social are continuing it. They've been core contributors for a while now, and really get the spirit!

4. JAX or PyTorch? → Well well well, that is the question, isn't it?

Alexander Kolesnikov@handle.invalid

1. Moving to SF? → Nope, opening office in Zürich. I said I'll never leave this city, I mean it. (Only my wife has more power.)

2. Will you stop publishing/OSS? → Dunno. I guess it's a good default assumption. That being said, the team at OAI has released some pretty banger papers and models over the last years, albeit sparsely, so never say never.

Lucas Beyer (bl16) reposted

Alexander Kolesnikov

@handle.invalid

Lucas Beyer (bl16) reposted

Dmytro Mishkin

@ducha-aiki.bsky.social

3 Dec 2024

JetFormer: An Autoregressive Generative Model of Raw Images and Text @mtschannen.bsky.social @asusanopinto.bsky.social @kolesnikov.ch tl;dr: VQGAN quality w/o VQGAN, but optimizing pixel tokens with NLL. Also some normalizing flow magic, which I have to read up arxiv.org/abs/2411.19722

3 Dec 2024

So this was the second cool thing that we* got this week. * this time the « we » is really just me, but hey, academic we 🙃

Lucas Beyer (bl16)@giffmana.ai

Our big_vision codebase is really good! And it's *the* reference for ViT, SigLIP, PaliGemma, JetFormer, ... including fine-tuning them. However, it's criminally undocumented. I tried using it outside Google to fine-tune PaliGemma and SigLIP on GPUs, and wrote a tutorial: lb.eyer.be/a/bv_tuto.html

It's relatively long, so just a few highlights: 1. It works really well on GPUs out of the box. 2. I made ready-to-go docker templates for runpod. 3. Fine-tuning is cheap!! A sweep of 72 runs cost me less than $1 4. PaliGemma and SigLIP nail the "line intersections" task with 128 examples:

Lucas Beyer (bl16) reposted

Alexander Kolesnikov

@handle.invalid

2 Dec 2024

The answer has just dropped: bsky.app/profile/kole...

Jia-Bin Huang@jbhuang0604.bsky.social

2021: Replace every CNN with a Transformer 2022: Replace every GAN with diffusion models 2023: Replace every NeRF with 3DGS 2024: Replace every diffusion model with Flow Matching 2025: ???

2 Dec 2024

The first of the cool things we* got this week! Typically, you'd train a VQ-VAE/GAN tokenizer first, and then use its tokens for your LLM/DiT/... But we all know eventually end-to-end wins over pipelines. With flow models, you can actually learn pixel-LLM-pixel end-to-end!

Michael Tschannen@mtschannen.bsky.social

Have you ever wondered how to train an autoregressive generative transformer on text and raw pixels, without a pretrained visual tokenizer (e.g. VQ-VAE)? We have been pondering this during summer and developed a new model: JetFormer 🌊🤖 arxiv.org/abs/2411.19722 A thread 👇 1/

2 Dec 2024

*we = our team Brain/GDM Zürich. However, I sadly wasn't directly involved this time, as I was busy with other things. This work was carried by my close colleagues @mtschannen.bsky.social @asusanopinto.bsky.social @kolesnikov.ch, and I like it a lot!

Some recent discussions made me write up a short read on how I think about doing computer vision research when there's clear potential for abuse. Alternative title: why I decided to stop working on tracking. Curious about other's thoughts on this. lb.eyer.be/s/cv-ethics....

24 Nov 2024

So how long do we give it before the bots and spam arrive here, and how well do we think @bsky.app is prepared? 1. L I N K I N B I O. 2. Crypto spam 3. If you’re not doing these 10 GenAI things, you’re falling behind 4. Wow. This changes everything. My guesses: 2 months, and not well.

29 Nov 2024

Aaaaand it's starting:

28 Nov 2024

hahahahhahaah holy cow, paligemma is good =)

BusyBrain⌨@w42.bsky.social

28 Nov 2024

segment cow not on the beach, but as you may know I like cows in "out of distribution" places and poses. huggingface.co/spaces/big-v...

quokkka.bsky.social@quokkka.bsky.social

I think that it's about implementation because the idea as @giffmana.bsky.social has noted was well-known and unsurprisingly claimed by the Father of AI. But the method, which is the implementation of the technology, convinced people to use it

Sorry, but I was strangely attracted... @5trange4ttractor.bsky.social =D

Here's a fun real-life prompt where 30min of Googling didn't really help me. The small/fast/mini chatbots all failed miserably The large ones work great (except Gemini). Also: 1. I really like C3.6 giving only answer and asking if explain 2. Wild that Chat understood the calls and added comments!

Here's Gemini latest, which is wrong in both variants, and also yaps way too much :(

Reminder that robots.txt is a gentleman’s agreement, not a legal document. The good actors all respect it, but there’s many bad actors out there on the internet.

Bluesky@bsky.app

Bluesky is an open and public social network, much like websites on the Internet itself. Websites can specify whether they consent to outside companies crawling their data with a robots.txt file, and we’re investigating a similar practice here.

No need to be a deep pocketed company, or HF. Scraping one million posts is not much. I did more than 14M just for fun on my single cheap desktop during PhD. Please don’t be intimidated too easily.

Jeremy Howard @howard.fm

I'm glad @hf.co is doing this. It brings down the barriers to allow more people to benefit from AI, rather than keeping it exclusively in the realm of deep pocketed giant companies. AI can help open the gates, to allow regular people to do things they couldn't do before. (Which can be threatening!)

At the same time, however: bsky.app/profile/bsky...

Bluesky@bsky.app

Brief update on our ongoing efforts to allow users to specify consent (or not) for AI training: 🧵

26 Nov 2024

A sunny winter morning near the lake in Zürich is probably one of the best mood enhancers available world-wide. I don't think I'll ever get tired of this.

25 Nov 2024

In the end I got emotional and couldn't bring myself to kill little `riesling`, who's remained loyal this whole decade. So I embarked on a journey to update Ubuntu 14.04 to 24.04. What an adventure! `failed to execute process '/usr/bin/ln'` almost killed me. But I made it, a decade of updates!

Lucas Beyer (bl16)@giffmana.ai

soooo, this marks the 10th anniversary of using digitalocean for my personal server. It's been running smoothly on 512MB RAM all that time. Now I'm going to splurge on a 1GB RAM, just so it feels like an upgrade! (Same price.)