François Fleuret

26 Nov 2024

My deep learning course at the University of Geneva is available on-line. 1000+ slides, ~20h of screen-casts. Full of examples in PyTorch. fleuret.org/dlc/ And my "Little Book of Deep Learning" is available as a phone-formatted pdf (nearing 700k downloads!) fleuret.org/lbdl/

10 Sept 2025

I asked "on the other platform" what were the most important improvements to the original 2017 transformer. That was quite popular and here is a synthesis of the responses:

View full thread

- Multi-token prediction: sums the training over multiple future tokens, possibly with additional readout heads. - FlashAttention: computes the attention on the fly, avoiding a memory footprint O(T^2) (+ optimizes very carefully for the GPU!)

- Ring Attention: takes advantage of multi-node hardware to scale the computation according to the sequence length - Speculative decoding: a cheaper model generates tokens, and a rejection process corrects this generation to march the full-model distribution.

I asked "on the other platform" what were the most important improvements to the original 2017 transformer. That was quite popular and here is a synthesis of the responses:

View full thread

- RoPE (Rotary Positional Embedding): makes the attention depend only on the relative Q/K positions - MoE (Mixture of Experts): The FFN block is implemented with multiple MLPs and a gating mechanism selects which ones process each token.

- Warmup: very short ramping-up of the learning rate, starting from 0 - Cosine schedule: the learning rate varies less at the beginning and end of the schedule - AdamW: decouples weight includes decay from Adam

I asked "on the other platform" what were the most important improvements to the original 2017 transformer. That was quite popular and here is a synthesis of the responses:

- Prenorm: normalization in the residual blocks before the attention operation and the FFN respectively - GQA (Group Query Attention): more Q than (K, V)

- RMSNorm instead of Layernorm: normalize only the scaling - MLA (Multi-head Latent Attention): stores a low-rank projection of the attention block input and compute the KV from it - SwiGLU: non-linearity for the FFN block with per-component gating

1 Apr 2025

"You are in Paris, enjoy the city, stop obsessing with AI" Paris:

28 Feb 2025

Maybe the wall was the friends we made during that journey Ted.

Ted Underwood@tedunderwood.com

The year is 2435. Human beings — now sentient spheres of glowing gas — finally understand why matter exists. Our knowledge of the world is complete, and can go no farther! One the telepresence screen, a simulation of a 21c internet pundit pops up: "Told you deep learning was hitting a wall!"

François Fleuret reposted

Ramon

@noctrog.bsky.social

14 Feb 2025

What is the true depth of an LLM? Together with @danielepal.bsky.social , @matpagliardini.bsky.social, M. Jaggi and @francois.fleuret.org we show that LLMs have a smaller effective depth that can be exploited to increase inference speeds on multi-GPU settings! arxiv.org/abs/2502.02790 (1/N)

rts.ch19h30 - Play RTSPlay RTS vous permet de visionner ou d'écouter de nombreuses émissions tv ou radio, quand et aussi souvent que vous le souhaitez.

11 Feb 2025

J'étais l'invité du journal de 19h30 sur la @radiotelesuisse.bsky.social ce soir pour parler d'Intelligence Artificielle. www.rts.ch/play/tv/19h3...

6 Feb 2025

It is hard to overstate how cool and powerful is flex attention. @chhillee.bsky.social pytorch.org/blog/flexatten… TL;DR: it is an implementation of the attention operator in pytorch that allows in particular to efficiently "carve" the attention matrix. 1/3

pytorch.orgpytorch.org

6 Feb 2025

It does this by generating an optimized cuda kernel on the fly. So it's cool for causal masks, but it also allows an amazing trick to deal with batches of sequences of various lengths *without padding*! 2/3

6 Feb 2025

To do so, you concatenate all the sequences to make a batch of a single sequence, and carve the attention matrix into a block-diagonal one (possibly with causal structure in each block) so that sequences cannot look at each other. Magic! 3/3

François Fleuret reposted

Kajetan Dymkiewicz

@kdymkiewicz.bsky.social

10 Jan 2025

Finally got this beautiful piece from @francois.fleuret.org

31 Dec 2024

Happy new year you all! 2025 is certainly full of promise.

25 Dec 2024

Happy Christmas you all!

22 Dec 2024

Whatever you say about the whole field of AI: It's not boring.

rectangleapp.comRectangleMove and resize windows in macOS using keyboard shortcuts or snap areas. The official page for Rectangle.

20 Dec 2024

Some tools that keep me sane on Mac: rectangleapp.com karabiner-elements.pqrs.org

20 Dec 2024

It's Friday!

19 Dec 2024

19 Dec 2024

Oh boy, GTA 6 has to be good. And Half Life 3.

luokai@luok.ai

Odyssey has developed a generative world model called Explorer using Gaussian Splats technology, with the support of Pixar co-founder Ed Catmull.

youtube.comAlignment faking in large language modelsYouTube video by Anthropic

19 Dec 2024

This is very great.

Sam Bowman@sleepinyourhat.bsky.social

Alongside our paper, we also recorded a roundtable video featuring four of the paper’s authors discussing the results and their implications in detail:

You young post-2012 researchers, you realise that us the old people are *amazed* by what we witness?

This post is unavailable.

Ah, the deleted message was not expected. This was some impressive generated video. Unclear what the reason for delete was.

Something that annoys me much is when people (generally in political / "societal" discussions) equate the validity of a statement with the validity of *what "people" may understand* and generally then fix a true statement they consider false with a false statement they consider true. 1/2

The typical example would be "you get overweight if you eat more calories than you spend". In 90% of discussions you will face someone saying "that's not true" because they think this implies one ignores the complex societal / psychological / medical reasons why this would happen. 2/2

François Fleuret reposted

Bálint Máté

@balintmate.bsky.social

hello bluesky! we have a new preprint on solvation free energies: tl;dr: We define an interpolating density by its sampling process, and learn the corresponding equilibrium potential with score matching. arxiv.org/abs/2410.15815 with @francois.fleuret.org and @tbereau.bsky.social (1/n)

"The expletive error rate is the number of times you call your algorithm a 'dense mf' when looking at one run"

General structure of a paper: - general ideas - general case - general case - general case - what we actually do how it should be: - what we actually do - why we think it's great as one method of a general class - how we got there - how we got there - how we got there

Today's mood "wow GPTs are dumb"

youtube.comWanderers - a short film by Erik Wernquist [Official Version]YouTube video by Erik Wernquist - Digital Artist

One of the goosebumpsiest short movie ever (where this quote comes from). We will have this sound snippet played at the ceremony when we will leave earth for the stars. www.youtube.com/watch?v=YH3c...

François Fleuret@francois.fleuret.org

"We invest far-off places with a certain romance. This appeal, I suspect, has been meticulously crafted by natural selection as an essential element in our survival."

"We invest far-off places with a certain romance. This appeal, I suspect, has been meticulously crafted by natural selection as an essential element in our survival."

The closest I'll ever be to have to build my own hut.

Again I control myself because "f*cking" before hut would have been funnier. But we want to be classy.

m.youtube.comAlphaGo - The Movie | Full award-winning documentaryYouTube video by Google DeepMind

16 Dec 2024

If you have not seen it, you really should. m.youtube.com/watch?v=WXuK...

15 Dec 2024

🎶 This is the end 🎶 @neuripsconf.bsky.social

François Fleuret reposted

syhw.bsky.social

@syhw.bsky.social

docs.google.com[NeurIPS HackerCup 2024] Grounding LLMs in Code ExecutionGrounding LLMs in Code Execution Gabriel Synnaeve, Meta, FAIR

Just gave a talk on "Grounding LLMs in Code Execution" at the NeurIPS Hacker-Cup AI Competition, here are the slides docs.google.com/presentation...

15 Dec 2024

That The shining vibe.

My political weight is obviously immense since there is coffee available all the time at @neuripsconf.bsky.social this year. For next year: can we have at e.g. 10 places in the center some huge colorful discs with numbers 1...10 in them to serve as convenient meeting points?

Make me the "Stuff to make things convenient" Chair. I can foresee my first clash when I'll ask for one armchair per attendee.

8th gen

François Fleuret@francois.fleuret.org

The topologists seem to be taking over. Large pictures + a README about this experiment at fleuret.org/public/culture