Post

A couple comments about what I’ve been learning about algo engineering, but first off: if you’re interested in helping us improve Discover and all of our algorithmic systems, we are hiring algo engineers jobs.gem.com/bluesky/am9i...

My background isn’t ML so I recently had to start training in my off time. It’s actually been very fun; wish I had done this 5 years ago. But that’s my opening caveat- I’m still relatively junior on this subject matter

The way Discover works is by sampling from multiple different “Candidate Groups (CGs).” This can be things like followed users, liked by followed users, social clusters, and topics you’ve voiced interest in.

These get put through a predictive ranker, which is a large statistical model that tries to guess whether the viewer would like a post. Model #1 for us to train.

The CGs each work differently. The topical ones require classification. Model #2 for us to train. This also requires a good embedding model, which I’m pretty sure (?) we don’t train or fine tune ourselves but requires a lot of experimentation.

The social cluster CGs don’t necessarily need ML, though I’m sure there are models which can generate them. My recent posts about neo4j have been my side work to add more graph analysis to our stack while the algo team focuses on deep learning systems

So it’s a lot of interlocking models - plus a good data pipeline and evaluation system - that all have to combine. Even though I also wish that the progress was faster, selfishly I have enjoyed learning how it’s done and I intend to keep sharing what we learn.

One of the things bugging Dan is that “show less” isn’t having the effect it should. I’ll have to ask the team about it this week but the answer is probably time prioritization with limited resources. Which is why I’m excited we’re hiring for this role, and that I can give it more attention.

I’ll tack on spare thoughts as I have them Topic classification for posts has been very tough because they’re so short. I think Ian has found some promising papers that look at the entirety of an account to essentially act as hints. Manual labels by users (aka tagging) could also help

For what it's worth, I've found standard LDA (implemented in mallet/tomotopy) works quite well on Tweets, despite their length. Then perhaps apply a very small LM to transform the topic words / top documents to human-readable labels?

+1 for this. Depending on size of dataset and other requirements, I'd personally also try BERTopic for the first step

I typically get worse results with BERTopic (qualitatively and empirically), but I think in some cases it can be useful

Hm, I have gotten both Mallet/Tomotopy to work well (default settings), although maxing out in the 100k range. In my case, I didn't have mixed languages, but I suppose I'd do langid first there? If I do work with document embeddings, then I just do so directly and skip BERTopic (w/ k-means/medoids)

Mixed language data is common on, e.g., Chinese Twitter which we found to be very diverse. Since topics are distributions over tokens and a single doc is usually just one language, the only way I see to make LDA work is by translating Tweets?

Yeah, in that case distributed embeddings make a lot of sense (although I remain skeptical of the UMAP+DBSCAN approach from BERTopic, see aclanthology.org/2022.naacl-m...) This reminds me, we'll have a paper up on arXiv tomorrow that adopts a simple technique to improve multilingual clustering!

Yeah, that makes a lot of sense. I think of BERTopic as a convenient, quick way to try that on your own data, which is, I think, another reason why less techy people like to use it.

Yeah, I very much get it. At this point it's just my bugbear---I'm on a couple papers showing it's worse empirically (see below), and several topic modeling people I know also have found the same (qualitatively) aclanthology.org/2024.naacl-l... arxiv.org/pdf/2507.00828 [ACL 2025]