Post

A couple comments about what I’ve been learning about algo engineering, but first off: if you’re interested in helping us improve Discover and all of our algorithmic systems, we are hiring algo engineers jobs.gem.com/bluesky/am9i...

My background isn’t ML so I recently had to start training in my off time. It’s actually been very fun; wish I had done this 5 years ago. But that’s my opening caveat- I’m still relatively junior on this subject matter

The way Discover works is by sampling from multiple different “Candidate Groups (CGs).” This can be things like followed users, liked by followed users, social clusters, and topics you’ve voiced interest in.

These get put through a predictive ranker, which is a large statistical model that tries to guess whether the viewer would like a post. Model #1 for us to train.

The CGs each work differently. The topical ones require classification. Model #2 for us to train. This also requires a good embedding model, which I’m pretty sure (?) we don’t train or fine tune ourselves but requires a lot of experimentation.

The social cluster CGs don’t necessarily need ML, though I’m sure there are models which can generate them. My recent posts about neo4j have been my side work to add more graph analysis to our stack while the algo team focuses on deep learning systems

So it’s a lot of interlocking models - plus a good data pipeline and evaluation system - that all have to combine. Even though I also wish that the progress was faster, selfishly I have enjoyed learning how it’s done and I intend to keep sharing what we learn.

One of the things bugging Dan is that “show less” isn’t having the effect it should. I’ll have to ask the team about it this week but the answer is probably time prioritization with limited resources. Which is why I’m excited we’re hiring for this role, and that I can give it more attention.

I’ll tack on spare thoughts as I have them Topic classification for posts has been very tough because they’re so short. I think Ian has found some promising papers that look at the entirety of an account to essentially act as hints. Manual labels by users (aka tagging) could also help