Eugene Yan

I've been nerdsniped by the idea of Semantic IDs. Here's the result of my training runs: • RQ-VAE to compress item embeddings into tokens • SASRec to predict the next item (i.e., 4-tokens) exactly • Qwen3-8B that can return recs and natural language! eugeneyan.com/writing/sema...

demo of the LLM-recommender hybrid returning both semantic IDs & english, and: • steering recs via natural language • explaining the recommendation • naming the bundle of recommendations • multi-turn conversation to get recs watch till the end for the bloopers lol www.youtube.com/watch?v=_0n4...

I've been nerdsniped by the idea of Semantic IDs. Here's the result of my training runs: • RQ-VAE to compress item embeddings into tokens • SASRec to predict the next item (i.e., 4-tokens) exactly • Qwen3-8B that can return recs and natural language! eugeneyan.com/writing/sema...

The best leaders I’ve worked with operate with perma-urgency. They act like early founders, mindful of existential threats. And they can balance speed, sustainability, and repay tech debt. Ultimately, customers love it and teams thrive when we ship fast to deliver delight.

Had a fun couple of hours this weekend with Codex & Windsurf • Migrated off deprecated jekyll-algolia to official sdk (better indexing) • Added recommendations + relevance scores to each post • Improved site responsiveness; fixed dark mode flicker • Marie Kondo-ed unused files & dead code

Previously, these tasks weren't worth the effort but now they can be done in hours. What an amazing time to build and play =D

To better understand MCPs and agentic workflows, I built news-agents to generate a daily news recap. The main agent spawns sub-agents, assigning them news feeds to parse and summarize, and then generates a final overall summary plus analysis. eugeneyan.com/writing/news...

Here's the code for the mcp-server (src), prompts (context), and generated summaries from May 4th (summaries). github.com/eugeneyan/ne...

To better understand MCPs and agentic workflows, I built news-agents to generate a daily news recap. The main agent spawns sub-agents, assigning them news feeds to parse and summarize, and then generates a final overall summary plus analysis. eugeneyan.com/writing/news...

Interesting paper from Google that challenges a core assumption in translation evaluation—a single metric can measure both accuracy & naturalness. They found that the best systems had neural metrics that did not correlate with human preferences. arxiv.org/abs/2503.24013

@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts: • Don't skip error analysis • Don't skip looking at your data • Don't gatekeep who can write prompts • Don't let zero users be a roadblock • Don't be blindsided by criteria drift

Great example of generate -> validate loop + error analysis > "the most effective route to improve outcomes was brute force: retry steps until they passed or reached a limit. We give the validation errors ... to the LLM and built a loop runner"

> "1. Run failing files to find common issues. 2. Select a sample of files with a common issue. 3. Update prompts and scripts to address that issue. 4. Re-run against the sample of files to validate fix. 5. Repeat by running against remaining files. ...

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

• MultiDoc2Dial: Modeling dialogues grounded in multiple documents. Evaluates ability to integrate info over multiple docs • Frustratingly Hard Evidence Retrieval for QA Over Books: Reframed NarrativeQA as open-domain task where book text must be retrieved

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

• Qasper: Similar to NarrativeQA, but with academic documents that are 5-10k tokens, and includes evaluation of answer spans • LongBench: Average of 6.7k words across fiction and technical docs • LongBench v2: Extension of LongBench, but evals are MCQ only

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

3. Potential challenges • Open-ended questions may have no single correct answer, making reference-based evals trickly. For example: What is the theme of this novel? • Questions should be representative of prod traffic, with a mix of factual, inferential, summarization, definitional questions

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

• Evaluate separately: Faithfulness = binary label -> LLM-evaluator; Helpfulness = pairwise comparisons -> reward model

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

1. Two metrics that come to mind • Faithfulness: Grounding of answers in document's content. Not to be confused with correctness—an answer can be correct (based on updated information) but not faithful to the document. Sub-metric: Precision of citations

Can't wait for when I can vibe code a production recommender system. Until then, here's some system designs: • Retrieval vs. Ranking: eugeneyan.com/writing/syst... • Real-time retrieval: eugeneyan.com/writing/real... • Personalization: eugeneyan.com/writing/patt...