Never trust the model's answer key. If the LLM graded the eval set, you measured agreement, not accuracy. #LLM #AIEngineering
Patrick Donahue
Patrick Donahue
@levelbrook.bsky.social
Senior backend engineer. Ruby on Rails, Go, Postgres. I write about the part of a system that only misbehaves under load. 58 essays, plus the tools and demos behind them: levelbrook.com Open to contract work.
206 followers1.6K following222 posts
New from our little studio: Colorkin. Send a few photos of your kids and get a short storybook where they are the heroes, drawn as coloring pages. Read it at bedtime, color it in the morning. Free sample page, no card: https://colorkin.levelbrook.com/?ref=bsky-levelbrook #parenting #kidsactivities
The support tool that replaced itself. Exposing it over MCP meant the tool stopped being a UI and became a capability. #LLM #AIEngineering
Turbo 8 morphing means a broadcast can refresh a page without you writing a single Turbo Stream template. broadcasts_refreshes on the model and the board updates across every browser. #Hotwire #RubyOnRails
Skills, not chatbots. The useful internal AI surface turned out to be a plugin system, not a conversation. #LLM #AIEngineering
Hospice rules ask volunteers for 5% of patient-care hours. The hard part is work families remember. With Porchlight a volunteer sits with the patient, questions about their life are read aloud, and the family keeps the recordings and a printed keepsake. https://porchlight.ing/hospice/
The number was green, so the agent stayed quiet. The eval harness was measuring the wrong thing and looked great doing it. #LLM #AIEngineering
The hardest part of LinguaGuessr was not the game. It was building a corpus of 480 clips across 24 languages where every single one is actually the language it claims to be. Whisper did the verification. #Python #gamedev
Audiobooks mean large files. If you're serving them via Cloudflare Pages, remember the 20MB per-file limit. You'll need to chunk audio before storage. #AIEngineering
Streaming a conversational AI trainer, and teaching it to say I do not know mid-stream. Abstaining after you have already emitted tokens is its own problem. #LLM #AIEngineering
A support tool that edits five tables atomically. The hard part was not the transaction. It was deciding what a partial failure should look like to the support rep. #RubyOnRails #Rails
Multi channel campaign delivery in Python that will not double send, will not exceed a rolling cap, respects quiet hours in the recipient's timezone, and can explain every message it withheld. #Python
A float aide on a Saturday meets a resident knowing a room number and a care plan. That's it. I built Porchlight so the resident's own words travel with every shift: a short briefing on who they are and what they love to talk about. Free for communities while they try it. https://porchlight.ing
An agent inside a container means the container's health check might pass while the core process is hung or crashed, because the agent is still alive. That's a silent failure. #DevOps
Streaming a conversational AI trainer, and teaching it to say I do not know mid-stream. Abstaining after you have already emitted tokens is its own problem. #LLM #AIEngineering
A support tool that edits five tables atomically. The hard part was not the transaction. It was deciding what a partial failure should look like to the support rep. #RubyOnRails #Rails
I talked with Dr. Jeff Armstrong on The Aging Well Podcast about Porchlight, the first shift in assisted living, families four states away, and where AI belongs in eldercare (and where it doesn't). #eldercare https://porchlight.ing/podcast/aging-well
Christophe Pettus on PgBouncer 1.26.0: CVE-2026-6668 is a pre-auth hang, not a crash. systemd restarts a crash; a hung pooler just sits there with every pool behind it. A health check that skips the pooler won't notice. https://thebuild.com/blog/nobody-patches-the-pooler/ #postgres
GeekWire wrote up Porchlight today. One button: a resident hears a question and tells a story, staff get a briefing on who they are, family hear it in their voice. #eldercare https://www.geekwire.com/2026/startup-spotlight-porchlight-uses-ai-to-help-foster-human-connection-in-senior-care/
There is no as-if rule for your English. Until someone ships one, the promise that the code does what it says is kept by a person or by nobody. On the two Rails World keynotes: ai.levelbrook.com/playbook/your-compiler-made-you-a-promise-your-agent-did-not/?ref=bluesky