Timothy Keyes

Timothy Keyes

@timothykeyes.bsky.social

Data Scientist @ Stanford Health Care and perpetual MD/PhD student. 🏳️‍🌈 AI/ML for health, #rstats and #pydata

756 followers241 following39 posts

Timothy Keyes
@timothykeyes.bsky.social

my most recent hot take: after a year working inside my hospital’s IT system, i’m increasingly convinced that clinical informatics should have been a residency (like pathology or radiology), not a fellowship.

Timothy Keyes
@timothykeyes.bsky.social

as i prepare to go back to med school, i am haunted daily by the number of my med school classmates (who are now residents, fellows, or attendings) who have reached out to me to learn about my side-career in health tech because they want to explore alternatives to a full-time clinical career

Timothy Keyes
@timothykeyes.bsky.social

idk what it means that so many people I know to be thoughtful, compassionate, and brilliant physicians have told me that - while they love taking care of patients - the growing frustrations and exhaustion they experience in clinical life make the whole thing feel unsustainable in the long-term

Timothy Keyes
@timothykeyes.bsky.social

I feel like every paper I read about LLMs in health care takes for granted that the encouraging, but not necessarily transformative, results we’ve seen so far are going to scale upward as LLMs improve. But are we really safe to assume that LLMs will keep getting better? I’m not so sure.

Timothy Keyes reposted

Roxana Daneshjou, MD, PhD
@roxanadaneshjou.bsky.social

It’s finally out! We brought a multidisciplinary team of physicians, computer scientists, and engineers to red team LLMs for healthcare uses. And we have shared the dataset! www.nature.com/articles/s41...

nature.comRed teaming ChatGPT in medicine to yield real-world insights on model behavior - npj Digital Medicinenpj Digital Medicine - Red teaming ChatGPT in medicine to yield real-world insights on model behavior
Timothy Keyes
@timothykeyes.bsky.social

i’m working on a few really cool applications of LLMs in health right now (in the real world, in a deployment environment) and my most valuable asset by far in this work isn’t being the most technical person on the team (i’m not); it’s knowing just enough about medicine to ask the right questions

Timothy Keyes
@timothykeyes.bsky.social

disclaimer: I haven’t finished medical school yet (i will someday!) so this is mostly stolen valor, but even having a small amount of clinical training has imo helped me so much to understand small details that my technical colleagues don’t really notice or care about

Timothy Keyes
@timothykeyes.bsky.social

i have to respect how non-technical folks use spreadsheets - they give you these beautiful murals with all kinds of colors and spacing and intricate patterns. i must say it really breaks my heart to immediately flatten it all with read_csv

Timothy Keyes
@timothykeyes.bsky.social

i’ve been trying to understand why reporting some estimate of confidence intervals around performance metrics (or null hypothesis significance testing) is not more common in the machine learning/AI literature. i think this is changing, but there’s still a lot of weird (or missing) statistics ime

Timothy Keyes
@timothykeyes.bsky.social

(i appreciate the use of benchmarks and think they’re really valuable, but benchmarks must be drawn from some population of possible observations, right? so there will still be sampling error in any metrics computed using a benchmark, even if all models use the same benchmark)

Timothy Keyes
@timothykeyes.bsky.social

started my day writing a few (simple) statistical proofs for a data science project (to justify simplifying a calculation from something complicated to something simpler and equivalent) and wow it really is nice to dust off the old PhD and put it to use every once in a while

Timothy Keyes
@timothykeyes.bsky.social

i rounded on stanford’s palliative medicine service today, and it was such a powerful reminder that end-of-life care clinicians are some of the most kind and empathetic people in health care. truly in awe of their ability to bear witness to (and guide people through) such difficult moments

Timothy Keyes
@timothykeyes.bsky.social

if i could distill my time in the world of health ai down to a single thesis statement, i think it would be that imo everyone in health care (patients, health care workers, researchers, caregivers of all kinds) should have a very, very healthy dose of skepticism about every ai product they encounter

Timothy Keyes
@timothykeyes.bsky.social

i am not an ai doomer by any stretch, but building ai systems (big or small) that solve useful problems in health care is really hard! validation experiments are really hard (and sometimes really expensive) to do! very few ai products work exactly as advertised, and many don’t work at all!

Timothy Keyes
@timothykeyes.bsky.social

when someone tells you that they’ve solved an important problem using ai, that’s very exciting! but you should not take their word for it. if they don’t provide very transparent tools for monitoring/evaluating their ai product, it’s pretty safe to assume even *they* don’t know if it actually works

Timothy Keyes reposted

John Paul Helveston
@jhelvy.bsky.social

This isn't really an #rstats vs #python thing but rather a tidyverse vs pandas thing, and for data wrangling the tidyverse API is just so darn good it's hard to compete with, which is why there are ports of it to other languages, e.g. Tidier in #julia, siuba in python

Damien Janvrin@geddan3.bsky.social

A good example about why Pandas is not an elegant tool for data analysis and wrangling. #SundayProvocation #python #rstats www.sumsar.net/blog/pandas-...

sumsar.netWhy pandas feels clunky when coming from RFive years ago I started a new role and I suddenly found myself, a staunch R fan, having to code in Python on a daily basis. Working with data, most of my Python work involved using pandas, the …
Timothy Keyes
@timothykeyes.bsky.social

one of the guiding principles of my life as a scientist (inherited from a mentor who has since passed away) has been to resist the incentives that foster egoism, possessiveness, and competition over collaboration in science - and to instead lead with generosity, openness, and lifting others up.

Timothy Keyes
@timothykeyes.bsky.social

today, i’m struggling a bit with how rare it feels to see those values reciprocated. but i refuse to become bitter! imo scientists who only care about how much they publish in fancy journals have missed the point. community, thoughtfulness, being a dependable colleague - these are all more important

Timothy Keyes
@timothykeyes.bsky.social

i’ve been working on the ai team at stanford health care for about a year and the overwhelming take-home message for me so far is to be increasingly horrified at how aggressively ai vendors will market their products without any apparent obligation to demonstrate that they actually work

Timothy Keyes
@timothykeyes.bsky.social

i am lucky to be on a strong team within a health system very committed to deploying ai responsibly, but i worry about smaller orgs who are not as well-resourced to sniff out the snake oil rampant in this field. i worry that they’ll take vendors’ claims at face value when they really shouldn’t

Timothy Keyes
@timothykeyes.bsky.social

i’ve been off the internet (ie tw*tter) for so long that i actually don’t remember how to write in pithy short-form anymore. all of my “terminally online” references are from 2022. i have 0 memes saved in my camera roll. i have a phd now like the vibes have shifted bro what do i do

You are up to date.