Post

Saloni
@scientificdiscovery.dev

yeah I mean I agree with all the points Julia makes, but I thought the whole piece was very poorly argued, and the specific things I agreed with didn't support the argument that was being made. The screenshot was particularly bad but IMO also succinctly captured what was wrong with the piece

Ryan Hisner
@ryanhisner.bsky.social

Is the problem in assuming that including more covariates are always better? Or in asserting that observational evidence + theoretical considerations are often the best option? I'm too dull to tell.

Seungju Kim (he/him)
@follhim.net

Im sure there are many reasons why this is problematic, at least in clinical psychology, it’s a measurement problem for me. Are there others (more consequential) that I am missing?

Seungju Kim (he/him)
@follhim.net

More covariates do not equal enhanced partial effects. It’s good in theory, but not in practice. Most items or multidimensional variables are polluted with measurement error.

Seungju Kim (he/him)
@follhim.net

So when you start adding covariates to partial out shared variance, you actually remove meaningful variable, and leave a greater share of ME and small bits of the original construct.

Seungju Kim (he/him)
@follhim.net

So by adding in as many covariates as possible, whatever variance your primary predictor of interest, is no longer the same construct you had at the start.

Thom Volker
@thomvolker.bsky.social

At the risk of making an idiot of myself in public: most of this is not wrong, right? I cannot really comment on the statement that there are usually too few covariates in ph, but it is strictly more likely that you can block the paths that need to be blocked if you have access to more variables.

Thom Volker
@thomvolker.bsky.social

That is, at least you can ignore something that you do have, rather than control for something that you don't have. So, I guess it is always better to have more data (if there would be no financial and storage constraints of course).

Anne Scheel
@annemscheel.bsky.social

In my view there are two points here: 1) It’s about the comparison with RCTs, and RCTs largely rid you of the need to find & control for all confounders. 2) It’s not clear at all whether more covariates = better, since you might as well end up conditioning on a collider or post-treatment variable.

Dr. Alexander
@etotheipie.bsky.social

I'd also include 3) "include covariates in the model" is not using "causal methods" as such, even if focusing on confounders. Like yes, only confounders should be included, but causal do-calculus is broader than simply interrogating whether any individual variable is a confounder.

Chao Wang
@excel-wang.bsky.social

Adjusting collider and instrument variables can lead to bias link.springer.com/article/10.1... so it is not necessarily true that the more covariates in a model, the better.

link.springer.comPrinciples of confounder selection - European Journal of EpidemiologySelecting an appropriate set of confounders for which to control is critical for reliable causal inference. Recent theoretical and methodological developments have helped clarify a number of principle...
Thom Volker
@thomvolker.bsky.social

I think you misunderstand my point. You can decide to leave out colliders only if you have them, but you cannot control for something that you do not happen to have.

Chao Wang
@excel-wang.bsky.social

Yes you can leave out colliders (and instruments) but this "just collecting more covariates so RCT is no longer needed" approach actually makes the analysis harder not easier, as you have to go through the long list of covariates to decide which to include or exclude.

Chao Wang@excel-wang.bsky.social

So yes the more covariates we have, the higher chances we catch all confounders (this part is true); but it also means the higher chances we catch collider and instrument variables... then you need to think carefully which covariates to adjust.

Chao Wang
@excel-wang.bsky.social

So yes the more covariates we have, the higher chances we catch all confounders (this part is true); but it also means the higher chances we catch collider and instrument variables... then you need to think carefully which covariates to adjust.

Saloni
@scientificdiscovery.dev

I think the paragraph, and the paragraphs surrounding it, are both talking about modelling the likelihood of treatment; it's using 'data' as a short hand for the data within that model, not the number of covariates in the broader dataset that you have access to

Casaubon
@zas.bsky.social

Lord knows I took far too many metrics classes to not feel like a troglodyte for asking this but…what specifically is wrong with this? I’m forgetting so much.

Saloni
@scientificdiscovery.dev

Not stupid at all! I don't think these are obvious problems, which is probably how it got through the editing. But I'm surprised a statistician wrote it...

Casaubon
@zas.bsky.social

I mean at least senile! Barely more than two years ago I was taking the most advanced undergrad metrics classes my uni had 💀

Ben Hanowell
@hanowell.me

This reminds me of a conversation I had with the a principal applied scientist at a certain large firm I once worked at. Guy legitimately thought the best way to do causal inference was to throw as many variables in autoML as we could then do SHAP.

Ben Hanowell
@hanowell.me

If every principal applied scientist thought the way he did and acted on those thoughts, company would collapse. Thanks god they employ a ton of economists.

Ben Hanowell
@hanowell.me

If the quarterly theme of Asterisk is ever “Reflection” I’m gonna write an essay that reflects on some of the stinkers that have come through into an otherwise fun mag to read.

kfan99.bsky.social
@kfan99.bsky.social

Hey Johannes This isn't just another post in your feed. It's a family's story, shared with hope that someone will care. Please visit my page and read the pinned post. ❤️

Lachlan Cribb
@lachcribb.bsky.social

Besides the last statement, none of this seems unreasonable? A richer dataset will make it easier to adjust for confounding, all else equal. And most PH research probably does suffer more from underadjustment than from overfitting.

Ben Hanowell
@hanowell.me

Go read the article that the metaculus guy wrote about whether prediction markets are good at predicting things and then go and have a drink.

Nick DeVito
@ndevito1.bsky.social

On giving the whole piece a read, that’s def not a great paragraph but it feels more clumsy writing due to the simple writing/communication style they were going for more than fundamental misunderstanding I think? Can’t know for sure but my initial 😬 is slightly pulled back now.