Berna Devezer

Berna Devezer

@devezer.bsky.social

Metascientist @ uidaho. Behavioral sciences, stats, modeling, phil of science. Cat slave, cook, sailor (see: https://bsky.app/profile/kumrusails.bsky.social), photographer, clarinet student, adventurer, nature lover. Allergic to unsolicited advice.

11.1K followers998 following34.3K posts

Pinned

A revised version of this preprint is now posted on arXiv. The theory section is rewritten, improving model exposition based on feedback we received, we fixed some confusion and errors, contextualized the models more carefully so the interpretation becomes clearer, and improved the overall writing.

arxiv.orgThe Difference Between "Replicable" and "Not Replicable" Is Not Itself Scientifically ReplicableReplication studies estimate the replicability rate of scientific results by aggregating binary verdicts of experiments as ``replicated'' and ``not replicated''. These estimates assume rarely attainab...
Berna Devezer@devezer.bsky.social

📣 The difference between replicable and not replicable is not itself scientifically replicable. 📣 New work with Erkan Buzbas, showing that verdicts such as "X% of results replicated" are based on an inferential machinery that doesn't work. arxiv.org/abs/2604.26268

little bit of a tangent: i once had a department chair who physically threatened me because the graduate student he was having an affair with was struggling in my class. i ended up resigning a few months later. so dating a student can have far-reaching consequences beyond the abuse of the student.

This post is unavailable.

my favorite answer in turkish is: slightly better than living with in-laws

Jakub Jaraczewski@jakubjaraczewski.bsky.social

🇵🇱 Polish range of answers to "How are you?" ("Fine, thanks, how are you?" is not an option) - somehow it's flying (jakoś leci) - dickish but stable (chujowo ale stabilnie) - old poverty (stara bieda) - could be worse (mogło być gorzej, peak expression of Polish happiness)

got my shots (covid arm feeling like a block of lead, flu arm is fine), finally finished my passport renewal application (been procrastinating), voted, posted exam reviews, did my back and biceps workout despite the lead arm... feeling weirdly productive! might just watch a horror movie tonight.

i wonder if prop 1 has a shot in idaho. sure hope so. women's health care is suffering in the state because half the ob-gyn docs left out of fear and helplessness. didn't vote for the state gun tho. wtf?! this ballot question brought a vulgar turkish idiom to mind: idle shopkeeper weighs his balls

anyhow for the movie we chose the mist (2007). ugh should have stuck to the 90s. not good. way too long. not the best cgi. and terrible ending (different from stephen king's original story). interestingly stranger things theme though; i was surprised by that.

tbh i had no idea we had metrics to evaluate peer review. if we knew what it should be like, we'd have already done a better job at it. but i have faith we can goodhart our way out of every problem, with or without ai.

View full thread

The unpopular part of the popular Box quote comes to my mind every time we worry about everything that's conditioned on the research design instead of worrying about the design: "It is inappropriate to be concerned about mice when there are tigers abroad"

Like in this retraction debacle, everyone got fixated on the "missing" preregistration (where in fact it wasn't missing but the paper completely disregarded the existing one) instead of discussing how terrible the design was to test the research question the paper purported to answer.

Berna Devezer@devezer.bsky.social

One thing I wanna add to Joe's fascinating write-up is: Ask yourself what would change if the project always intended to study replicability even though it doesn't appear in any preregistration from early on? Who would design *this study* to test the relationship between reforms and replicability?

tbh i had no idea we had metrics to evaluate peer review. if we knew what it should be like, we'd have already done a better job at it. but i have faith we can goodhart our way out of every problem, with or without ai.

View full thread

Still bewildered abt this wildly inappropriate design to address the research question. Peer review is an instrumental process. What does it mean to measure whether it's "good" if the "what for?" question is never asked? Why should anyone care how three editors and AI evaluate peer review content?

I am feeling increasingly frustrated with this disconnect between questions people ask and designs they execute. We keep talking about *fixing science* via tools, checklists, transparency, honesty signals... but this core problem in some fields cannot be fixed with any of those.

I like that it's now normal to be an introvert and say no to social occasions. Ofc that could also be a result of having moved from Turkey to the US. Either way, I like not being forced into social situations as often. So many weddings I had to hide under the table.

Social media loves talking about how everything is getting worse. But what’s something that you’ve seen get _better_ in your lifetime?

I also like that shoes are vastly more comfortable now. No more bandaid covered feet. And hair products OMG. Magic! I used to hate my unruly hair. No more.

Berna Devezer reposted

idea in the shower --> prompt --> chatbot --> paper --> chatbot --> reviewers --> chatbot --> reubuttal --> chatbot --> associate editor --> chatbot --> random number generator --> acceptance --> CV --> profit

tbh i had no idea we had metrics to evaluate peer review. if we knew what it should be like, we'd have already done a better job at it. but i have faith we can goodhart our way out of every problem, with or without ai.

Berna Devezer reposted

Replying to @devezer.bsky.social

The purpose of peer review is not primarily to help authors improve their papers.

Berna Devezer reposted

Replying to @devezer.bsky.social

Chris Donkin and I make a similar point here: doi.org/10.1080/0951... Ironically, It's a concept that invites misuse. Accusations of "QRP" are easy to make and difficult to dispute, even with preregistration etc.

Berna Devezer reposted

Replying to @devezer.bsky.social

My experience has been that some of these folks (not all, but some) they believe their *own* practices are unquestionable, so there is that

Berna Devezer reposted

a few of these guys have me blocked from 2020 on twitter — some have called me names because of this retraction we helped facilitate (see right side "Responses" www.eneuro.org/content/6/6/...) — so I have no idea what they are up to here nor on twitter (because I also do not go there at all), but...

I truly dislike the term "Questionable Research Practices," most of all because it implies the existence of "Unquestionable Research Practices," which itself would be highly questionable if you think about it.

But I also dislike it because it's not a theoretical framework to make sense of science and does not provide any understanding. At best, it's a listicle, and this type of listicle approach to science gives me indigestion.

Berna Devezer reposted

Anyone in academia (or anywhere else) know anything about Korn Ferry Consulting? They are going to run the search for the new president at the University of Idaho. Would greatly appreciate any info. #highered #academia #academicsky www.kornferry.com

kornferry.comKorn Ferry | Organizational ConsultingKorn Ferry is a global organizational consulting firm, bringing together strategy and talent to drive superior performance for our clients.

sometimes i get quoted in an affirmative fashion but the quoter goes a step too far. like when i critique some research and they quote me to insult the researcher. makes me feel trapped. i don't like insults and ad hominems. i don't want to be part of that but detaching post feels too hostile.

i think this is a companion piece to what i posted yesterday. in general my way of maintaining some social media hygiene is to not partake in toxicity or personal attacks, to maintain some basic level of respect and good faith, and to leave the conversation when i assume/start assuming bad faith.

Berna Devezer@devezer.bsky.social

one bad habit i have not imported from twitter is to lift irritating, rude, or otherwise bizarre replies from my mentions with a qt. it's one little thing i do for the world i'd like to live in and for positive vibes during the day. i reply or mute or block but i don't try to expose or shame anyone

one bad habit i have not imported from twitter is to lift irritating, rude, or otherwise bizarre replies from my mentions with a qt. it's one little thing i do for the world i'd like to live in and for positive vibes during the day. i reply or mute or block but i don't try to expose or shame anyone

for replying to me on here. even if, and especially if, i feel like many of you would agree with me. i think it's a good thing most of the time. and that's the type of content i don't like seeing on the timeline myself.

i'm not above subposting though. that's more venting than pointing fingers. i still try not to subpost mutuals though. that feels rather iffy.

Watched two nontypical entries in the vampire genre: Guillermo del Toro's directorial debut Cronos (1993) and Park Chan-wook's Thirst (2009). Both worth watching. Cronos looks very fresh for being that old tho writing could be better. Thirst, based on Zola's Thérèse Raquin, is so odd (complimentary)

Cronos was full of curious details, very on brand for del Toro, and a couple of unexpectedly hilarious scenes. The acting in Thirst was so over-the-top but so good. Both explore the notion of a monster from an empathetic pov and feel more sad than horrifying.

ad copy is probs one of the worst places to use llm-generated text. both bec it's so sparse that there's nothing to hide behind and it's the last place you want to sound generic or cliché. just watched an online ad for vuori that started with "every performance has a before and an after." o rly 😂

Berna Devezer reposted

Enjoyed seeing this woman's work. Sustainability isn't just about producer decisions, but also consumer decisions, namely their willingness to repair things when needed. This type of repair is easier on natural fibers, rather than stretch fabrics. Consider that when shoppping. IG loosethreadsrepair

finally made my double-vaxx appointments for friday. fingers crossed i won't catch anything until then. thanks everyone who's been posting about getting theirs or i'd have forgotten about it.

Berna Devezer reposted

Replying to @devezer.bsky.social

I see the same thing regularly on here with people complaining about economics. It's a brainworm that could be cured just by taking a couple of higher level economics courses. Always easier to find and crucify an out group rather than accept that things are nuanced.