Post

That's the spirit!

This stuff is second level fake. The first level is citing something you haven't read yourself and relying on what someone else said about it. Then the next level is citing something you haven't read that was supplied by an LLM.

In no way minimizing the problem s you cite and its effect on scholarship and research, I wonder about the many other areas of life where AI generated falsities will be applied.

Good questions, all. I know what I think should be the answers, but admittedly I haven't been in this area for years.

Like "delete me" for fake research papers

They can be made up, same problem.

Hadn't realised that - it's even more horrific

You probably won't see this but is this happening on LexusNexus too?

It’s such a trivial thing to do in the proofing stage, quite shocking that it apparently isn’t. As always, most work is left to the reviewers who do that for free.

I use scholar-google several times/week to look up exclusively climate stuff. Yet even among climate stuff, I already came across maybe 5-8 "papers" that only have a citation stump without 🔗 to an article. Google adding a 🔗to the stump in the author's pub list, like on Nelli Piattoeva's, is slander.

One would hope so…

I tried this with student term papers this fall. GPTzero has a website that claims to check the veracity of a citation list. It hallucinates. I fed it a list with some fake citations (from a student paper) and it gave them a green light, giving false DOIs that “prove” they are real.

😔

😔

That's (almost but not quite) the same fake citation I first found, before I discovered its variant on Scholar. Thanks for sleuthing this out. At some point I must try diagramming its evolution and appearances to figure out its "origin".

ChatGPT was not open access at that time but GPT1 was already available to academic researchers since 2018... The fake sources are seemingly based on a mix-up of your articles, including with Piattoeva, and other sources, e.g. a mix-up of sources of this editorial here (see journal in 1st footnote)

We had llms and transformer architecture for a couple of years before chatgpt though. Just without RLHF. ....and a shiny public interface. But would make more sense what you say here. This is an interesting development. Pinging @aarontay.bsky.social

politicalresearch.org/2025/03/25/c...

Google Scholar has always had some fundamental issues in how it operates, as it tries to index as much as possible yet has limitations in how it can verify data. In that sense what we see now is just additional error sources. In this 2010 study it e.g. indexed nonsensical generated papers.

Which makes it especially weird that in the paper I was reviewing today the precise same, totally blandified title is credited in a different journal and strips out the coauthor. Is a new fake reference being generated from the last?...

Um, it's up to 43 now.

I am not surprised by this at all, and it's only partly the fault of LLMs. Many, many people cite work that they have not bothered to read. I suspect this is especially egregious in the torrent of papers on AI. It's scholarly pollution that is rarely caught :/

We need a new iteration of retraction watch for retracting publications that are not publications.

You found this paper (the actual published paper?) or you found that bibliographic info? How— in someone’s reference list? What did you find?

We are only just realising the hellscape that AI will release. AI doesn't have a conscience. And there is nobody to punish when it plagiarises, lies or misinforms.

@author-for-sale.bsky.social

Good God. That's just insane.

Saving this thread for my Research Methods class in January. (I do have colleagues excitedly praising how much time you save letting LLMs do your lit reviews for you, so holding undergrads to a higher standard, I guess?)

AI is fulfilling the "The party told you to reject the evidence of your eyes and ears" 1984 overused cliche, contributing to the death of meaning.

Does this mean the AIs are working together to increase their influence already? ;-)

The problem is an old one adapted and amplified to infinity: nobody reads. With a few honorable exceptions, academia loves AI that cites endless papers that don't exist because, in any case, it wasn't going to read the citations. Let alone the papers. By reading both, none of this can happen.

It’s AI. I’ve seen musicians complaining that there are now album credits online for projects that they were not involved in or do not exist.

Frankly, Grammarly is an essential part of my everyday life, and I couldn't get by without it - I have two neurospicy kids, and whilst I haven't been diagnosed, they didn't get it from their mum!

Well that's the truth!

I have no need for it. My brain is very good at these things.

Quite so . I’m actually tracing about 15 SI generated papers that were submitted to one of the journals that I edit, to see where they will end up, how often they will get cited, to what extent author names and affiliations change etc.

No one cares enough about this to solve it.

You might be interested to read ‘Spurious scholarship’, a little thing I wrote some time ago now. academia.edu/resource/wor... I had thought things were improving, but yours is the second recent instance to worry me.

The AI bots scrape all authors’ CVs & make really poor citations for non-existent material.

AI is not ready for prime time, if ever!

Yeah, a lot of copy and paste rather than reading and evaluating the lit.

Another AI, obviously, duh. (/s)

Maybe Universities should establish cite checking departments and push all professors to have that department review their papers before they submit them to journals. The University has a strong interest in maintaining it's reputation as a source of reliable research.

Google Scholar lists one of my advisers as the author of my dissertation. So far it has been cited with me as the author but it's frustrating that there seems to be no way to get Google to correct their error.

So do you automatically reject a ms with fabricated references?

Should be fairly easy to write a (not ai powered) tool for peer reviewers, which checks validity of citations in a paper. Would be even easier if the defacto publication format wasn't pdf, but that shouldn't be a showstopper.

Needless to say, in consultation with the other editors, we decided not to publish the book review.

Again, RePEc does the right thing bsky.app/profile/czim...

Back in the day, people used to read papers before citing them. What happened to that?

Horror!

@forbetterscience.bsky.social

The big question is this: Can you (will you) sue the AI company/companies which are responsible for these fabrications, which undoubtedly damage your reputation as a scholar in your field? Real accountability, I think it’s over due!

I work in transportation planning. I've told my staff we can use AI to find information, but they have to go to the original source (which our company's version of AI provides) and read the source material themselves. If they haven't read it, they can't cite it.

This is going to make some people's vivas especially tense.

IMO the greatest problem in science publication isn't the difficulty of finding reviewers, fraudulent authors, abusive reviewers, or exploitative journals, but editors becoming lazier and lazier. The others are massive problems too, but lazy editors cause the first and make the others worse.

That's gross

That should be cause for automatic paper retraction and academic discipline and journals that don’t enforce it should be blacklisted.

Problem is, if this 'improves' H scores etc, there is less motivation for authors to remove non-existent-yet-well-cited papers. 42 refs in the last year according to OP, that sounds like something you'd put on your cv to show how awesome your research is 🤨

Academic publishing is borked. (Becoming an evergreen post.)

There could be a new kind of notary public.

We do tend to find solutions that make matters worse, don't we? This could mean good folks who've worked in good faith have legit work retroactively besmirched… Hmm. Instead of rooting out heretics, maybe we need to better stress what "good faith" really means and demand it of all as we move ahead.

Not a new job. Just a new task for an overworked, underpaid grad student.

By *a lot* do you mean padded? 😉

I'm being sarcastic. As lecturers we suddenly need to spend so much time MANUALLY checking references of students, it's the only way to assess genAI cheating rather than reasonable use, there has to be an app, please someone, invent it or tell me how you are checking this??

Which they'll give to an AI and on and on...🙄

That’s my job, currently. Not my whole job, but I did recently search for every single citation on a paper that had more than 100 refs. All legit, though I think the AI slop will be slow to come to our niche.

They already do apparently.

They'll probably insist the authors check themselves, tick a box that they did, and there's going to be an extra publication charge for "AI-corrected citation verification" or something equally unhelpful.

They are blacklisted.

Yikes

Seen this, @joshgellers.bsky.social ?

Yes, of course we rejected it. It violates even the relatively permissive publisher policy and we told the author so.

I'm curious what your response will be to the author of the paper you reviewed?

As a former government scientist (prior to being RIF’d), the fact that all these papers were published with this fake citation, and no one noticed until you HAPPENED to get it during peer review, is WILD. People clearly need to be better at using AI or don’t use it all all.

Say, from one journal editor to another, I wrote a thing to dump references sections into & get out a properly APA 7th formatted list with suspected hallucinations flagged. Free of charge/open source ofc. AI powered but reliable (though slow) jenkleiman.com/reference-ch.... Already caught 2..sigh.

@blakeplease.bsky.social

That is just poor, from the authors that clearly did not check what they were citing, to the reviewers, to the editors. But the first blame should be with the authors, that is just sloppy and unscientific to be citing papers that do not exist.