Aaron Tay

Comments to this are a shitshow. Essentially xkcd.com/927/. Honestly why did ORCID not succeed the way ror.org and to a great extent Crossref did? I'm sure there are many reasons but I do think they made a fundamental strategic error by refusing to own the researcher profile.

I guess they underestimated the incentives researchers have to curate their own profile. It didn't help Google scholar profile launched and became the most popular yet ignored ORCID. Orcid idea is elegant but in reality integrations to other systems are not easy.

/1 Took some time to read and understand this. In short RCP-nDCG combines LLM based listwise relevance comparisons with rubric-based relevance judgements, then uses Item Response Theory to calibrate the comparative scores.https://cohere.com/blog/rcp-ndcg

/5 In fact the paper does so with "Count-nDCG" where an LLM judges every document against 5 relevance criteria and the proportion of criteria passed becomes its relevance gain for nDCG. Interestingly the proportion of pairs discriminated is very similar to the full RCP-NDCG but still had many ties

/1 Took some time to read and understand this. In short RCP-nDCG combines LLM based listwise relevance comparisons with rubric-based relevance judgements, then uses Item Response Theory to calibrate the comparative scores.https://cohere.com/blog/rcp-ndcg

/3 of course this comes from the traditional information retrieval perspective, where Cranfield style eval rely on relevance judgements that are often incomplete because of pooling methods. RCP-nDCG proposes a more sophisticated LLM-as-a-judge method to address this problem.

/1 Took some time to read and understand this. In short RCP-nDCG combines LLM based listwise relevance comparisons with rubric-based relevance judgements, then uses Item Response Theory to calibrate the comparative scores.https://cohere.com/blog/rcp-ndcg

Oxford library agent. Uses Nexus Connect /1

How it looks like /2

Murdoch university rubric for evaluation of AI tools. This isnt one of those in-depth ones that needs some understanding of the tool mechanics, but its not bad for a vibe based/subjective one with 3 out of 5 criteria that are arguably performance based /1

That'd why for relevancy i favour hard tests. Minimally precision, recall if possible. Rather than qualitive rating /4

Murdoch university rubric for evaluation of AI tools. This isnt one of those in-depth ones that needs some understanding of the tool mechanics, but its not bad for a vibe based/subjective one with 3 out of 5 criteria that are arguably performance based /1

They used Microsoft forms. Instructions to testers are not bad thought i would suggest testers use "tricky" or "hard" queries to stress test relevancy, how search tool reacts to no good results. I find librarians often test with generic overly easy queries which makes things too easy /2

I keep noticing this weird librarian narrative. First they note NLP is not new. True. Deep learning not new. True. Then they conclude hence AI/LLM is not new and over hyped. Huh??

Frankly if your objection to "ai" is environment, learning what RAG is later obviously wont change your mind. But they probably don't realise what is often branded "AI" are much smaller encoder models that have far less environment impact... unfortunately human tend to cling on to what they believe

I keep noticing this weird librarian narrative. First they note NLP is not new. True. Deep learning not new. True. Then they conclude hence AI/LLM is not new and over hyped. Huh??

Sure transformer architecture is just a neural net architecture trained using DL. But are we going to pretend since 2017 this led to insane performance that we could not even dream of? Call it AI or whatever you want it doesn't change how it performs

This exactly shows what I have been saying. On relevancy Primo RA wins by landslide. Its not due to llm creating boolean since ask anything has that (not exactly same.). It is due to additional reranking of top 30 using embeddings in Primo ra!

And yes if you are evidence synthesis person I know for you the BOOLEAN is the point. But that's not who I am addressing. Most people don't search for evidence synthesis

This exactly shows what I have been saying. On relevancy Primo RA wins by landslide. Its not due to llm creating boolean since ask anything has that (not exactly same.). It is due to additional reranking of top 30 using embeddings in Primo ra!

They also tested currency/authority etc. Practically no difference . So overall Primo Ra won due to relevancy. Notice how it won despite librarians complaining the boolean Primo RA produces is objectively weird? Again reranking is important!

Closing keynote. If you ask me to do this, at very least librarians must up our game in understanding information retrieval, agentic flows etc

This exactly shows what I have been saying. On relevancy Primo RA wins by landslide. Its not due to llm creating boolean since ask anything has that (not exactly same.). It is due to additional reranking of top 30 using embeddings in Primo ra!

SDSU bento style search using LLM + various apis Primo/Springshare etc

Very nice. Though I'm still skeptical how good the Primo collection search for topical searches are since its just boolean search

SDSU bento style search using LLM + various apis Primo/Springshare etc

This feels like what I found out. Because context windows so large you don't have to do RAG for everything eg recommend librarian, just toss json/ everything to llm.

SDSU bento style search using LLM + various apis Primo/Springshare etc

Very nice. Though I'm still skeptical how good the Primo collection search for topical searches are since its just boolean search