Post

@theegoddesssydney.bsky.social

This thread fails to mention that the release (even though based on PD material) prohibits commercial use and comes with other unusual terms (which open-source project wants to hire lawyers to determine whether your lawyers would agree that it is "unaffiliated with commercial ... intent"?)

Goddess Sydney J

This is so amazing!!! Thank you

We analyzed the dataset’s coverage across time, topic, and language and found: - 40% of English text + long tail of 254 languages - 20 clear topical tranches - Largely published in the 19th and 20th centuries Technical report here: arxiv.org/abs/2506.08300

We included extensive volume-level metadata with both original and generated components, such as results from text-level language detection.

As part of our refinement work, we supplemented the original OCR-extracted text with a post-processed version that utilizes line detection to reassemble the text according to the line type.

huggingface.coinstitutional/institutional-books-1.0 · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

We look forward to growing Institutional Books through community. We welcome collaboration from researchers and model makers as we: - Evaluate the dataset’s impact on model outputs - Continuing to refine our OCR pipelines View the dataset on Hugging Face: huggingface.co/datasets/ins...

polecopub

@polecopub.bsky.social

@thebigotbasher.bsky.social

Fantastic! I have a question: you mention on the website, non commercial use, but is there a CC license on the whole thing? Is there a different licence (CC0?) on the books metadata? Thanks a lot...

The Bigotbasher

This is pretty amazing

Mark Thakkar

@brunellus.com

“refined for accuracy”: with even cursory checks by humans? I ask because if you sort the metadata by date (ascending) you‘ll see “Union shop case, 1951-52. Pamphlets” with a date of 1295. There are bound to be *some* mistakes, of course, but that’s not the only obvious one you can find this way.

Bevan Crews Tumblin

@thehotelcat.bsky.social

📌

myedison.bsky.social

@myedison.bsky.social

This is great, but WHY do I need to give you personal information about myself to sign up?

Just me

@97wolverine.bsky.social