Post

When a newspaper becomes a wall of OCR text, columns collapse, headlines mix with ads, and reading order disappears. The Institutional Data Initiative (@institutional.org) and Boston Public Library (@bpl.boston.gov) built a pipeline and dataset to make historical newspapers more accessible.🧵

institutional.orgMaking 1.47 Million Historical Newspaper Pages Available as High-Quality, Granular Data | IDIThe Institutional Data Initiative at the Harvard Law School Library and Boston Public Library release an open dataset and processing pipeline that make historical newspapers more accessible for resear...

IDI

@institutional.org

2 Sept

Institutional Newspapers contains 1.47 million pages from the Boston Public Library’s collections, representing 16.3 billion tokens and a multilingual record of Boston’s communities across 73 languages.

IDI

@institutional.org

2 Sept

Instead of treating each page as one image, our pipeline divides it into individual crops. Those crops are processed as articles, advertisements, photographs, cartoons, headings, mastheads, or empty space, with OCR tailored to each segment.

IDI

@institutional.org

The dataset supports more than keyword search: you can follow reading order, identify people and places, and examine how newspaper design and advertising changed across time. Institutional Newspapers includes the dataset, models, and an open-source processing pipeline:

2/9/26, 4:30 pm0 reposts2 likes

IDI

@institutional.org

2 Sept

The Institutional Data Initiative at the @hls.harvard.edu Library works with knowledge institutions to increase the availability, diversity, and responsible use of training data for AI. Reach out and join us.