Post

We just released "German Commons", the largest openly-licensed German text dataset for LLM training: 154B tokens with clear usage rights for research and commercial use. huggingface.co/datasets/coral-nlp/german-commons

huggingface.cocoral-nlp/german-commons · Datasets at Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

27/10/25, 12:45 pm9 reposts20 likes

Webis Group

@webis.de

27 Oct 2025

The current problem: training data is primarily sourced from Web crawls, which give you scale but unclear licensing. This blocks models from commercial deployment and research. We took a different path: systematically collecting German text from 41 institutional sources with explicit open licenses.

Webis Group

@webis.de

27 Oct 2025

This means: ✅ Every document has verifiable usage rights (min. CC-BY-SA 4.0 and allows commercial use) ✅ Full institutional provenance for reduced compliance risks ✅ Systematic PII removal + quality filtering, ready for training ✅ Rich metadata for downstream customization

Webis Group

@webis.de

27 Oct 2025

The data spans 7 text domains: 🌐 Web: Wikipedia, GitHub, social media 💬 Political: Parliamentary proceedings, speeches ⚖️ Legal: Court decisions, federal & EU law 📰 News: Newspaper archives 🏦 Economics: public tenders 📚 Cultural: Digital heritage collections 🔬 Scientific: Papers, books, journals

Webis Group

@webis.de

27 Oct 2025

For full technical details + compliance Datasheet see our preprint @ arxiv.org/abs/2510.13996 As for German-specific models trained on this data... stay tuned 👀

arxiv.orgThe German Commons - 154 Billion Tokens of Openly Licensed Text for German Language ModelsLarge language model development relies on large-scale training corpora, yet most contain data of unclear licensing status, limiting the development of truly open models. This problem is exacerbated f...