Post

I am very excited to release WordLevel, an open source Python package for comparing texts at the word level! wordlevel.dev WordLevel makes it easy to compare texts with measures ranging from the Jensen-Shannon divergence to dictionary-based sentiment, and visualize them as word shift graphs.

Ryan J. Gallagher

@ryanjgallag.com

WordLevel is designed to be fast and reliable. It has a modern Python interface with a declarative API powered by Polars. You can use it with any text that you can tokenize, and it integrates with any Narwhals-supported dataframe. The word shift graphs are publication ready and fully customizable.

Ryan J. Gallagher

@ryanjgallag.com

I’ve been working on this passion project for a year and a half. If you have been following my work for a while, you may recognize this as a new and improved version of Shifterator, the first package I developed for word shift graphs over 8 years ago

12/10/26, 1:22 pm0 reposts2 likes

Ryan J. Gallagher

@ryanjgallag.com

I’m excited to share WordLevel, which incorporates a lot of the lessons in software engineering, data analysis, and scientific computing that I have learned since Shifterator. If you want to learn more, dive into the docs or check out the code! Docs: wordlevel.dev Code: github.com/ryanjgallagh...

wordlevel.devwordlevelPython package for natural language processing at the word level

Ryan J. Gallagher

@ryanjgallag.com

If you would like to learn more about word shift graphs, please read our paper! link.springer.com/article/10.1...

link.springer.comGeneralized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts - EPJ Data ScienceA common task in computational text analyses is to quantify how two corpora differ according to a measurement like word frequency, sentiment, or information content. However, collapsing the texts’ ric...