Reuven M. Lerner

Hi there! 👋 I'm Reuven, and I've been teaching #Python and Pandas around the world since 1995. Just about every day, my students ask great questions. I love sharing those questions (and my answers), along with other resources to help you improve your Python/Pandas fluency.

Grab part of a #Python #Pandas data frame with multi-indexed columns with xs, specifying axis='columns': ( pd .read_parquet('nyc_taxi_2025-11.parquet') .groupby('passenger_count')[['trip_distance', 'total_amount']].agg(['mean', 'std']) .xs('std', level=1, axis='columns') )

To retrieve from a multi-indexed #Python #Pandas data frame/series based on an inner index layer, use xs ("cross section"): ( pd .read_parquet('nyc_taxi_2025-11.parquet') .groupby(['passenger_count', 'VendorID'])['total_amount'].mean() .xs(1, level='VendorID') )

To retrieve all rows from a multi-indexed #Python #Pandas data frame or series, specifying the two outer layers, pass a tuple (not a list!): ( pd .read_parquet('nyc_taxi_2025-11.parquet') .groupby(['passenger_count', 'VendorID'])['total_amount'].mean() .loc[(1.0, 1)] )

To retrieve all rows from a multi-indexed #Python #Pandas data frame or series based on the outermost index, use "loc": ( pd .read_parquet('/Users/reuven/Data/nyc_taxi_2025-11.parquet') .groupby(['passenger_count', 'VendorID'])['total_amount'].mean() .loc[1.0] )

Using resample in #Python #Pandas? You'll want a list of "offset aliases," the codes that let you specify units. Some are surprising, such as "ME" and "MS" for "month end" and "month start". They're at: buff.ly/7d6ngkx

Using "resample" groups a #Python #Pandas data frame with a datetime index with arbitrary time units, similar to Grouper. For example, mean total_amount every 8-hours: df.resample('8h')['total_amount'].mean() Or tip_amount per day: df.resample('1D')['tip_amount'].mean()

I had a great time just now presenting at the "All Day AI" conference about "Teaching #Python with three types of AI" with #ClaudeCode. My slides are available here: buff.ly/OAwoxN7 Thanks to the organizers for the invitation to participate!

Want to select a full month from a #Python #Pandas data frame with a datetime index? Use a slice: df.loc['2025-11-01':'2025-11-30'] But why work so hard? Omit the day: df.loc['2025-11'] General rule: Omit a small unit in a datetime, and it becomes a wildcard.

Slice an unsorted datetime index in #Python #Pandas, and you'll likely get: KeyError: 'Value based partial slicing on non-monotonic DatetimeIndexes with non-existing keys is not allowed.' Why? If the dates are unsorted, where should Pandas start/stop? Solution: sort_index!

Want to select a chunk of time from a #Python #Pandas data frame? Make the datetime column the index, then use .loc and a slice to choose dates: ( df .set_index('tpep_pickup_datetime') .sort_index() .loc['2025-11-10':'2025-11-20'] ['trip_distance'] .mean() )

Want to avoid tons of NaNs when grouping with Grouper in #Python #Pandas? Trim the dates to avoid extremes: ( df .loc[ pd.col('tpep_pickup_datetime').dt.month == 11 ] .groupby(pd.Grouper(key='tpep_pickup_datetime', freq='14D'))['trip_distance'].mean() )

Got NaNs when using Grouper on a #Python #Pandas data frame? groupby+Grouper chops up the datetime values, from the earliest to the latest, by the specified granularity. If even one datetime is super early/late, you'll get tons of NaNs. dropna is often helpful here.

Want to change the granularity when grouping on a #Python #Pandas data frame's datetime column? Use Grouper. For example: Here we get the mean trip_distance per 14-day period: df.groupby(pd.Grouper(key='tpep_pickup_datetime', freq='14D'))['trip_distance'].mean()

This year's Nobel Prize winners have been named — and in the latest Bamboo Weekly, we use #Python #Polars to find interesting stats about laureates' countries, ages, and genders. Level up your data-analysis skills every Wednesday: buff.ly/rlIN6nG

If you have a datetime column in your #Python #Pandas data frame, you can groupby on the output of a dt method, such as day_name: df.groupby(df['tpep_pickup_datetime'].dt.day_name())['trip_distance'].mean() The above gives the mean trip_distance for each day.

If your #Python #Pandas data frame has a datetime column, you can group on a portion of that via the "dt" accessor. For example, let's get mean trip_distance for each hour of the day: df.groupby(df['tpep_pickup_datetime'].dt.hour)['trip_distance'].mean()

Python 3.15 is out! Not sure what that means? Or what features it offers? Not to worry: Here's my YouTube playlist, with demos of this new version and its major features: buff.ly/G3Lhg2t

Coding in 2026? There's so much to know: - Programming - Git - Data - #AgenticCoding Get up to speed in the 10th cohort of my #Python, Data, and AI Bootcamp. Learn, practice, and interact. (And yes, roll your eyes at my jokes.) Info at buff.ly/2WxFth8 or join Sunday's webinar: buff.ly/ygFfZNJ

You can mix and match columns and methods, and also avoid a multi-index, when grouping on #Python #Pandas data frames with "agg" and kwargs: - Each keyword is an output column name - Each value is a tuple of (COLNAME, METHOD) For example, agg(total_std=('total_amount', 'std'))

Does #Python #Pandas allow multiple data columns and multiple methods? Yes! df.groupby('passenger_count')[['trip_distance', 'total_amount']].agg(['mean', 'std']) The above returns a data frame. Its 4 columns are a 2-level multi-index with mean/std in the inner level.

Does #Python #Pandas allow multiple categorical columns and multiple methods? Yes! df.groupby(['passenger_count', 'VendorID'])['trip_distance'].agg(['mean', 'std']) The above returns: - A 2-column data frame (mean + std) - A 2-level index (passenger_count + VendorID)

Call multiple methods in a #Python #Pandas groupby with agg. Pass a list of strings (method names), and get a data frame back, one column per method. df.groupby('passenger_count')['total_amount'].agg(['mean', 'std']) Here, both methods (mean and std) are computed per group.

groupby on a single column in #Python #Pandas returns a series. Want a single-column data frame? Instead of df.groupby('passenger_count')['total_amount'].mean() pass a one-element list of data columns: df.groupby('passenger_count')[['total_amount']].mean()

groupby in #Python #Pandas is: df.groupby(CATEGORICAL)[NUMERIC].AGG_METHOD() CATEGORICAL means "not too many values." Pass a many-valued float, and Pandas won't refuse — here, 1,219 groups, most with one row. The mean of one number is... that number.

Invoke "groupby" with two categorical columns in #Python #Pandas, and get a two-part multi-index: g = df.groupby(['passenger_count', 'VendorID'])['trip_distance'].mean() Turn into a data frame with unstack: g.unstack() # index 0-6 (passenger_count), columns 1-2 (VendorID)

A "groupby" call in #Python #Pandas is normally sorted by index. But if you're grouping by month name, April will come before January. Pass "sort=False", and the index will reflect the order of appearance: df.groupby('passenger_count', sort=False)['trip_distance'].mean()

The simplest grouping in #Python #Pandas is groupby: df.groupby(CATEGORICAL)[NUMERIC].AGG_METHOD() For example: df.groupby('passenger_count')['trip_distance'].mean() Returns a series whose index is the unique values from passenger_count.

In #Python #Pandas, dt still works on NaT ("not a time"). You'll get NaN, but that's better than an error: df['tpep_pickup_datetime'].dt.year # float year (forced by NaN) or NaN df['tpep_pickup_datetime'].dt.day_name() # string or NaN

Performing date arithmetic in #Python #Pandas, and one value is NaT? Then the timedelta will also get a value of NaT. Remember, timedelta describes a length of time without a set start or end. But NaT works there, too, rather than NaN.

In a #Python #Pandas datetime column, you use pd.NaT ("not a time"), not NaN: - It's pd.NaT, native to Pandas - It can be stored in datetime columns - Can appear when text cannot be interpreted as datetime - It is ignored in aggregations, like NaN - dropna/fillna handle it, too

How fast are data centers getting built? Where? Is electric capacity growing, too? Who is building them? These questions are from the latest Bamboo Weekly. Use #Python #Pandas to solve problems based on current events and public data sets. Learn more: buff.ly/scuO3Dj

Stuck on a #Python problem? Or Pandas, Git, or #ClaudeCode? Join me: 1 hour, just us, on whatever you're wrestling with. Bring code to review or a design you're unsure about. You keep the recording + whatever we build. More about my personal coaching: buff.ly/bqjCyUC

Reading a CSV file into #Python #Pandas, and the file uses "-" or "." instead of NaN or a blank cell? Have Pandas treat those as NaN by passing na_values (and a list of strings) to read_csv: df = pd.read_csv(filename, na_values=['-', '.'])

AI coding is easy, right? Write a #ClaudeCode prompt, and you're done. Um, no: It's new techniques to learn and master Join the 6th cohort of HOPPy (Hands-on projects in #Python). You'll implement a game of your own design, with me mentoring. More info: buff.ly/cKGgcyf

Replace NaN values in a #Python #Pandas data frame with fillna: 1. Pass it a single value to replace everywhere df.fillna(999) 2. Pass a series/dict whose keys/index match columns; values fill matching columns. df.fillna(df.mean()) df.fillna({'x':999, 'y':888, 'z':777})

Are you using #ClaudeCode? As you know, building robust, maintainable software is more than prompts. At tomorrow's Advanced Claude Code workshop, you'll learn to build more robust, maintainable code. Take your Claude Code to the next level: Learn more: buff.ly/qwOKjsW

Drop rows with *only* NaN values in a #Python #Pandas data frame with the how='all' keyword argument: df.dropna(how='all') This removes completely empty rows, leaving rows containing even one non-NaN value (aka thresh=1).

I'm excited to announce the next HOPPy (Hands-On Projects in #Python), a 5-week course in which you'll use #ClaudeCode to write a game that *you* design. No theory — just learning by doing. Want to hear more? Come to Tuesday's free HOPPy 5 info session: buff.ly/VvPbxHO

Drop rows containing NaN in a #Python #Pandas data frame, but only in specific columns, with "subset": df.dropna() # drops rows containing NaN df.dropna(subset='x') # drops rows where column x is NaN df.dropna(subset=['x', 'y']) # drops rows where x or y is NaN

You can remove NaN from a #Python #Pandas data frame with dropna, but be careful: It removes rows with even one NaN, which can be overkill. Pass "thresh" to allow some rows to remain, even if they contain NaN: df.dropna(thresh=2) # keeps rows with 2 or more non-NaN values

Want to remove NaN from a #Python #Pandas series? Three basic options, all returning a new series: - s.dropna, NaNs (and their indexes) go away - s.fillna(n), NaNs are replaced by n - s.interpolate, NaNs are replaced via a linear function

Coming to #Python #Pandas from NumPy? You'll reach for np.isnan: s = Series([10, np.nan, 30]) s.loc[ ~np.isnan(s) ] # returns 10, 30 Unfortunately, this works. Better, use s.isna (or s.isnull). But the best way to drop NaN? Use dropna: s.dropna() # returns 10, 30

It's easy to write #Python with AI. But it's not as easy to build maintainable, well-engineered applications. On Wednesday, join my hands-on, advanced Claude Code workshop. Topics include custom skills, testing, and telemetry. More info at: buff.ly/qwOKjsW

Want to write NaN in #Python #Pandas, but are annoyed that NumPy removed it? Or: Tired of importing NumPy, just for np.nan? I learned *yesterday* that you can say: float('nan') or float('NaN') Both return a perfectly usable NaN, good in NumPy and Pandas. Wow. 🤯

How often are AI incidents happening? Are they becoming more frequent? Who is getting harmed? In the latest Bamboo Weekly, we analyze the AI incident database using #Python #Pandas — plus joins, JSON, pivot tables, and Plotly. More info: buff.ly/BO8uMFy

PyArrow dtypes in #Python #Pandas are nullable (with pd.NA): s = Series([10, pd.NA, 30], dtype='int64[pyarrow]') s is: 0 10 1 <NA> 2 30 dtype: int64[pyarrow] The dtype is int64, but allows nulls. (Use np.nan? It's turned into pd.NA.)

You have a #Python #Pandas series with ints + NaN. You don't want float forced on you. Solution: Use the "extension" type Int64 (note Initial Caps) and pd.NA: s = Series([10, pd.NA, 30], dtype='Int64') # must name explicitly s is: 0 10 1 <NA> 2 30 dtype: Int64

Values in a #Python #Pandas series all have the same dtype. np.nan is a float, so (1) the series gets a float dtype, or (2) it's "object," meaning "something unspecific." Annoying for integers. Awful for other types. For this, we need nullable types! (Coming tomorrow...)

Missing data? NumPy calls it nan. #Python #Pandas displays it as NaN. But: Pandas doesn't define pd.nan or pd.NaN. NumPy removed np.NaN in version 2.0. So you have to refer to np.nan from within Pandas, and it'll be displayed as NaN.

NumPy is very strict about nan. But #Python #Pandas throws nan values away: s = Series([10, 20, 30, np.nan, 50]) s.sum() # np.float64(110.0) s.mean() # np.float64(27.5) s.count() # np.int64(4) Get stricter behavior with skipna=False: s.sum(skipna=False) # np.float64(nan)