pandas
The pandas pack has 137 rules, and xpd can fix 81 of them.
The pandas rules find code that is wrong on pandas 3, where copy-on-write is always on, and code that has a shorter or faster form. Read the notes of a rule before you apply a fix that can change the result, as the table shows.
| Fix | Result | How to apply it |
|---|---|---|
| safe | the same result | xpd check --fix |
| unsafe | can change the result | xpd check --fix --unsafe-fixes |
| none, with the replacement shown as advice | can change the result | edit the code by hand |
Examples assume import pandas as pd and import numpy as np, and names such as df,
output_columns and mapping stand for values in your program. A fix that writes pd. or np.
applies only in a file that imports pandas as pd or NumPy as np, and in any other file xpd
reports the code without a fix.
xpd skips a rule that asks for a newer API than the installed pandas has. The pseudocode shows how xpd finds the installed version.
venv = $VIRTUAL_ENV, or else the nearest .venv in or above the working directoryif venv holds a folder named pandas-<version>.dist-info skip each rule in the table that needs a newer pandas than <version>else run every rule| Rule | Runs on |
|---|---|
lambda-in-assign |
pandas 3.0 and later |
concat-then-category |
pandas 3.1 and later |
| every other pandas rule | every pandas version |
Code that is wrong or broken on pandas 3
Section titled “Code that is wrong or broken on pandas 3”Each rule in this group finds code that gives a wrong result, raises, or warns on pandas 3.0 and 3.1.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| chained-assignment | error | unsafe | Under copy-on-write, assigning through a chain such as df["a"][i] = v writes to a copy, so the frame keeps its old values. |
| inplace-on-column | error | unsafe | Under copy-on-write, an inplace=True call on a selected column changes a copy, so the frame keeps its old values. |
| inplace-result-assigned | error | unsafe | A call with inplace=True returns None, so df = df.dropna(inplace=True) sets df to None. |
| no-inplace | info | unsafe | inplace=True gives no speedup and keeps the call out of a method chain. |
| epoch-from-int-cast | error | unsafe | pandas 3 stores datetimes in microseconds by default, so casting a datetime to an int and dividing by 10**9 is off by a factor of 1,000. |
| compare-to-missing | error | unsafe | A comparison with a missing value, such as df["b"] == np.nan, is never true. |
| in-column | error | unsafe | The in operator on a column looks among the index labels, not the values, so 2 in df["a"] can be wrong. |
| mask-precedence | error | unsafe | Python applies & and | before comparisons, so df["a"] > 1 & df["b"] < 3 compares against the wrong value. |
| write-through-values | error | none | Under copy-on-write, the array behind a column is read-only, so writing through .values raises ValueError. |
| object-dtype-check | error | unsafe | pandas 3 stores text in the str dtype, so a check for object is False for a text column. |
| datetime-unit-check | error | unsafe | pandas 3 stores datetimes in microseconds by default, so a check for datetime64[ns] is False for them. |
| groupby-apply-reads-key | error | none | groupby.apply no longer passes the grouping column to the function, so reading it inside the function raises KeyError. |
| removed-in-pandas-3 | error | unsafe | These names and arguments were removed in pandas 3, so the code raises. |
| reduction-level-argument | error | unsafe | The level argument of reductions such as sum was removed, so the call raises TypeError on pandas 3. |
| stack-dropna | error | unsafe | stack no longer takes dropna, so the call raises ValueError on pandas 3. |
| frame-append | error | none | DataFrame.append and Series.append were removed, so the call raises AttributeError on pandas 3. |
| str-replace-regex-default | error | unsafe | Since pandas 2.0, str.replace reads its pattern as plain text, so a regular expression replaces nothing. |
| positional-key-on-labels | error | unsafe | On pandas 3, an integer key on a Series is a label, so value_counts()[0] and similar lookups raise KeyError unless 0 is one of the labels. |
| day-frequency-with-time-zone | warning | unsafe | On pandas 3, "D" means a calendar day, so shifting a time zone aware index by a day can raise at a daylight saving change. |
| mixed-offsets-without-utc | warning | none | On pandas 3, to_datetime raises ValueError when texts with a UTC offset hold more than one offset, as local times do across a daylight saving change. |
| deprecated-in-pandas-3 | warning | unsafe | These names and arguments warn on pandas 3 and will stop working in a later release. |
Shorter forms, including the pandas 3.0 and 3.1 API
Section titled “Shorter forms, including the pandas 3.0 and 3.1 API”Each rule in this group names a shorter way to write the same code, often with an API that pandas 3.0 or 3.1 added. Most of the rules are warnings, and the rest are info.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| sort-then-reset-index | info | unsafe | reset_index(drop=True) after a sort renumbers the rows in a second step. |
| set-index-then-resample | warning | unsafe | Setting a column as the index only to resample takes an extra step. |
| subtract-shift | warning | unsafe | Subtracting a column's own shift() computes the difference from the previous row by hand. |
| range-mask | warning | unsafe | Two comparisons joined with & state one range twice. |
| groupby-then-reset-index | warning | unsafe | reset_index() after a groupby moves the keys back into columns in a second step. |
| rename-after-agg | warning | none | Renaming columns right after agg names them in a second step. |
| merge-indicator-anti-join | warning | none | Filtering the _merge column after merge(indicator=True) builds an anti join by hand. |
| value-counts-share | warning | unsafe | Dividing value_counts() by the length computes shares by hand. |
| nested-np-where | warning | none | Nested np.where calls read from the inside out and build an array for every level. |
| query-string | info | none | A query string hides column names from editors, type checkers and xpd. |
| filter-items | warning | unsafe | filter(items=) skips a missing name without an error, and pandas 3.1 plans to deprecate it. |
| inline-column-list | info | none | A list of three or more column names written inside a call is hard to read and to reuse. |
| concat-then-category | warning | unsafe | Casting to category after concat makes a second pass over the data. |
Python work per row or per value
Section titled “Python work per row or per value”Each rule in this group finds code that calls a Python function once per row or once per value, and names the pandas method that works on the whole column at once. Several rules don't run on test files, because a test builds its expected value row by row on purpose.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| apply-known-function | warning | unsafe | Passing a function such as len, pd.isna or pd.to_datetime to apply calls it once for every value. |
| literal-contains | warning | unsafe | str.contains treats its pattern as a regular expression by default, even when the pattern is plain text. |
| copy-after-indexing | warning | unsafe | Under copy-on-write a selection already behaves as a copy, so .copy() after it only costs memory and time. |
| lookup-per-row | warning | unsafe | Looking up a key in another frame inside apply reads that frame once for every row. |
| apply-recomputes-aggregate | warning | none | A lambda that computes df["b"].mean() runs that mean again for every row, so the time grows with the square of the row count. |
| enumerate-map-encoding | warning | unsafe | Numbering values with a dict built from enumerate(unique()) looks up every value in Python, which was 7 to 17 times slower. |
| copy-in-chain | warning | unsafe | Methods such as rename and assign return a new frame, so a .copy() next to them is thrown away. |
| row-loop | warning | none | iterrows() builds a Series for every row, and a loop that only computes from row values runs Python once for every row. |
| apply-axis-1 | warning | none | apply(axis=1) calls a Python function once for every row, which was hundreds of times slower than working on whole columns. |
| apply-text-method | warning | unsafe | Calling a text method inside apply or map runs Python once for every value. |
| apply-date-part | warning | unsafe | Reading a date part such as the year inside apply runs Python once for every value. |
| apply-condition | warning | none | A condition inside apply runs Python once for every value. |
| apply-arithmetic | warning | unsafe | Arithmetic inside apply runs Python once for every value. |
| builtin-on-column | warning | unsafe | A Python builtin such as sum, set or list walks a column one value at a time, which was up to 483 times slower. |
| list-over-column | warning | none | A list comprehension or a loop over a column runs Python once for every value, which was 13 to 19 times slower. |
| frame-growth-in-loop | warning | none | Adding one row at a time with out.loc[len(out)] = row copies the frame on each pass. |
| len-of-filter | warning | unsafe | len(df[mask]) builds a filtered frame only to count its rows. |
| sort-then-head | warning | unsafe | Sorting a whole frame to keep its first or last few rows does more work than needed. |
| replace-with-mapping | warning | unsafe | replace with a mapping takes longer for every key in the mapping. |
| concat-in-loop | warning | none | Calling concat inside a loop copies every earlier row on each pass. |
| apply-math-function | warning | unsafe | Passing a math function such as np.sqrt to apply or map calls it once for every value. |
| apply-cast | warning | unsafe | Casting with apply(int) or map(float) converts one value at a time in Python. |
| apply-comparison | warning | unsafe | A comparison inside apply runs Python once for every value. |
| apply-pd-series | warning | unsafe | apply(pd.Series) builds one Series for every row to split a column of lists or dicts. |
| index-loop | warning | none | A loop over index labels that reads the frame with loc or at runs Python once for every row. |
| zip-over-columns | warning | none | A loop over zipped columns runs Python once for every row. |
| column-loop | warning | none | A loop that reduces one column per pass makes a separate call for every column. |
| filter-before-frame | warning | none | Filtering records in a Python comprehension before you build a frame tests every record in Python. |
groupby, agg and transform
Section titled “groupby, agg and transform”Replace a lambda or apply in a groupby with a built-in aggregation, which pandas runs in compiled code for all groups at once.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| groupby-lambda | warning | unsafe | A lambda or a NumPy function in agg runs Python once for every group. |
| groupby-lambda-window | warning | unsafe | A lambda in transform that calls shift, diff or a similar method runs Python once for every group. |
| groupby-lambda-arithmetic | warning | unsafe | A lambda in transform that divides or subtracts a group total runs Python once for every group. |
| groupby-filter-lambda | warning | unsafe | groupby(...).filter(lambda g: len(g) > n) runs Python once for every group only to count its rows. |
| groupby-apply | warning | none | groupby(...).apply with a lambda runs a Python function once for every group. |
| aggregate-then-merge-back | warning | none | Merging a group total back onto its frame builds a second frame and joins it, when transform already gives each row the total of its group. |
| named-agg-lambda | warning | none | A lambda inside named aggregation runs Python once for every group. |
| groupby-loop-collects | warning | none | A loop over groupby that appends one summary for each group builds the result by hand. |
| groupby-first-skips-missing | warning | unsafe | groupby(...).first() and last() take the first value that is not missing in each column, so one result row can mix values from different rows. |
Tidy data
Section titled “Tidy data”Each rule in this group finds a longer way to reshape a frame, or a default of a reshaping method that changes values without a warning. A tidy frame has one row per observation and one column per variable. The measurements used pandas 3.1.1.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| column-loop-to-long | info | none | A loop that builds one frame per column and stacks them does what melt does in one call. |
| stack-then-reset-index | info | none | stack().reset_index() gives a long frame with columns named level_1 and 0. |
| melt-default-names | info | none | melt without var_name names the column of old column names variable, which says nothing about them. |
| pivot-table-default-mean | warning | none | pivot_table without aggfunc takes the mean of the rows that share a cell and says nothing, so two rows of 10 and 30 become 20.0. |
| unstack-without-fill | warning | unsafe | After unstack, a pair without rows becomes a missing value, which turns int64 counts into float64. |
Index alignment and joins
Section titled “Index alignment and joins”Each rule in this group finds code that turns off the pairing of rows by index label without a
warning, or code that pairs by index through a longer form. pandas pairs rows by index label in an
assignment, in concat(axis=1) and in a join. The measurements used pandas 3.1.1.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| assign-drops-alignment | warning | unsafe | Assigning .values, .to_numpy() or .tolist() of another frame drops its index, so pandas pairs the rows by position. |
| concat-columns-unchecked | info | none | concat(axis=1) pairs rows by index label and keeps every label of every frame, so one differing label adds a row of missing values and turns int columns into floats. |
| merge-on-both-indexes | info | unsafe | A merge on both indexes is a join, but the merge call hides that rows are paired by index label. |
| merge-asof-unsorted | warning | unsafe | merge_asof raises ValueError unless both frames are sorted by the key, and nothing here sorts them. |
MultiIndex
Section titled “MultiIndex”Each rule in this group finds a MultiIndex that the code builds or reads one row at a time, and names the method that works on whole levels.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| multiindex-from-tuples | warning | unsafe | MultiIndex.from_tuples with zip or product builds one Python tuple for every row. |
| level-values-unique | warning | unsafe | get_level_values(level).unique() builds one value for every row before it finds the unique ones. |
| tuple-slice-on-levels | info | safe | A tuple with slice(None) in .loc is hard to read. |
Time series
Section titled “Time series”Each rule in this group finds date and time code that works one value at a time, goes through
text, or breaks across a daylight saving change. The time zone rules were run on pandas 2.3.3 and
3.1.1 with quarter-hourly data in Europe/Copenhagen.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| rolling-apply | warning | unsafe | rolling(...).apply with a lambda or a NumPy function runs Python once for every window, which was up to 1,031 times slower. |
| text-time-key | warning | none | Grouping on dates formatted as text turns every date into text first. |
| object-dates | warning | none | dt.date and dt.time turn a datetime column into a column of Python objects, which is slow and loses the datetime methods. |
| apply-on-dates | warning | unsafe | Date arithmetic inside apply runs Python once for every date. |
| strftime-date-part | warning | unsafe | Formatting dates as text and parsing a number back out was 33 to 198 times slower than reading the part. |
| apply-date-expression | warning | none | Reading a date part inside apply and comparing it runs Python once for every value, and was 22 to 94 times slower. |
| strftime-round-trip | warning | unsafe | Formatting dates as text and parsing them again to drop the time was 18 to 202 times slower. |
| index-year-mask | warning | unsafe | Selecting a year with df.index.year == 2023 computes the year of every row. |
| apply-date-method | warning | unsafe | Calling a date method inside apply runs Python once for every value. |
| apply-parse-date | warning | unsafe | Parsing dates with strptime inside apply reads one text at a time, and was 24 times slower on a million rows. |
| timedelta-per-value | warning | unsafe | Building a Timedelta for every value in Python was 1,549 times slower than converting the column. |
| tz-localize-without-dst | warning | none | tz_localize raises on a time that repeats or is skipped at a daylight saving change unless you say what those times become. |
| date-range-then-localize | warning | unsafe | Calling tz_localize on a date_range builds the steps in local clock time and raises at a daylight saving change. |
| to-datetime-then-localize | warning | unsafe | tz_localize("UTC") after to_datetime raises on text that carries an offset. |
| convert-then-drop-zone | warning | safe | tz_convert("UTC").tz_localize(None) takes two steps to get UTC times without a time zone. |
| index-plus-offset | warning | unsafe | Adding an offset to the index by hand replaces the index in a separate step. |
| date-range-slice | warning | unsafe | Slicing off the last step of date_range with [:-1] does what inclusive="left" does. |
| index-time-mask | warning | unsafe | Selecting rows by index.time or index.hour builds a Python time or an hour for every row. |
| reindex-to-date-range | warning | unsafe | Reindexing to a date_range from the first to the last time fills in missing steps by hand. |
| divide-by-shift | warning | unsafe | Dividing a column by its own shift computes the change from the earlier row by hand. |
| date-part-key | warning | none | Grouping on a year and a month uses two keys for one month and leaves out months with no rows. |
| row-window-on-uneven-times | warning | none | A rolling window given as a number counts rows, even with on= naming a time column, so on days with gaps it reaches further back than you meant. |
Stated intent
Section titled “Stated intent”Each rule in this group asks for an argument that states what the code expects, so that pandas raises when the data doesn't match instead of giving a wrong result.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| repeated-reset-index | warning | none | A function that calls reset_index three times or more keeps moving keys between the index and the columns. |
| repeat-rows-by-index | warning | none | Repeating rows through index.repeat leaves a repeated index to reset and no column for the position. |
| numpy-round-trip | warning | none | A function that takes columns out with to_numpy three times or more and writes arrays back keeps the row order by hand. |
| frame-rebuilt-from-columns | warning | none | Building a new frame from three or more columns of another frame copies them one at a time and is easy to get out of step. |
| empty-guard | warning | unsafe | The mean, median and similar reductions of an empty Series are already NaN, with no error. |
| astype-then-to-numpy | warning | unsafe | astype builds a whole Series only for to_numpy to take its values. |
| merge-without-validate | warning | none | A merge without validate= duplicates rows without a warning when a key repeats. |
| merge-without-on | warning | none | Without on=, merge joins on every column name the two frames share, so a new shared column silently changes the join. |
| to-csv-index | warning | none | to_csv writes the index as a first column, which read_csv reads back as Unnamed: 0. |
Method chains
Section titled “Method chains”Each rule in this group asks for one chain of methods where the code builds a frame in separate
steps. A chain names the frame once and lists the steps in the order they run. No variable
holds a frame that is half done. The rules of this group are info, and xpd reports them as
warnings in a project with style = "strict" in its settings, which
xpd check describes.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| lambda-in-assign | info | safe | A lambda that only reads columns, as in assign(b=lambda d: d["a"] * 2), hides a simple expression. |
| reassign-ladder | info | none | Assigning a frame three or more times in a row from its own methods leaves a half-built frame under the final name after each line. |
| filter-ladder | info | none | Filtering a frame three or more times in a row, each time into itself, leaves a half-filtered frame under the final name after each line. |
| frame-column-write | info | none | frame["x"] = ... changes the frame it is called on, so a function that was given the frame changes the frame of its caller. |
| nested-step-calls | info | unsafe | Three functions called one inside the other read from the inside out, so the first step is written last. |
| script-steps | info | none | A module that builds its frames in ten or more statements at its top, with no function that takes a frame, runs only from its first line to its last. |
Charts
Section titled “Charts”Each rule in this group asks a file to draw with plotly as the plotting backend of pandas, so that
xpd can check the figure with the rules of its plotly pack. pandas draws df.plot with
matplotlib unless the plotting backend is set. With pd.options.plotting.backend = "plotly",
df.plot.bar(x="team", y="score") returned a figure equal, as JSON, to
px.bar(df, x="team", y="score"). The measurements used pandas 3.1.1 with plotly 7.1.0 and
matplotlib installed.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| plot-without-plotly-backend | info | unsafe | A file that draws with df.plot and sets no backend gets matplotlib charts that xpd's plotly rules cannot check. |
| plot-kind-argument | info | safe | df.plot(kind="bar") hides the kind in a text argument. |
| matplotlib-on-frame | info | none | Drawing frame columns with matplotlib or seaborn gives a chart that xpd's plotly rules cannot check. |
Reading and writing files
Section titled “Reading and writing files”Each rule in this group finds a call that reads or writes a file in a slower way than it needs to, or that writes a CSV file that reads back wrong.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| read-csv-slow-option | warning | unsafe | engine="python", sep=None and low_memory=False turn off the fast C reader of read_csv, and sep=None was 68 times slower. |
| read-then-head | warning | unsafe | read_csv(path).head(n) parses the whole file to keep a few rows. |
| read-csv-converters | warning | none | converters in read_csv calls a Python function once for every value, which was 3.6 to 5.1 times slower. |
| read-json-lines-engine | warning | unsafe | read_json(lines=True) without an engine parses one line at a time. |
| to-csv-append-header | error | none | to_csv(mode="a") writes the header again on every call, so header rows end up among the data. |
Parquet storage
Section titled “Parquet storage”Each rule in this group asks a file that stores a frame to use Parquet, or to read only the
columns and rows it needs from a Parquet file. to_parquet and read_parquet returned a frame
equal to the one written. The measurements used pandas 3.1.1 with pyarrow 26.0.1 and a frame of
1,000,000 rows with the dtypes int64, float64, str, datetime64[us], an ordered category
and Int64. The table gives the file size of the frame in each format.
| Format | File size |
|---|---|
| Parquet | 14.7 MB |
| CSV | 47.5 MB |
| pickle | 49.0 MB |
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| csv-round-trip | warning | none | Writing a frame to CSV and reading it back makes pandas guess every dtype again, so dates, categories and nullable integers come back as other types. |
| pickle-or-hdf-storage | info | none | read_pickle runs whatever code the file tells it to, so a pickle from someone else can run any code. |
| read-parquet-all-columns | warning | safe | Selecting columns after read_parquet reads every column of the file first. |
| read-parquet-then-filter | info | none | Filtering rows right after read_parquet reads every row of the file first. |
| chunked-csv-concat | warning | unsafe | Reading a CSV in chunks and joining every chunk keeps the whole file in memory anyway, and took 3 times as long as one read. |
Packages
Section titled “Packages”xpd check warns once per run about each package in the table that the project's venv lacks,
when the venv has pandas. The warning points at the nearest pyproject.toml, or at the venv
when there is none. A project without a venv gets no package warning, and xpd hook never gives
one.
| Check | Package | Extra condition |
|---|---|---|
missing-pyarrow |
pyarrow | none |
missing-bottleneck |
bottleneck | the checked code calls std, var, min, max or median |
missing-plotly |
plotly | the project uses matplotlib or seaborn |
missing-pandas-stubs |
pandas-stubs | the project uses mypy, pyright, basedpyright, ty or pyrefly |
A project uses a tool when the venv has it or pyproject.toml has a [tool.<name>] table for it.
The output shows the four warnings for a venv that holds pandas and matplotlib, next to a
pyproject.toml with a [tool.mypy] table, and a report.py that calls std on line 4.
$ xpd check --format text report.pypyproject.toml: warning [pandas/missing-pyarrow] pyarrow is not installed, so text columns are stored as Python strings. With pyarrow the .str methods ran 3 to 11 times faster on 1,000,000 rows. Add pyarrow to the project.pyproject.toml: warning [pandas/missing-bottleneck] bottleneck is not installed. pandas uses it for std, var, min, max and median. This code calls them first at report.py:4, and the number of such calls is 1. On 10,000,000 rows std ran 5.7 times faster and min 4.7 times. Add bottleneck to the project.pyproject.toml: warning [pandas/missing-plotly] The project has matplotlib and no plotly, so df.plot returns a static matplotlib Axes. With plotly installed and pd.options.plotting.backend = "plotly", df.plot.bar(x="team", y="score") returned the same figure as px.bar(df, x="team", y="score"), which xpd checks with its plotly rules. Add plotly to the project.pyproject.toml: warning [pandas/missing-pandas-stubs] The project has a type checker but not pandas-stubs, so calls to pandas are not type checked. Add pandas-stubs as a development dependency.xpd: 4 issuesnumexpr has no check, because it changed little in the measurements. numba has no check, because
it helps only a function passed to rolling(...).apply. The rolling-apply message names numba.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| missing-pyarrow | warning | none | pyarrow is not installed, so pandas stores text columns as Python strings and the .str methods run slower. |
| missing-bottleneck | warning | none | bottleneck is not installed, so pandas computes std, var, min, max and median without it. |
| missing-plotly | warning | none | The project has matplotlib and no plotly, so df.plot draws static matplotlib charts that xpd's plotly rules cannot check. |
| missing-pandas-stubs | warning | none | The project has a type checker but not pandas-stubs, so calls to pandas are not type checked. |