Skip to content

pandas/read-parquet-all-columns

Rule read-parquet-all-columns
Pack pandas
Severity warning
Fix safe, with xpd check --fix
Library versions every version

Selecting columns after read_parquet reads every column of the file first. Pass columns= to read_parquet, which read 2 of 50 columns 9 times faster.

prices = pd.read_parquet(path)[["id", "price"]]
prices = pd.read_parquet(path, columns=["id", "price"])

The rule reports pd.read_parquet(path)[["id", "price"]] and pd.read_parquet(path)["price"], which read every column and keep the named ones. The names must be strings. The two frames were equal, with the columns in the order of the list, and a file with a named index gave the same index both ways.

Columns read with columns= Faster by
2 of 50 9.6 and 9.2 times in two runs, measured with the reader's default threads on a 32-core machine
2 of 6 1.7 times

xpd check --fix applies the fix of this rule.

The safe fix writes columns=["id", "price"].

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

This reads every column of the file and keeps $L. Pass columns=$L to read_parquet, which read 2 of 50 columns 9 times faster with the reader's default threads on a 32-core machine and returned an equal frame.

Browse the other rules of the pandas pack.