Skip to content

pandas/read-parquet-then-filter

Rule read-parquet-then-filter
Pack pandas
Severity info
Fix none, edit by hand
Library versions every version

Filtering rows right after read_parquet reads every row of the file first. Pass the test as filters=, so pyarrow skips the parts of the file that hold no matching row.

df = pd.read_parquet(path)
df = df[df["id"] >= 900_000]
df = pd.read_parquet(path, filters=[("id", ">=", 900_000)])

The rule reports df = pd.read_parquet(path) followed directly by df = df[df["id"] >= value], with >, >=, <, <=, ==, != or isin. With filters=, pyarrow skips each row group that holds no matching row, and the values were equal. A row group is a block of rows that Parquet stores together. The rule has no fix, because the index can differ. A frame without a stored index came back numbered from 0, where the mask kept the old labels.

File, filtered to the last tenth of a sorted column Faster by
10 row groups 5.3 and 4.6 times in two runs, measured with the reader's default threads on a 32-core machine
1 row group 1.3 to 1.5 times

xpd has no automatic fix, so make the change by hand.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

This reads every row of the file and then keeps the rows where $C passes the test. Pass the test to read_parquet, as in filters=[("id", ">=", 900000)], which skips the row groups that hold no such row. It read the last tenth of a file with 10 row groups 5 times faster with the reader's default threads on a 32-core machine. The rows come back numbered from 0 when the file stores no index, where the mask kept their old labels.

Browse the other rules of the pandas pack.