pandas/read-parquet-then-filter
| Rule | read-parquet-then-filter |
|---|---|
| Pack | pandas |
| Severity | info |
| Fix | none, edit by hand |
| Library versions | every version |
Filtering rows right after read_parquet reads every row of the file first. Pass the test as filters=, so pyarrow skips the parts of the file that hold no matching row.
df = pd.read_parquet(path)df = df[df["id"] >= 900_000]df = pd.read_parquet(path, filters=[("id", ">=", 900_000)])What the rule reports
Section titled “What the rule reports”The rule reports df = pd.read_parquet(path) followed directly by df = df[df["id"] >= value], with >, >=, <, <=, ==, != or isin. With filters=, pyarrow skips each row group that holds no matching row, and the values were equal. A row group is a block of rows that Parquet stores together. The rule has no fix, because the index can differ. A frame without a stored index came back numbered from 0, where the mask kept the old labels.
| File, filtered to the last tenth of a sorted column | Faster by |
|---|---|
| 10 row groups | 5.3 and 4.6 times in two runs, measured with the reader's default threads on a 32-core machine |
| 1 row group | 1.3 to 1.5 times |
How to fix it
Section titled “How to fix it”xpd has no automatic fix, so make the change by hand.
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
This reads every row of the file and then keeps the rows where $C passes the test. Pass the test to read_parquet, as in filters=[("id", ">=", 900000)], which skips the row groups that hold no such row. It read the last tenth of a file with 10 row groups 5 times faster with the reader's default threads on a 32-core machine. The rows come back numbered from 0 when the file stores no index, where the mask kept their old labels.Browse the other rules of the pandas pack.