pandas/chunked-csv-concat
| Rule | chunked-csv-concat |
|---|---|
| Pack | pandas |
| Severity | warning |
| Fix | unsafe, with xpd check --fix --unsafe-fixes |
| Library versions | every version |
Reading a CSV in chunks and joining every chunk keeps the whole file in memory anyway, and took 3 times as long as one read. Drop chunksize, or reduce each chunk before you join them.
df = pd.concat(pd.read_csv(path, chunksize=100_000))df = pd.read_csv(path)What the rule reports
Section titled “What the rule reports”The rule reports pd.concat(pd.read_csv(path, chunksize=n)), also with list(...) or a comprehension around the reader. The code joins every chunk again, so the whole file is in memory anyway. The two frames were equal, and the table shows how long the chunked read took in two runs against one read_csv without chunksize.
| Threads of the reader | Time against one read_csv |
|---|---|
| the default, on a 32-core machine | 3.1 and 2.9 times as long |
| one | 1.1 and 1.0 times as long |
The rule leaves a loop alone that reduces each chunk before the concat.
How to fix it
Section titled “How to fix it”xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.
The unsafe fix writes pd.read_csv(path) without chunksize, when concat takes nothing but the chunks and ignore_index=True. pandas guesses the dtypes per chunk, so a column with mixed values can differ. A concat with other arguments, such as keys=, has no fix.
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
This reads the file in chunks and joins every chunk, so the whole file is in memory anyway. With the reader's default threads on a 32-core machine it took 3 times as long as one read_csv without chunksize, which returned an equal frame, and on one thread it took as long. Drop chunksize, or reduce each chunk before the concat. To read only some columns or rows, store the data as Parquet and pass columns= or filters= to read_parquet.Browse the other rules of the pandas pack.