Skip to content

pandas/chunked-csv-concat

Rule chunked-csv-concat
Pack pandas
Severity warning
Fix unsafe, with xpd check --fix --unsafe-fixes
Library versions every version

Reading a CSV in chunks and joining every chunk keeps the whole file in memory anyway, and took 3 times as long as one read. Drop chunksize, or reduce each chunk before you join them.

df = pd.concat(pd.read_csv(path, chunksize=100_000))
df = pd.read_csv(path)

The rule reports pd.concat(pd.read_csv(path, chunksize=n)), also with list(...) or a comprehension around the reader. The code joins every chunk again, so the whole file is in memory anyway. The two frames were equal, and the table shows how long the chunked read took in two runs against one read_csv without chunksize.

Threads of the reader Time against one read_csv
the default, on a 32-core machine 3.1 and 2.9 times as long
one 1.1 and 1.0 times as long

The rule leaves a loop alone that reduces each chunk before the concat.

xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.

The unsafe fix writes pd.read_csv(path) without chunksize, when concat takes nothing but the chunks and ignore_index=True. pandas guesses the dtypes per chunk, so a column with mixed values can differ. A concat with other arguments, such as keys=, has no fix.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

This reads the file in chunks and joins every chunk, so the whole file is in memory anyway. With the reader's default threads on a 32-core machine it took 3 times as long as one read_csv without chunksize, which returned an equal frame, and on one thread it took as long. Drop chunksize, or reduce each chunk before the concat. To read only some columns or rows, store the data as Parquet and pass columns= or filters= to read_parquet.

Browse the other rules of the pandas pack.