Skip to content

pandas/read-then-head

Rule read-then-head
Pack pandas
Severity warning
Fix unsafe, with xpd check --fix --unsafe-fixes
Library versions every version

read_csv(path).head(n) parses the whole file to keep a few rows. Pass nrows= to read_csv, which was 45 times faster.

sample = pd.read_csv(path).head(1000)
sample = pd.read_csv(path, nrows=1000)

The rule reports pd.read_csv(path).head(1000) and .iloc[:1000], which parse the whole file to keep the first rows. nrows was 45 times faster than head, measured with the reader's default threads on a 32-core machine.

xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.

The unsafe fix writes nrows=1000, nrows=5 for head() and nrows=n for iloc[:n] with a whole number. iloc[:n] with any other count has no fix.

The fix is unsafe, because pandas guesses the dtypes from the rows it reads, so a column can get another dtype.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

This reads and parses the whole file and keeps $N rows. Pass nrows=$N to read_csv.

Browse the other rules of the pandas pack.