pandas/csv-round-trip
| Rule | csv-round-trip |
|---|---|
| Pack | pandas |
| Severity | warning |
| Fix | none, edit by hand |
| Library versions | every version |
Writing a frame to CSV and reading it back makes pandas guess every dtype again, so dates, categories and nullable integers come back as other types. Use to_parquet and read_parquet, which kept the frame equal in a smaller file.
df.to_csv("clean.csv", index=False)again = pd.read_csv("clean.csv")df.to_parquet("clean.parquet")again = pd.read_parquet("clean.parquet")What the rule reports
Section titled “What the rule reports”The rule reports df.to_csv(path) in a file that also calls pd.read_csv(path) on the same path, written the same way in both calls. read_csv guesses every dtype again, and the table shows the dtypes that changed. to_csv took 0.93 seconds and to_parquet took 0.09 seconds.
| Column written | Read back from CSV |
|---|---|
datetime64[us] |
str |
| ordered category | str |
Int64 |
float64 |
The rule leaves a file alone that only writes a CSV, because another program may read it. It also leaves to_csv with mode= alone, which appends to a file.
How to fix it
Section titled “How to fix it”xpd has no automatic fix, so make the change by hand.
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
This file writes $P with to_csv and reads it back with read_csv, which guesses every dtype again. A datetime64 column and an ordered category came back as str, and a nullable Int64 column as float64. to_parquet and read_parquet returned a frame equal to the one written, in a file 3.2 times smaller, on 1,000,000 rows.Browse the other rules of the pandas pack.