pandas/pickle-or-hdf-storage
| Rule | pickle-or-hdf-storage |
|---|---|
| Pack | pandas |
| Severity | info |
| Fix | none, edit by hand |
| Library versions | every version |
read_pickle runs whatever code the file tells it to, so a pickle from someone else can run any code. Store frames as Parquet, which holds data only and kept every dtype in a smaller file.
df.to_pickle("clean.pkl")again = pd.read_pickle("clean.pkl")df.to_parquet("clean.parquet")again = pd.read_parquet("clean.parquet")What the rule reports
Section titled “What the rule reports”The rule reports to_pickle, pd.read_pickle, to_hdf, pd.read_hdf and pd.HDFStore. read_pickle runs the function that the file names. For example, a pickle that named print made pd.read_pickle print a line. HDF needs the PyTables package, and without it to_hdf raised ImportError. Parquet holds data only, and it needs pyarrow.
How to fix it
Section titled “How to fix it”xpd has no automatic fix, so make the change by hand.
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
read_pickle runs what the file tells it to call, so a pickle from someone else can run any code. Parquet holds data only. to_parquet and read_parquet returned a frame equal to the one written, with every dtype, in a file 3.3 times smaller than the pickle, on 1,000,000 rows.Browse the other rules of the pandas pack.