Skip to content

pandas/pickle-or-hdf-storage

Rule pickle-or-hdf-storage
Pack pandas
Severity info
Fix none, edit by hand
Library versions every version

read_pickle runs whatever code the file tells it to, so a pickle from someone else can run any code. Store frames as Parquet, which holds data only and kept every dtype in a smaller file.

df.to_pickle("clean.pkl")
again = pd.read_pickle("clean.pkl")
df.to_parquet("clean.parquet")
again = pd.read_parquet("clean.parquet")

The rule reports to_pickle, pd.read_pickle, to_hdf, pd.read_hdf and pd.HDFStore. read_pickle runs the function that the file names. For example, a pickle that named print made pd.read_pickle print a line. HDF needs the PyTables package, and without it to_hdf raised ImportError. Parquet holds data only, and it needs pyarrow.

xpd has no automatic fix, so make the change by hand.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

read_pickle runs what the file tells it to call, so a pickle from someone else can run any code. Parquet holds data only. to_parquet and read_parquet returned a frame equal to the one written, with every dtype, in a file 3.3 times smaller than the pickle, on 1,000,000 rows.

Browse the other rules of the pandas pack.