Skip to content

pandas/read-json-lines-engine

Rule read-json-lines-engine
Pack pandas
Severity warning
Fix unsafe, with xpd check --fix --unsafe-fixes
Library versions every version

read_json(lines=True) without an engine parses one line at a time. Pass engine="pyarrow", which read the file 25 times faster.

df = pd.read_json(path, lines=True)
df = pd.read_json(path, lines=True, engine="pyarrow")

The rule reports pd.read_json(path, lines=True) without an engine, and pd.DataFrame([json.loads(line) for line in f]). read_json with the pyarrow engine was 25 times faster than without an engine, and 48 times faster than the json.loads loop, measured with the reader's default threads on a 32-core machine. The rule leaves a call alone that reads from a StringIO, because the pyarrow engine raises on a text buffer.

xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.

The unsafe fix writes engine="pyarrow". The json.loads loop has no fix.

The fix is unsafe, because pyarrow can pick another dtype for a column.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

read_json with lines=True parses one line at a time. Pass engine="pyarrow" to parse the file on several threads.

Browse the other rules of the pandas pack.