Skip to content

sklearn/stateful-function-transformer

Rule stateful-function-transformer
Pack scikit-learn
Severity info
Fix none, edit by hand
Library versions every version

A FunctionTransformer that computes statistics of its input scales test rows by test statistics, so a prediction changes with the other rows in the batch. Use StandardScaler or MinMaxScaler, which learn the statistics at fit.

pipe = make_pipeline(FunctionTransformer(lambda X: (X - X.mean(axis=0)) / X.std(axis=0)), model)
pipe = make_pipeline(StandardScaler(), model)

The rule reports FunctionTransformer(lambda X: (X - X.mean(axis=0)) / X.std(axis=0)), a function that takes statistics of the rows it gets. At predict time it then scales by the test rows.

xpd has no automatic fix, so make the change by hand.

The rule is info, because on data with no signal it doesn't move a score. Predicted in batches of 5 rows it scored 0.484 against 0.494 for StandardScaler. On data with signal, over 30 seeds, LogisticRegression scored lower behind the function:

Batch of test rows Behind the function Behind StandardScaler
5 0.773 0.847
20 0.824 0.847

A reduction along axis 1 works per row and is left alone. Recall was 24 of 30, and the rule found none of 6 functions defined with def and passed by name.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

This function takes statistics of the rows it gets, so at predict time it scales by the test rows, and a prediction changes with the other rows in the batch. Use StandardScaler or MinMaxScaler, which learn the statistics at fit.

Browse the other rules of the scikit-learn pack.