Skip to content

scikit-learn

The scikit-learn pack has 32 rules, and xpd can fix 17 of them.

The scikit-learn rules report code that gives a leaked or wrong score, code that raises or warns on scikit-learn 1.9.1, and code that does the same work twice, in each file that imports sklearn. Every number about the scikit-learn rules was measured on scikit-learn 1.9.1. The examples assume that the scikit-learn names they use are imported, and names such as X_train, y_test and model stand for values from your program.

Several rules read the names of variables, such as X_test and y_pred, because xpd reads the code and never the values it holds. A name that misleads therefore gives a wrong report.

A rule document whose advice needs a newer release of scikit-learn carries since. xpd reads the installed version from the project's venv and skips the document on an older release. When the venv has no scikit-learn, or there is no venv, every document runs.

Each rule document ships only with a saved proof that its old form is wrong. A proof is one of the following kinds:

Proof What runs When it passes
score The old and the new form, over 30 seeds on data with no signal, where the honest score is 0.5. The old form's mean is more than three standard errors above the honest score, and the new form's mean is within 0.02 of it.
interpreter The old form under python -W error. The old form raises an exception or a warning, and the new form runs clean.
speed Both forms on one thread, as the median of 7 runs, at two data sizes. The new form is at least 1.5 times faster at both sizes.

A document is info when no proof passed for it. Such a document either changes a score in a way the proofs can't show, or changes nothing that can be measured. For example, a ranking metric given class labels scores 0.5 on data with no signal in both forms, and a missing seed changes nothing that can be measured.

The rules were then judged on 1,414 held-back files from 268 public repositories, which were set aside before any rule was written. The judging had the following parts:

  1. Two readers who did not write the rules labelled the same 478 reports, each without the other, and marked every report whose advice was wrong.
  2. Recall was measured by planting 2,749 old forms in copies of the held-back files and counting the plants that a rule reported.
  3. Every fix was run before and after, on its held-back sites and on cases built for each form, and the fitted model and its predictions were compared bit for bit.
Rule Severity Proof Held-back reports Wrong advice Recall Fix
fit-on-test error, info score: 0.997 against 0.525 AUC 20 0 and 0 of 20 103 of 130 unsafe
fit-before-split error score: 0.771 against 0.497 AUC 24 0 and 0 of 24 91 of 162 none
cross-val-on-fitted-data error score: 0.764 against 0.518 AUC 10 0 and 0 of 10 72 of 78 none
tune-on-test error score: 0.597 against 0.507 accuracy 1 0 and 0 of 1 15 of 50 none
stateful-function-transformer info none of the three 0 no reports 24 of 30 none
groups-ignored error score: 0.849 against 0.480 accuracy 0 no reports 56 of 57 unsafe
missing-groups error interpreter: ValueError 0 no reports 48 of 50 none
leave-one-out-score error interpreter: UndefinedMetricWarning 0 no reports 14 of 33 none
auc-from-labels error, info score: log loss 18.00 against 0.70 20 2 and 2 of 20 55 of 72 unsafe
metric-arguments-swapped error, info score: precision 0.471 against 0.294 43 0 and 0 of 30 66 of 100 unsafe
score-on-training-rows error score: 1.000 against 0.510 accuracy 0 no reports 10 of 27 none
unknown-scoring-name error interpreter: InvalidParameterError, ValueError 4 0 and 0 of 4 16 of 32 none
removed-in-sklearn error interpreter: ModuleNotFoundError, ImportError, TypeError, InvalidParameterError 321 1 and 0 of 58 517 of 600 unsafe
deprecated-in-sklearn warning interpreter: FutureWarning 101 3 and 0 of 57 274 of 308 safe, unsafe
removed-metric-api error interpreter: ImportError, TypeError 13 0 and 0 of 13 74 of 74 unsafe
deprecated-metric-api warning interpreter: FutureWarning 0 no reports 48 of 48 safe
fit-params-keyword error interpreter: TypeError 4 0 and 0 of 4 32 of 32 unsafe
column-transformer-removed error interpreter: TypeError 3 0 and 0 of 3 30 of 30 unsafe
metadata-routing error interpreter: RuntimeError, ValueError, TypeError, UnsetMetadataPassedError 0 no reports 34 of 34 unsafe
pipeline-fit-params error interpreter: ValueError 0 no reports 23 of 30 none
pipeline-step-form error interpreter: ValueError, TypeError 0 no reports 29 of 35 unsafe
transformed-target-pair error interpreter: ValueError 0 no reports 31 of 36 unsafe
pandas-output-sparse error interpreter: ValueError 0 no reports 29 of 31 unsafe
predict-per-row warning speed: 2.77 times 2 0 and 2 of 2 28 of 32 none
score-after-predict warning speed: 1.62 times 17 0 and 0 of 17 31 of 32 unsafe
predict-and-predict-proba warning speed: 1.83 times 22 0 and 0 of 22 30 of 30 none
cross-val-score-per-metric warning speed: 1.60 times 14 0 and 0 of 14 30 of 30 none
vectorizer-fit-then-transform warning speed: 1.63 times 12 0 and 4 of 12 19 of 25 unsafe
refit-per-n-estimators warning speed: 1.64 times 0 no reports 30 of 30 none
tsne-exact info none of the three 0 no reports 21 of 30 unsafe
random-state-without-shuffle error interpreter: ValueError 9 0 and 0 of 9 54 of 54 none
missing-random-state info none of the three 527 1 and 1 of 52 79 of 80 none
Column What it gives
Proof A score proof gives the old form's mean against the new form's mean. A speed proof gives the lowest ratio over the classes the rule reports and both sizes.
Wrong advice The reports that reader A and reader B called wrong, out of the reports they labelled. A rule with more reports than labels was sampled. A rule with no report on the held-back files has no number.
Recall The plants that the rule reported, out of the plants made for it.

The rules were narrowed after the readers' labels came in. For a narrowed rule, wrong advice counts the labelled reports that the rule still makes and that are still wrong, so the number is not a clean measurement. Recall was measured on the rules before that narrowing.

Recall on code in other projects is likely lower than the table says. Most plants were inserted into a file in shapes the judges chose, and such shapes are closer to the documented examples than the code in other projects is. For the metric and pipeline rules, recall was 190 of 280 on plants turned from a right form already in the file, against 311 of 357 on inserted plants.

Each rule in the group reports a form that lets the test rows, or every row, shape what the model learns, so the score on the test rows is too high. The score proofs use data whose target is drawn apart from every feature, where an honest AUC is 0.5, and give the mean over 30 seeds.

A step that doesn't read the target moved the test AUC by at most 0.006 when it was fitted before the split, so the rules leave it alone. Such steps are a scaler, an imputer, PCA, a binner, a one-hot encoder and a text vectorizer.

Rule Severity Fix What it finds
fit-on-test error unsafe Fitting on the test rows makes a score on those rows meaningless, and a model fitted this way scored 0.997 AUC on data with no signal.
fit-before-split error none A step that learned from the target on every row before the split lets the test rows choose what the model sees, so the test score is too high.
cross-val-on-fitted-data error none A step that learned from the target on every row before cross-validation lets each fold be scored on rows whose labels chose the features.
tune-on-test error none Picking a setting by its score on the test rows makes the best score too high.
stateful-function-transformer info none A FunctionTransformer that computes statistics of its input scales test rows by test statistics, so a prediction changes with the other rows in the batch.

Each rule in the group reports a split of the rows that leaks or gives no score.

Rule Severity Fix What it finds
groups-ignored error unsafe The chosen cv ignores groups=, so rows of one group land in both the training and the test folds, and the score is too high.
missing-groups error none A group splitter without groups= raises ValueError.
leave-one-out-score error none With LeaveOneOut each fold tests one row, and scores such as R2 or ROC AUC need more than one row, so every fold scores nan.

Each rule in the group reports a metric given the wrong values, a score on the wrong rows, or a scorer name that doesn't exist.

Rule Severity Fix What it finds
auc-from-labels error unsafe A metric that ranks rows by score, such as roc_auc_score or log_loss, gets class labels from predict, which leaves only two values to rank.
metric-arguments-swapped error unsafe Metrics take the true values first and the predictions second, so swapping them gives another number, such as the recall in place of the precision.
score-on-training-rows error none Scoring a model on the rows it was fitted on says nothing about new data, and a decision tree scored 1.00 this way on data with no signal.
unknown-scoring-name error none A scoring text that is not a scorer name raises.

Each rule in the group reports a form that raises or warns on scikit-learn 1.9.1. Each form was run under python -W error, where the old form raised and the new form ran clean. The messages take each replacement from the 1.9.1 source or from the error text, such as the docstring that says sparse was renamed sparse_output in 1.2.

A fix of a removed form turns code that raises into code that runs, so no such fix can be bit equal or safe. A fix that drops a keyword deletes the comments inside the argument list, and it joins a call written over several lines onto one line.

Rule Severity Fix What it finds
removed-in-sklearn error unsafe These modules, names and arguments were removed from scikit-learn, so the code raises.
deprecated-in-sklearn warning unsafe These forms warn FutureWarning on scikit-learn 1.9.1 because a later release removes them or changes their default.
removed-metric-api error unsafe These metric names were removed from scikit-learn, so the code raises.
deprecated-metric-api warning safe These metric arguments and names warn on scikit-learn 1.9.1 and go away in a later release.
fit-params-keyword error unsafe fit_params= was removed and raises TypeError on scikit-learn 1.9.
column-transformer-removed error unsafe ColumnTransformer no longer takes force_int_remainder_cols, so the call raises TypeError on scikit-learn 1.9.

Each rule in the group reports a form that raises at construction or at fit on scikit-learn 1.9.1. No held-back file uses metadata routing, a misnamed step, TransformedTargetRegressor in a wrong mix, or pandas output on a sparse encoder. The rules in the group therefore made no report there and have no number for wrong advice.

Rule Severity Fix What it finds
metadata-routing error unsafe These forms work only with metadata routing on, or only with it off, so the call raises RuntimeError or ValueError here.
pipeline-fit-params error none Pipeline.fit raises ValueError on a bare sample_weight=, because it takes step parameters as the step name, two underscores and the name.
pipeline-step-form error unsafe Step names with two underscores in a row, and Pipelines built from the wrong kind of argument, raise ValueError at fit.
transformed-target-pair error unsafe TransformedTargetRegressor raises ValueError at fit when it gets func without inverse_func, or both a transformer and functions.
pandas-output-sparse error unsafe OneHotEncoder returns a sparse matrix by default, and pandas output raises ValueError on it.

Each rule in the group reports work that scikit-learn already did, and the new form does it once. The numbers are speed proofs on scikit-learn 1.9.1, on one thread, as the median of 7 runs of each form at two data sizes. A rule reports only the classes where both sizes passed 1.5 times.

Rule Severity Fix What it finds
predict-per-row warning none Calling predict once per row checks the input and sets up the model again for every row.
score-after-predict warning unsafe model.score(X, y) predicts X again when you already hold the predictions.
predict-and-predict-proba warning none predict computes the same probabilities as predict_proba and returns the class with the largest one, so calling both does the work twice.
cross-val-score-per-metric warning none Each cross_val_score call fits the model on every fold again to compute one more score.
vectorizer-fit-then-transform warning unsafe fit already builds the matrix of the documents and drops it, so transform reads them a second time.
refit-per-n-estimators warning none Fitting a new forest for each tree count in a loop grows every tree again.
tsne-exact info unsafe TSNE(method="exact") computes the force between every pair of rows, and the default method gave the same quality 4.5 to 16 times faster.

Pass random_state wherever scikit-learn shuffles or draws, so two runs give the same split and the same score.

Rule Severity Fix What it finds
random-state-without-shuffle error none KFold uses random_state only when it shuffles, so without shuffle=True the call raises ValueError.
missing-random-state info none Without random_state, a split or a model takes a new seed on every run, so the score changes from run to run.