scikit-learn
The scikit-learn pack has 32 rules, and xpd can fix 17 of them.
The scikit-learn rules report code that gives a leaked or wrong score, code that raises or warns
on scikit-learn 1.9.1, and code that does the same work twice, in each file that imports
sklearn. Every number about the scikit-learn rules was measured on scikit-learn 1.9.1. The examples assume
that the scikit-learn names they use are imported, and names such as X_train, y_test and
model stand for values from your program.
Several rules read the names of variables, such as X_test and y_pred, because xpd reads the
code and never the values it holds. A name that misleads therefore gives a wrong report.
A rule document whose advice needs a newer release of scikit-learn carries since. xpd reads the
installed version from the project's venv and skips the document on an older release. When the
venv has no scikit-learn, or there is no venv, every document runs.
The proof each rule needs
Section titled “The proof each rule needs”Each rule document ships only with a saved proof that its old form is wrong. A proof is one of the following kinds:
| Proof | What runs | When it passes |
|---|---|---|
| score | The old and the new form, over 30 seeds on data with no signal, where the honest score is 0.5. | The old form's mean is more than three standard errors above the honest score, and the new form's mean is within 0.02 of it. |
| interpreter | The old form under python -W error. |
The old form raises an exception or a warning, and the new form runs clean. |
| speed | Both forms on one thread, as the median of 7 runs, at two data sizes. | The new form is at least 1.5 times faster at both sizes. |
A document is info when no proof passed for it. Such a document either changes a score in a
way the proofs can't show, or changes nothing that can be measured. For example, a ranking metric
given class labels scores 0.5 on data with no signal in both forms, and a missing seed changes
nothing that can be measured.
The rules were then judged on 1,414 held-back files from 268 public repositories, which were set aside before any rule was written. The judging had the following parts:
- Two readers who did not write the rules labelled the same 478 reports, each without the other, and marked every report whose advice was wrong.
- Recall was measured by planting 2,749 old forms in copies of the held-back files and counting the plants that a rule reported.
- Every fix was run before and after, on its held-back sites and on cases built for each form, and the fitted model and its predictions were compared bit for bit.
| Rule | Severity | Proof | Held-back reports | Wrong advice | Recall | Fix |
|---|---|---|---|---|---|---|
fit-on-test |
error, info | score: 0.997 against 0.525 AUC | 20 | 0 and 0 of 20 | 103 of 130 | unsafe |
fit-before-split |
error | score: 0.771 against 0.497 AUC | 24 | 0 and 0 of 24 | 91 of 162 | none |
cross-val-on-fitted-data |
error | score: 0.764 against 0.518 AUC | 10 | 0 and 0 of 10 | 72 of 78 | none |
tune-on-test |
error | score: 0.597 against 0.507 accuracy | 1 | 0 and 0 of 1 | 15 of 50 | none |
stateful-function-transformer |
info | none of the three | 0 | no reports | 24 of 30 | none |
groups-ignored |
error | score: 0.849 against 0.480 accuracy | 0 | no reports | 56 of 57 | unsafe |
missing-groups |
error | interpreter: ValueError | 0 | no reports | 48 of 50 | none |
leave-one-out-score |
error | interpreter: UndefinedMetricWarning | 0 | no reports | 14 of 33 | none |
auc-from-labels |
error, info | score: log loss 18.00 against 0.70 | 20 | 2 and 2 of 20 | 55 of 72 | unsafe |
metric-arguments-swapped |
error, info | score: precision 0.471 against 0.294 | 43 | 0 and 0 of 30 | 66 of 100 | unsafe |
score-on-training-rows |
error | score: 1.000 against 0.510 accuracy | 0 | no reports | 10 of 27 | none |
unknown-scoring-name |
error | interpreter: InvalidParameterError, ValueError | 4 | 0 and 0 of 4 | 16 of 32 | none |
removed-in-sklearn |
error | interpreter: ModuleNotFoundError, ImportError, TypeError, InvalidParameterError | 321 | 1 and 0 of 58 | 517 of 600 | unsafe |
deprecated-in-sklearn |
warning | interpreter: FutureWarning | 101 | 3 and 0 of 57 | 274 of 308 | safe, unsafe |
removed-metric-api |
error | interpreter: ImportError, TypeError | 13 | 0 and 0 of 13 | 74 of 74 | unsafe |
deprecated-metric-api |
warning | interpreter: FutureWarning | 0 | no reports | 48 of 48 | safe |
fit-params-keyword |
error | interpreter: TypeError | 4 | 0 and 0 of 4 | 32 of 32 | unsafe |
column-transformer-removed |
error | interpreter: TypeError | 3 | 0 and 0 of 3 | 30 of 30 | unsafe |
metadata-routing |
error | interpreter: RuntimeError, ValueError, TypeError, UnsetMetadataPassedError | 0 | no reports | 34 of 34 | unsafe |
pipeline-fit-params |
error | interpreter: ValueError | 0 | no reports | 23 of 30 | none |
pipeline-step-form |
error | interpreter: ValueError, TypeError | 0 | no reports | 29 of 35 | unsafe |
transformed-target-pair |
error | interpreter: ValueError | 0 | no reports | 31 of 36 | unsafe |
pandas-output-sparse |
error | interpreter: ValueError | 0 | no reports | 29 of 31 | unsafe |
predict-per-row |
warning | speed: 2.77 times | 2 | 0 and 2 of 2 | 28 of 32 | none |
score-after-predict |
warning | speed: 1.62 times | 17 | 0 and 0 of 17 | 31 of 32 | unsafe |
predict-and-predict-proba |
warning | speed: 1.83 times | 22 | 0 and 0 of 22 | 30 of 30 | none |
cross-val-score-per-metric |
warning | speed: 1.60 times | 14 | 0 and 0 of 14 | 30 of 30 | none |
vectorizer-fit-then-transform |
warning | speed: 1.63 times | 12 | 0 and 4 of 12 | 19 of 25 | unsafe |
refit-per-n-estimators |
warning | speed: 1.64 times | 0 | no reports | 30 of 30 | none |
tsne-exact |
info | none of the three | 0 | no reports | 21 of 30 | unsafe |
random-state-without-shuffle |
error | interpreter: ValueError | 9 | 0 and 0 of 9 | 54 of 54 | none |
missing-random-state |
info | none of the three | 527 | 1 and 1 of 52 | 79 of 80 | none |
| Column | What it gives |
|---|---|
| Proof | A score proof gives the old form's mean against the new form's mean. A speed proof gives the lowest ratio over the classes the rule reports and both sizes. |
| Wrong advice | The reports that reader A and reader B called wrong, out of the reports they labelled. A rule with more reports than labels was sampled. A rule with no report on the held-back files has no number. |
| Recall | The plants that the rule reported, out of the plants made for it. |
The rules were narrowed after the readers' labels came in. For a narrowed rule, wrong advice counts the labelled reports that the rule still makes and that are still wrong, so the number is not a clean measurement. Recall was measured on the rules before that narrowing.
Recall on code in other projects is likely lower than the table says. Most plants were inserted into a file in shapes the judges chose, and such shapes are closer to the documented examples than the code in other projects is. For the metric and pipeline rules, recall was 190 of 280 on plants turned from a right form already in the file, against 311 of 357 on inserted plants.
Leakage
Section titled “Leakage”Each rule in the group reports a form that lets the test rows, or every row, shape what the model learns, so the score on the test rows is too high. The score proofs use data whose target is drawn apart from every feature, where an honest AUC is 0.5, and give the mean over 30 seeds.
A step that doesn't read the target moved the test AUC by at most 0.006 when it was fitted before the split, so the rules leave it alone. Such steps are a scaler, an imputer, PCA, a binner, a one-hot encoder and a text vectorizer.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| fit-on-test | error | unsafe | Fitting on the test rows makes a score on those rows meaningless, and a model fitted this way scored 0.997 AUC on data with no signal. |
| fit-before-split | error | none | A step that learned from the target on every row before the split lets the test rows choose what the model sees, so the test score is too high. |
| cross-val-on-fitted-data | error | none | A step that learned from the target on every row before cross-validation lets each fold be scored on rows whose labels chose the features. |
| tune-on-test | error | none | Picking a setting by its score on the test rows makes the best score too high. |
| stateful-function-transformer | info | none | A FunctionTransformer that computes statistics of its input scales test rows by test statistics, so a prediction changes with the other rows in the batch. |
Splits and groups
Section titled “Splits and groups”Each rule in the group reports a split of the rows that leaks or gives no score.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| groups-ignored | error | unsafe | The chosen cv ignores groups=, so rows of one group land in both the training and the test folds, and the score is too high. |
| missing-groups | error | none | A group splitter without groups= raises ValueError. |
| leave-one-out-score | error | none | With LeaveOneOut each fold tests one row, and scores such as R2 or ROC AUC need more than one row, so every fold scores nan. |
Metrics and scoring
Section titled “Metrics and scoring”Each rule in the group reports a metric given the wrong values, a score on the wrong rows, or a scorer name that doesn't exist.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| auc-from-labels | error | unsafe | A metric that ranks rows by score, such as roc_auc_score or log_loss, gets class labels from predict, which leaves only two values to rank. |
| metric-arguments-swapped | error | unsafe | Metrics take the true values first and the predictions second, so swapping them gives another number, such as the recall in place of the precision. |
| score-on-training-rows | error | none | Scoring a model on the rows it was fitted on says nothing about new data, and a decision tree scored 1.00 this way on data with no signal. |
| unknown-scoring-name | error | none | A scoring text that is not a scorer name raises. |
Removed and deprecated forms
Section titled “Removed and deprecated forms”Each rule in the group reports a form that raises or warns on scikit-learn 1.9.1. Each form was
run under python -W error, where the old form raised and the new form ran clean. The messages
take each replacement from the 1.9.1 source or from the error text, such as the docstring that
says sparse was renamed sparse_output in 1.2.
A fix of a removed form turns code that raises into code that runs, so no such fix can be bit equal or safe. A fix that drops a keyword deletes the comments inside the argument list, and it joins a call written over several lines onto one line.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| removed-in-sklearn | error | unsafe | These modules, names and arguments were removed from scikit-learn, so the code raises. |
| deprecated-in-sklearn | warning | unsafe | These forms warn FutureWarning on scikit-learn 1.9.1 because a later release removes them or changes their default. |
| removed-metric-api | error | unsafe | These metric names were removed from scikit-learn, so the code raises. |
| deprecated-metric-api | warning | safe | These metric arguments and names warn on scikit-learn 1.9.1 and go away in a later release. |
| fit-params-keyword | error | unsafe | fit_params= was removed and raises TypeError on scikit-learn 1.9. |
| column-transformer-removed | error | unsafe | ColumnTransformer no longer takes force_int_remainder_cols, so the call raises TypeError on scikit-learn 1.9. |
Pipeline and metadata routing
Section titled “Pipeline and metadata routing”Each rule in the group reports a form that raises at construction or at fit on scikit-learn
1.9.1. No held-back file uses metadata routing, a misnamed step, TransformedTargetRegressor in a
wrong mix, or pandas output on a sparse encoder. The rules in the group therefore made no report
there and have no number for wrong advice.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| metadata-routing | error | unsafe | These forms work only with metadata routing on, or only with it off, so the call raises RuntimeError or ValueError here. |
| pipeline-fit-params | error | none | Pipeline.fit raises ValueError on a bare sample_weight=, because it takes step parameters as the step name, two underscores and the name. |
| pipeline-step-form | error | unsafe | Step names with two underscores in a row, and Pipelines built from the wrong kind of argument, raise ValueError at fit. |
| transformed-target-pair | error | unsafe | TransformedTargetRegressor raises ValueError at fit when it gets func without inverse_func, or both a transformer and functions. |
| pandas-output-sparse | error | unsafe | OneHotEncoder returns a sparse matrix by default, and pandas output raises ValueError on it. |
Work done twice
Section titled “Work done twice”Each rule in the group reports work that scikit-learn already did, and the new form does it once. The numbers are speed proofs on scikit-learn 1.9.1, on one thread, as the median of 7 runs of each form at two data sizes. A rule reports only the classes where both sizes passed 1.5 times.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| predict-per-row | warning | none | Calling predict once per row checks the input and sets up the model again for every row. |
| score-after-predict | warning | unsafe | model.score(X, y) predicts X again when you already hold the predictions. |
| predict-and-predict-proba | warning | none | predict computes the same probabilities as predict_proba and returns the class with the largest one, so calling both does the work twice. |
| cross-val-score-per-metric | warning | none | Each cross_val_score call fits the model on every fold again to compute one more score. |
| vectorizer-fit-then-transform | warning | unsafe | fit already builds the matrix of the documents and drops it, so transform reads them a second time. |
| refit-per-n-estimators | warning | none | Fitting a new forest for each tree count in a loop grows every tree again. |
| tsne-exact | info | unsafe | TSNE(method="exact") computes the force between every pair of rows, and the default method gave the same quality 4.5 to 16 times faster. |
Reproducible results
Section titled “Reproducible results”Pass random_state wherever scikit-learn shuffles or draws, so two runs give the same split and the same score.
| Rule | Severity | Fix | What it finds |
|---|---|---|---|
| random-state-without-shuffle | error | none | KFold uses random_state only when it shuffles, so without shuffle=True the call raises ValueError. |
| missing-random-state | info | none | Without random_state, a split or a model takes a new seed on every run, so the score changes from run to run. |