sklearn/tune-on-test
| Rule | tune-on-test |
|---|---|
| Pack | scikit-learn |
| Severity | error |
| Fix | none, edit by hand |
| Library versions | every version |
Picking a setting by its score on the test rows makes the best score too high. Pick it with cross-validation on the training rows, and score the test rows once.
for k in range(1, 30): model = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train) scores.append(model.score(X_test, y_test))best = max(scores)search = GridSearchCV(KNeighborsClassifier(), {"n_neighbors": range(1, 30)}, cv=5)search.fit(X_train, y_train)test_score = search.score(X_test, y_test)What the rule reports
Section titled “What the rule reports”The rule reports a loop that builds one model per setting, scores each model on the test rows and keeps the best score.
How to fix it
Section titled “How to fix it”xpd has no automatic fix, so make the change by hand.
The best of 30 test accuracies for n_neighbors from 1 to 30 was 0.597, where a GridSearchCV
on the training rows gave 0.507. The rule needs all of the following in the loop:
- The loop variable is a keyword of the call that built the scored model.
- A
score,predictorpredict_probaruns on a name withtestin it. - The loop picks a setting. A pick is a
>comparison that reads or sets a name withbestormaxin it, or whose branch reads the setting. A latermax,argmaxoridxmaxof the list that the loop appends to is a pick too.
A keyword named like random_state, seed, method or model is left alone, because such a
loop compares or averages.
The rule as the readers saw it gave wrong advice in 2 and 2 of 3 labelled reports, on loops that
printed whether the training score was far above the test score. The narrowed rule no longer
makes the reports. Recall was 15 of 50. The rule found none of 20 thresholds picked by the test
score. It also misses scores kept in a dict and picked with max(results, key=results.get), a
loop over ParameterGrid that builds the model with **params, and a list comprehension picked
with np.argmax.
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
This loop picks $V by the score on the test rows, so the best score is too high. The best of 30 such scores was 0.597 accuracy on data with no signal. Pick $V with GridSearchCV or cross_val_score on the training rows, and score the test rows once.Browse the other rules of the scikit-learn pack.