Skip to content

sklego/scorer-called-as-metric

Rule scorer-called-as-metric
Pack scikit-lego
Severity error
Fix unsafe, with xpd check --fix --unsafe-fixes
Library versions every version

These fairness functions return a scorer that takes an estimator, so wrapping them in make_scorer or calling them on labels raises and makes every score NaN. Pass the scorer itself to scoring=.

search = GridSearchCV(model, grid, scoring=make_scorer(p_percent_score("sex")))
fairness = p_percent_score(y_test, y_pred)
search = GridSearchCV(model, grid, scoring=p_percent_score("sex"))
fairness = p_percent_score("sex")(model, X_test, y_test)

The rule reports make_scorer around p_percent_score, equal_opportunity_score, correlation_score or subset_score. It also reports p_percent_score(y_true, y_pred) and equal_opportunity_score(y_true, y_pred). Each function takes the column to look at and returns a scorer with the signature (estimator, X, y).

xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.

The unsafe fix writes p_percent_score("sex") for make_scorer(p_percent_score("sex")). The other forms have no fix.

make_scorer calls the scorer with (y_true, y_pred), which the scorer can't take:

Form Result
make_scorer around each of the four functions, called IndexError for three of them, and TypeError for equal_opportunity_score
the same scorer inside GridSearchCV a best score of NaN, because scikit-learn caught the error
p_percent_score(y, y_pred) a function, with no error
p_percent_score("sex") given to scoring= a score

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

$S returns a scorer that takes (estimator, X, y), and make_scorer calls it with (y_true, y_pred), which raises IndexError or TypeError and makes every score of a search NaN. Pass the scorer itself to scoring=.

Browse the other rules of the scikit-lego pack.