mapie/alpha-as-confidence-level
| Rule | alpha-as-confidence-level |
|---|---|
| Pack | MAPIE |
| Severity | error |
| Fix | unsafe, with xpd check --fix --unsafe-fixes |
| Library versions | every version |
confidence_level is the share of rows to cover, which is 1 minus alpha, so confidence_level=0.1 makes intervals that cover about a tenth of the rows. Write confidence_level=0.9 for alpha=0.1.
alpha = 0.1mapie = SplitConformalRegressor(model, confidence_level=alpha)score = regression_mwi_score(y_test, y_pis, alpha)mapie = SplitConformalRegressor(model, confidence_level=1 - alpha)score = regression_mwi_score(y_test, y_pis, 1 - alpha)What the rule reports
Section titled “What the rule reports”The rule reports a confidence_level that holds the share to miss, such as
confidence_level=0.1, confidence_level=[0.05, 0.1] and confidence_level=alpha. It also
reports the same values given to regression_mwi_score and coverage_width_based in the place of
the level, and a percent such as confidence_level=90.
How to fix it
Section titled “How to fix it”xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.
The severity is warning for a level below 0.5 and for a variable named alpha or delta, and error for a percent. unsafe, because the level changes to the one the code meant: confidence_level=1 - 0.1 for confidence_level=0.1, 1 minus each number of a list, 1 - alpha for the level of regression_mwi_score and coverage_width_based, and confidence_level=0.90 for confidence_level=90. A variable named alpha or delta and any other percent have no fix, because the file does not show whether the variable holds a list or which share the number means.
confidence_level |
Share of test rows covered | Width of the intervals |
|---|---|---|
| 0.1 | 0.108 | 0.292 |
| 0.9 | 0.907 | 3.609 |
MAPIE runs both levels without a warning. The rule reads a literal below 0.5 on the constructors
of the conformal classes and on TimeSeriesRegressor.predict, and it leaves alone 0.5 itself and a
list with a level of 0.5 or more. A risk controller is left alone, because its level is
1 - delta and a low one is a choice.
A percent raises "Invalid confidence_level or alpha. Allowed values are between 0 and 1", but only
from predict_interval or predict_set, after the model is fitted. A risk controller raises
"delta must be in (0, 1]".
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
confidence_level is the share of rows to cover, which is 1 - alpha, and $V looks like an alpha. With confidence_level=0.1 the intervals covered 0.108 of the test rows, against 0.907 with 0.9. Write confidence_level=1 - $V, such as 0.9 for alpha=0.1.Browse the other rules of the MAPIE pack.