Skip to content

sklego/pca-outlier-after-scaler

Rule pca-outlier-after-scaler
Pack scikit-lego
Severity error
Fix none, edit by hand
Library versions every version

After a centring step such as StandardScaler, the default relative error of PCAOutlierDetection divides by numbers near zero and flags many normal rows. Pass variant="absolute".

detector = make_pipeline(StandardScaler(), PCAOutlierDetection(n_components=1, threshold=0.05))
detector = make_pipeline(StandardScaler(), PCAOutlierDetection(n_components=1, threshold=2.0, variant="absolute"))

The rule reports PCAOutlierDetection and UMAPOutlierDetection without variant="absolute" in a pipeline that holds StandardScaler, RobustScaler or PowerTransformer. The default variant="relative" divides the reconstruction error of a row by the sum of the row's values, which a centring step puts near zero.

xpd has no automatic fix, so make the change by hand.

xpd gives the replacement as advice and doesn't apply it, because the threshold has other units.

Data, 1,000 rows near a line with 20 planted outliers Rows flagged by the relative variant, with threshold=0.1 By variant="absolute", with threshold=2.0
raw the 20 planted rows the 20 planted rows
after StandardScaler 320, of which 19 were planted the 20 planted rows

After the scaler, 510 rows had a negative ratio. A scaler built with with_mean=False, with_centering=False or standardize=False doesn't centre, and the rule leaves it alone.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

$C divides the reconstruction error by the sum of each row, which a centring step puts near zero. With 20 planted outliers in 1,000 rows it flagged 320 rows after StandardScaler, 19 of them planted, where variant="absolute" flagged exactly the 20. Pass variant="absolute" and a threshold in the units of the scaled data.

Browse the other rules of the scikit-lego pack.