Skip to content

mapie/shuffled-time-series-split

Rule shuffled-time-series-split
Pack MAPIE
Severity warning
Fix unsafe, with xpd check --fix --unsafe-fixes
Library versions every version

A shuffled split puts test rows between training rows in time, so a TimeSeriesRegressor looks better on them than on later rows. Pass shuffle=False.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, shuffle=False)
series = TimeSeriesRegressor(model, cv=blocks)
series.fit(X_train, y_train)

The rule reports train_test_split and train_conformalize_test_split without shuffle=False, where the first result goes into fit of a TimeSeriesRegressor.

xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.

The unsafe fix adds shuffle=False at the end of the arguments and drops the comments between them. The parts then hold other rows. A split with stratify has no fix, because scikit-learn refuses stratify with shuffle=False.

On a random walk of 900 rows, the intervals covered the following shares of the test rows:

Model Test rows Coverage
TimeSeriesRegressor a shuffled test part 0.921
TimeSeriesRegressor a test part after the training rows 0.536
SplitConformalRegressor on a shuffled split its own test rows 0.891
SplitConformalRegressor on a shuffled split the 300 rows that came after them 0.144

The rule can't see that the rows are in time order, so it is a warning.

xpd prints this message with the issue. A name that starts with $ stands for code from your file.

This split shuffles the rows, and $X goes into a TimeSeriesRegressor, so test rows sit between training rows in time. On a random walk the intervals covered 0.921 of a shuffled test part and 0.536 of the rows that came later, at confidence_level=0.9. Pass shuffle=False so that the test rows come after the training rows.

Browse the other rules of the MAPIE pack.