mapie/shuffled-time-series-split
| Rule | shuffled-time-series-split |
|---|---|
| Pack | MAPIE |
| Severity | warning |
| Fix | unsafe, with xpd check --fix --unsafe-fixes |
| Library versions | every version |
A shuffled split puts test rows between training rows in time, so a TimeSeriesRegressor looks better on them than on later rows. Pass shuffle=False.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, shuffle=False)series = TimeSeriesRegressor(model, cv=blocks)series.fit(X_train, y_train)What the rule reports
Section titled “What the rule reports”The rule reports train_test_split and train_conformalize_test_split without shuffle=False,
where the first result goes into fit of a TimeSeriesRegressor.
How to fix it
Section titled “How to fix it”xpd check --fix --unsafe-fixes applies the fixes of this rule. Check the result, because an unsafe fix can change what the code does.
The unsafe fix adds shuffle=False at the end of the arguments and drops the comments between them. The parts then hold other rows. A split with stratify has no fix, because scikit-learn refuses stratify with shuffle=False.
On a random walk of 900 rows, the intervals covered the following shares of the test rows:
| Model | Test rows | Coverage |
|---|---|---|
TimeSeriesRegressor |
a shuffled test part | 0.921 |
TimeSeriesRegressor |
a test part after the training rows | 0.536 |
SplitConformalRegressor on a shuffled split |
its own test rows | 0.891 |
SplitConformalRegressor on a shuffled split |
the 300 rows that came after them | 0.144 |
The rule can't see that the rows are in time order, so it is a warning.
Message
Section titled “Message”xpd prints this message with the issue. A name that starts with $ stands for code from your file.
This split shuffles the rows, and $X goes into a TimeSeriesRegressor, so test rows sit between training rows in time. On a random walk the intervals covered 0.921 of a shuffled test part and 0.536 of the rows that came later, at confidence_level=0.9. Pass shuffle=False so that the test rows come after the training rows.Browse the other rules of the MAPIE pack.