refit tool
Use refit to check that a saved scikit-learn Pipeline can be trained again from
its raw data, as xpd refit does. It fits a clone on the training rows that the
contract records and compares outputs and scores. joblib runs code when it loads a model, so
load only a model you trust.
refit(file)- May change files
- May overwrite what is there
- A second call may do more
- Runs code that may reach outside the project
Tags: pipeline.
Description
Section titled “Description”The server gives the agent this description of the tool.
Fit a clone of the saved Pipeline on raw training rows and compare held out outputs and validation scores. Load only a model you trust because joblib runs code. Needs pandas and scikit-learn in the project's Python. Changes no saved file.
Arguments
Section titled “Arguments”Type: string. Required.
The contract, a JSON file inside the project root.
Result
Section titled “Result”The 14 fields of the result
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
rows |
integer | yes | ||
train_rows |
integer | yes | ||
test_rows |
integer | yes | ||
output |
"predict_proba", "decision_function", "predict" |
yes | ||
largest_gap |
number or null | yes | ||
model_matches |
boolean | yes | ||
contract_test_score |
number | yes | ||
refit_test_score |
number | yes | ||
contract_cv_scores |
list of number | yes | ||
refit_cv_scores |
list of number | yes | ||
test_score_matches |
boolean | yes | ||
cv_scores_match |
boolean | yes | ||
counts_match |
boolean | yes | ||
passed |
boolean | yes |
Example
Section titled “Example”With the server's root at samples/pipeline, after train.py wrote its model:
{"file":"model/contract.json"}{"rows":344,"train_rows":258,"test_rows":86,"output":"predict_proba","largest_gap":0.0,"model_matches":true,"contract_test_score":0.9883720930232558,"refit_test_score":0.9883720930232558,"contract_cv_scores":[0.9807692307692307,0.9807692307692307,1.0,1.0,0.9803921568627451],"refit_cv_scores":[0.9807692307692307,0.9807692307692307,1.0,1.0,0.9803921568627451],"test_score_matches":true,"cv_scores_match":true,"counts_match":true,"passed":true}Errors
Section titled “Errors”A contract that cannot be read, a missing pandas or scikit-learn, or a model that fails to load
makes the call fail with the error that refit reports. A comparison that differs is a result,
with passed: false.