Skip to content

refit tool

Use refit to check that a saved scikit-learn Pipeline can be trained again from its raw data, as xpd refit does. It fits a clone on the training rows that the contract records and compares outputs and scores. joblib runs code when it loads a model, so load only a model you trust.

refit(file)
  • May change files
  • May overwrite what is there
  • A second call may do more
  • Runs code that may reach outside the project

Tags: pipeline.

The server gives the agent this description of the tool.

Fit a clone of the saved Pipeline on raw training rows and compare held out outputs and validation scores. Load only a model you trust because joblib runs code. Needs pandas and scikit-learn in the project's Python. Changes no saved file.

Type: string. Required.

The contract, a JSON file inside the project root.

The 14 fields of the result
Field Type Required Default Description
rows integer yes
train_rows integer yes
test_rows integer yes
output "predict_proba", "decision_function", "predict" yes
largest_gap number or null yes
model_matches boolean yes
contract_test_score number yes
refit_test_score number yes
contract_cv_scores list of number yes
refit_cv_scores list of number yes
test_score_matches boolean yes
cv_scores_match boolean yes
counts_match boolean yes
passed boolean yes

With the server's root at samples/pipeline, after train.py wrote its model:

{"file":"model/contract.json"}
{"rows":344,"train_rows":258,"test_rows":86,"output":"predict_proba","largest_gap":0.0,"model_matches":true,"contract_test_score":0.9883720930232558,"refit_test_score":0.9883720930232558,"contract_cv_scores":[0.9807692307692307,0.9807692307692307,1.0,1.0,0.9803921568627451],"refit_cv_scores":[0.9807692307692307,0.9807692307692307,1.0,1.0,0.9803921568627451],"test_score_matches":true,"cv_scores_match":true,"counts_match":true,"passed":true}

A contract that cannot be read, a missing pandas or scikit-learn, or a model that fails to load makes the call fail with the error that refit reports. A comparison that differs is a result, with passed: false.