Skip to content

gates tool

Use gates to judge a run from a data file to a saved model. It takes the results that profile, contract, check, graph, equiv, refit and discovery_gate returned, and checks each gate against them. It reads no file and runs nothing, so it does not prove that the files are unchanged since those calls. The run is done when it returns passed: true.

Field of proof The result of
discovery discovery_gate on the training, prediction and step files. Optional
lint check on the same files
dataflow graph on the file with the feature steps
reproducible equiv of the training script with itself
refit refit on the saved contract
served equiv of the training script with the script that loads the model
gates(profile, contract, proof)
  • Changes no file
  • A second call with the same arguments changes nothing more
  • Stays inside the project

Tags: pipeline.

The server gives the agent this description of the tool.

Judge a pipeline run from raw rows to a saved model. Each gate is one requirement, with the tools whose results decide it. The run is done when passed is true. Changes no file and runs nothing, so pass results of this run.

The gates: the data is the profiled file and its target has no nulls, the split covers every row, check reports no draw without a seed in scikit-learn, scikit-lego, MAPIE, Category Encoders, arch or pymoo code, the inputs have the names and dtypes of columns of the file, one Pipeline holds every step, the feature steps connect, check reports no error, the contract has a score for each fold, a raw training refit matches the saved model and the recorded scores, two training runs leave an equal result, the prediction script leaves that result too, and the versions are recorded. When discovery is supplied, include its indexed-api-calls result and require the contract's discovery record to match proof.discovery. Without discovery in either, thirteen gates apply.

A passed report does not judge choices made across separate runs or raw columns that stand in for the target. Pass fresh tool results from this project.

Type: object. Required.

The result of profile on the data file.

The 14 fields of `profile`
Field Type Required Default Description
profile.file string yes
profile.pandas string yes The pandas version that read the file.
profile.rows integer yes
profile.memory_bytes integer yes
profile.columns list of object yes
profile.columns[].column string yes
profile.columns[].dtype string yes
profile.columns[].nulls integer yes
profile.columns[].unique integer or null yes Distinct values, or null for unhashable cells.
profile.columns[].memory_bytes integer yes
profile.findings list of object no Column names that hold values, and text columns that suit a category.
profile.findings[].finding string yes values-as-column-names or category-candidate.
profile.findings[].columns list of string yes The columns the finding is about.
profile.findings[].message string yes What was measured on the file, and what to write.

Type: object. Required.

The result of contract on the file the training script saved.

The 82 fields of `contract`
Field Type Required Default Description
contract.discovery object or null no null
contract.discovery.question object yes The question that the agent asked find.
contract.discovery.question.task string yes The task, as it was given to find.
contract.discovery.candidates object yes Candidates ranked separately within each kind, including entries read through api.
contract.discovery.candidates.symbols list of object no
contract.discovery.candidates.symbols[].kind string yes
contract.discovery.candidates.symbols[].package string yes
contract.discovery.candidates.symbols[].name string yes
contract.discovery.candidates.symbols[].callable "class", "function", "method", "property" or null yes
contract.discovery.candidates.symbols[].parameters list of string yes
contract.discovery.candidates.symbols[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
contract.discovery.candidates.symbols[].summary string yes
contract.discovery.candidates.symbols[].members list of object yes
contract.discovery.candidates.symbols[].members[].name string yes
contract.discovery.candidates.symbols[].members[].parameters list of string yes
contract.discovery.candidates.symbols[].members[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
contract.discovery.candidates.symbols[].members[].summary string yes
contract.discovery.candidates.symbols[].members[].property boolean no false The member is read and not called.
contract.discovery.candidates.symbols[].rules list of string yes
contract.discovery.candidates.rules list of object no
contract.discovery.candidates.rules[].kind string yes
contract.discovery.candidates.rules[].package string yes
contract.discovery.candidates.rules[].name string yes
contract.discovery.candidates.rules[].callable "class", "function", "method", "property" or null yes
contract.discovery.candidates.rules[].parameters list of string yes
contract.discovery.candidates.rules[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
contract.discovery.candidates.rules[].summary string yes
contract.discovery.candidates.rules[].members list of object yes
contract.discovery.candidates.rules[].members[].name string yes
contract.discovery.candidates.rules[].members[].parameters list of string yes
contract.discovery.candidates.rules[].members[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
contract.discovery.candidates.rules[].members[].summary string yes
contract.discovery.candidates.rules[].members[].property boolean no false The member is read and not called.
contract.discovery.candidates.rules[].rules list of string yes
contract.discovery.candidates.recipes list of object no
contract.discovery.candidates.recipes[].kind string yes
contract.discovery.candidates.recipes[].package string yes
contract.discovery.candidates.recipes[].name string yes
contract.discovery.candidates.recipes[].callable "class", "function", "method", "property" or null yes
contract.discovery.candidates.recipes[].parameters list of string yes
contract.discovery.candidates.recipes[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
contract.discovery.candidates.recipes[].summary string yes
contract.discovery.candidates.recipes[].members list of object yes
contract.discovery.candidates.recipes[].members[].name string yes
contract.discovery.candidates.recipes[].members[].parameters list of string yes
contract.discovery.candidates.recipes[].members[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
contract.discovery.candidates.recipes[].members[].summary string yes
contract.discovery.candidates.recipes[].members[].property boolean no false The member is read and not called.
contract.discovery.candidates.recipes[].rules list of string yes
contract.discovery.choice object yes The candidate that the agent chose, and why.
contract.discovery.choice.name string yes The selected candidate's exact name.
contract.discovery.choice.reason string yes Why this candidate fits the task.
contract.discovery.choice.symbols list of string yes Indexed callable symbols the code must call, including a recipe's symbols.
contract.data object yes
contract.data.file string yes The data file, relative to the project root.
contract.data.target string yes The column the model predicts.
contract.data.rows integer yes Rows in the file, before the split.
contract.split object yes How the rows were divided once, before any step was fitted.
contract.split.seed integer yes The random_state of the split.
contract.split.test_size number yes The share of the rows held out.
contract.split.stratified boolean yes True when the split keeps the target's proportions.
contract.split.train_rows integer yes
contract.split.test_rows integer yes
contract.inputs list of object yes
contract.inputs[].column string yes
contract.inputs[].dtype string yes
contract.steps list of object yes
contract.steps[].name string yes
contract.steps[].estimator string yes The class name, such as ColumnTransformer.
contract.validation object yes
contract.validation.splitter object yes
contract.validation.splitter.class string yes
contract.validation.splitter.args object yes
contract.validation.scoring string yes The scorer name, such as accuracy.
contract.validation.folds integer yes
contract.validation.cv_scores list of number yes One score per fold.
contract.validation.test_score number yes The score on the held out rows.
contract.model object yes
contract.model.file string yes The saved model, relative to the project root.
contract.model.type string yes The class of the saved object, with its module.
contract.model.versions object yes Python and each library that wrote the file.
contract.model.code list of string no The project files the model imports when it loads, relative to the project root. A Pipeline saves a function such as the one in a FunctionTransformer by its module and name, so the model loads only where that module can be imported.

Type: object. Required.

lint is the result of check on the training, prediction and step files. dataflow is the result of graph on the file with the feature steps. reproducible compares two runs of the training script with equiv. refit is the result of refitting the saved contract. served is the result of equiv with the training script as old and the loading script as new.

The 133 fields of `proof`
Field Type Required Default Description
proof.discovery object or null no null
proof.discovery.discovery object yes
proof.discovery.discovery.question object yes The question that the agent asked find.
proof.discovery.discovery.question.task string yes The task, as it was given to find.
proof.discovery.discovery.candidates object yes Candidates ranked separately within each kind, including entries read through api.
proof.discovery.discovery.candidates.symbols list of object no
proof.discovery.discovery.candidates.symbols[].kind string yes
proof.discovery.discovery.candidates.symbols[].package string yes
proof.discovery.discovery.candidates.symbols[].name string yes
proof.discovery.discovery.candidates.symbols[].callable "class", "function", "method", "property" or null yes
proof.discovery.discovery.candidates.symbols[].parameters list of string yes
proof.discovery.discovery.candidates.symbols[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
proof.discovery.discovery.candidates.symbols[].summary string yes
proof.discovery.discovery.candidates.symbols[].members list of object yes
proof.discovery.discovery.candidates.symbols[].members[].name string yes
proof.discovery.discovery.candidates.symbols[].members[].parameters list of string yes
proof.discovery.discovery.candidates.symbols[].members[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
proof.discovery.discovery.candidates.symbols[].members[].summary string yes
proof.discovery.discovery.candidates.symbols[].members[].property boolean no false The member is read and not called.
proof.discovery.discovery.candidates.symbols[].rules list of string yes
proof.discovery.discovery.candidates.rules list of object no
proof.discovery.discovery.candidates.rules[].kind string yes
proof.discovery.discovery.candidates.rules[].package string yes
proof.discovery.discovery.candidates.rules[].name string yes
proof.discovery.discovery.candidates.rules[].callable "class", "function", "method", "property" or null yes
proof.discovery.discovery.candidates.rules[].parameters list of string yes
proof.discovery.discovery.candidates.rules[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
proof.discovery.discovery.candidates.rules[].summary string yes
proof.discovery.discovery.candidates.rules[].members list of object yes
proof.discovery.discovery.candidates.rules[].members[].name string yes
proof.discovery.discovery.candidates.rules[].members[].parameters list of string yes
proof.discovery.discovery.candidates.rules[].members[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
proof.discovery.discovery.candidates.rules[].members[].summary string yes
proof.discovery.discovery.candidates.rules[].members[].property boolean no false The member is read and not called.
proof.discovery.discovery.candidates.rules[].rules list of string yes
proof.discovery.discovery.candidates.recipes list of object no
proof.discovery.discovery.candidates.recipes[].kind string yes
proof.discovery.discovery.candidates.recipes[].package string yes
proof.discovery.discovery.candidates.recipes[].name string yes
proof.discovery.discovery.candidates.recipes[].callable "class", "function", "method", "property" or null yes
proof.discovery.discovery.candidates.recipes[].parameters list of string yes
proof.discovery.discovery.candidates.recipes[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
proof.discovery.discovery.candidates.recipes[].summary string yes
proof.discovery.discovery.candidates.recipes[].members list of object yes
proof.discovery.discovery.candidates.recipes[].members[].name string yes
proof.discovery.discovery.candidates.recipes[].members[].parameters list of string yes
proof.discovery.discovery.candidates.recipes[].members[].parameters_written list of string no The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional.
proof.discovery.discovery.candidates.recipes[].members[].summary string yes
proof.discovery.discovery.candidates.recipes[].members[].property boolean no false The member is read and not called.
proof.discovery.discovery.candidates.recipes[].rules list of string yes
proof.discovery.discovery.choice object yes The candidate that the agent chose, and why.
proof.discovery.discovery.choice.name string yes The selected candidate's exact name.
proof.discovery.discovery.choice.reason string yes Why this candidate fits the task.
proof.discovery.discovery.choice.symbols list of string yes Indexed callable symbols the code must call, including a recipe's symbols.
proof.discovery.paths list of string yes
proof.discovery.gate object yes One requirement of a pipeline run, and whether the evidence meets it.
proof.discovery.gate.name string yes
proof.discovery.gate.stage "discovery", "raw data", "profile", "split", "preprocessing", "model", "validation", "proof", "persisted model" yes
proof.discovery.gate.evidence list of "profile", "check", "graph", "equiv", "contract", "api", "source", "refit" yes The tools whose results decide the gate.
proof.discovery.gate.passed boolean yes
proof.discovery.gate.detail string yes What was found, or each reason the gate failed.
proof.lint object yes check on the training, prediction and step files.
proof.lint.fixed integer yes Number of fixes written to disk.
proof.lint.fixable integer yes Issues left that the fix tool fixes at the level asked for.
proof.lint.issues list of object yes
proof.lint.issues[].package string yes Module name of the rule's pack, such as pandas.
proof.lint.issues[].rule string yes Rule id, such as chained-assignment.
proof.lint.issues[].file string yes
proof.lint.issues[].line integer or null yes
proof.lint.issues[].column integer or null yes
proof.lint.issues[].severity "error", "warning", "info", "hint" yes
proof.lint.issues[].message string yes
proof.lint.issues[].fix "safe", "unsafe" or null yes How safe the rule's fix is, or null for no fix.
proof.dataflow object yes graph on the file with the feature steps.
proof.dataflow.pipelines list of object yes
proof.dataflow.pipelines[].name string yes
proof.dataflow.pipelines[].file string yes
proof.dataflow.pipelines[].line integer yes
proof.dataflow.pipelines[].steps list of object yes
proof.dataflow.pipelines[].steps[].name string yes The function, or the call of a chain link: .assign(gross).
proof.dataflow.pipelines[].steps[].kind string no "function" function, or link for a chain link.
proof.dataflow.pipelines[].steps[].line integer yes
proof.dataflow.pipelines[].steps[].reads list of string yes Columns the step reads.
proof.dataflow.pipelines[].steps[].columns list of string yes Columns the step adds.
proof.dataflow.pipelines[].steps[].computed boolean yes True when the step also adds columns named at run time.
proof.dataflow.pipelines[].links list of object yes
proof.dataflow.pipelines[].links[].from string yes
proof.dataflow.pipelines[].links[].to string yes
proof.dataflow.pipelines[].links[].columns list of string yes
proof.dataflow.pipelines[].source list of string no Columns the steps read that no earlier step adds: columns= of the read.
proof.dataflow.pipelines[].source_complete boolean no true False when a step also reads columns named at run time.
proof.dataflow.pipelines[].free list of string yes Steps that share no column with another step.
proof.dataflow.nodes list of object yes
proof.dataflow.nodes[].name string yes
proof.dataflow.nodes[].file string yes
proof.dataflow.nodes[].line integer yes
proof.dataflow.nodes[].function string yes
proof.dataflow.nodes[].inputs list of object yes
proof.dataflow.nodes[].inputs[].name string yes
proof.dataflow.nodes[].inputs[].optional boolean yes
proof.dataflow.nodes[].kind string yes function, parameterized, extracted or saved.
proof.dataflow.nodes[].config string or null yes The @config.when decorator, when there is one.
proof.dataflow.inputs list of object yes
proof.dataflow.inputs[].name string yes
proof.dataflow.inputs[].optional boolean yes
proof.dataflow.inputs[].readers list of string yes
proof.dataflow.unread list of string yes Nodes that no other node reads.
proof.dataflow.problems list of object yes
proof.dataflow.problems[].file string yes
proof.dataflow.problems[].line integer yes
proof.dataflow.problems[].severity "error", "warning", "note" yes
proof.dataflow.problems[].message string yes
proof.reproducible object yes equiv of the training script with itself.
proof.reproducible.equivalent boolean yes
proof.reproducible.detail string or null no null How the results differ.
proof.refit object yes refit on the saved contract.
proof.refit.rows integer yes
proof.refit.train_rows integer yes
proof.refit.test_rows integer yes
proof.refit.output "predict_proba", "decision_function", "predict" yes
proof.refit.largest_gap number or null yes
proof.refit.model_matches boolean yes
proof.refit.contract_test_score number yes
proof.refit.refit_test_score number yes
proof.refit.contract_cv_scores list of number yes
proof.refit.refit_cv_scores list of number yes
proof.refit.test_score_matches boolean yes
proof.refit.cv_scores_match boolean yes
proof.refit.counts_match boolean yes
proof.refit.passed boolean yes
proof.served object yes equiv of the training script with the script that loads the saved model.
proof.served.equivalent boolean yes
proof.served.detail string or null no null How the results differ.
Field Type Required Default Description
passed boolean yes True when every gate passed.
failed list of string yes The names of the gates that failed.
gates list of object yes
gates[].name string yes
gates[].stage "discovery", "raw data", "profile", "split", "preprocessing", "model", "validation", "proof", "persisted model" yes
gates[].evidence list of "profile", "check", "graph", "equiv", "contract", "api", "source", "refit" yes The tools whose results decide the gate.
gates[].passed boolean yes
gates[].detail string yes What was found, or each reason the gate failed.

In samples/pipeline, after train.py wrote its model, the results of the other tools passed every gate.

{"profile":"(profile of penguins.csv)","contract":"(contract of model/contract.json)","proof":{"discovery":"...","lint":"...","dataflow":"...","reproducible":"...","refit":"...","served":"..."}}
{"passed":true,"failed":[],"gates":[{"name":"data-is-profiled","stage":"raw data","evidence":["profile","contract"],"passed":true,"detail":"penguins.csv has 344 rows, and the target species has no nulls."},"..."]}

Each entry in gates corresponds to one requirement in the description above; the discovery gates are judged only when the contract or the proof holds a discovery record.

A gate that fails is a result, with its name in failed and a detail that says why. Arguments that do not match the schemas fail before any gate is judged.