gates tool
Use gates to judge a run from a data file to a saved model. It takes the results
that profile, contract, check,
graph, equiv, refit and
discovery_gate returned, and checks each gate against them. It reads no
file and runs nothing, so it does not prove that the files are unchanged since those calls. The
run is done when it returns passed: true.
Field of proof |
The result of |
|---|---|
discovery |
discovery_gate on the training, prediction and step files. Optional |
lint |
check on the same files |
dataflow |
graph on the file with the feature steps |
reproducible |
equiv of the training script with itself |
refit |
refit on the saved contract |
served |
equiv of the training script with the script that loads the model |
gates(profile, contract, proof)- Changes no file
- A second call with the same arguments changes nothing more
- Stays inside the project
Tags: pipeline.
Description
Section titled “Description”The server gives the agent this description of the tool.
Judge a pipeline run from raw rows to a saved model. Each gate is one requirement, with the tools whose results decide it. The run is done when passed is true. Changes no file and runs nothing, so pass results of this run.
The gates: the data is the profiled file and its target has no nulls, the split covers every row, check reports no draw without a seed in scikit-learn, scikit-lego, MAPIE, Category Encoders, arch or pymoo code, the inputs have the names and dtypes of columns of the file, one Pipeline holds every step, the feature steps connect, check reports no error, the contract has a score for each fold, a raw training refit matches the saved model and the recorded scores, two training runs leave an equal result, the prediction script leaves that result too, and the versions are recorded. When discovery is supplied, include its indexed-api-calls result and require the contract's discovery record to match proof.discovery. Without discovery in either, thirteen gates apply.
A passed report does not judge choices made across separate runs or raw columns that stand in for the target. Pass fresh tool results from this project.
Arguments
Section titled “Arguments”profile
Section titled “profile”Type: object. Required.
The result of profile on the data file.
The 14 fields of `profile`
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
profile.file |
string | yes | ||
profile.pandas |
string | yes | The pandas version that read the file. | |
profile.rows |
integer | yes | ||
profile.memory_bytes |
integer | yes | ||
profile.columns |
list of object | yes | ||
profile.columns[].column |
string | yes | ||
profile.columns[].dtype |
string | yes | ||
profile.columns[].nulls |
integer | yes | ||
profile.columns[].unique |
integer or null | yes | Distinct values, or null for unhashable cells. | |
profile.columns[].memory_bytes |
integer | yes | ||
profile.findings |
list of object | no | Column names that hold values, and text columns that suit a category. | |
profile.findings[].finding |
string | yes | values-as-column-names or category-candidate. | |
profile.findings[].columns |
list of string | yes | The columns the finding is about. | |
profile.findings[].message |
string | yes | What was measured on the file, and what to write. |
contract
Section titled “contract”Type: object. Required.
The result of contract on the file the training script saved.
The 82 fields of `contract`
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
contract.discovery |
object or null | no | null |
|
contract.discovery.question |
object | yes | The question that the agent asked find. | |
contract.discovery.question.task |
string | yes | The task, as it was given to find. | |
contract.discovery.candidates |
object | yes | Candidates ranked separately within each kind, including entries read through api. | |
contract.discovery.candidates.symbols |
list of object | no | ||
contract.discovery.candidates.symbols[].kind |
string | yes | ||
contract.discovery.candidates.symbols[].package |
string | yes | ||
contract.discovery.candidates.symbols[].name |
string | yes | ||
contract.discovery.candidates.symbols[].callable |
"class", "function", "method", "property" or null |
yes | ||
contract.discovery.candidates.symbols[].parameters |
list of string | yes | ||
contract.discovery.candidates.symbols[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
contract.discovery.candidates.symbols[].summary |
string | yes | ||
contract.discovery.candidates.symbols[].members |
list of object | yes | ||
contract.discovery.candidates.symbols[].members[].name |
string | yes | ||
contract.discovery.candidates.symbols[].members[].parameters |
list of string | yes | ||
contract.discovery.candidates.symbols[].members[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
contract.discovery.candidates.symbols[].members[].summary |
string | yes | ||
contract.discovery.candidates.symbols[].members[].property |
boolean | no | false |
The member is read and not called. |
contract.discovery.candidates.symbols[].rules |
list of string | yes | ||
contract.discovery.candidates.rules |
list of object | no | ||
contract.discovery.candidates.rules[].kind |
string | yes | ||
contract.discovery.candidates.rules[].package |
string | yes | ||
contract.discovery.candidates.rules[].name |
string | yes | ||
contract.discovery.candidates.rules[].callable |
"class", "function", "method", "property" or null |
yes | ||
contract.discovery.candidates.rules[].parameters |
list of string | yes | ||
contract.discovery.candidates.rules[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
contract.discovery.candidates.rules[].summary |
string | yes | ||
contract.discovery.candidates.rules[].members |
list of object | yes | ||
contract.discovery.candidates.rules[].members[].name |
string | yes | ||
contract.discovery.candidates.rules[].members[].parameters |
list of string | yes | ||
contract.discovery.candidates.rules[].members[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
contract.discovery.candidates.rules[].members[].summary |
string | yes | ||
contract.discovery.candidates.rules[].members[].property |
boolean | no | false |
The member is read and not called. |
contract.discovery.candidates.rules[].rules |
list of string | yes | ||
contract.discovery.candidates.recipes |
list of object | no | ||
contract.discovery.candidates.recipes[].kind |
string | yes | ||
contract.discovery.candidates.recipes[].package |
string | yes | ||
contract.discovery.candidates.recipes[].name |
string | yes | ||
contract.discovery.candidates.recipes[].callable |
"class", "function", "method", "property" or null |
yes | ||
contract.discovery.candidates.recipes[].parameters |
list of string | yes | ||
contract.discovery.candidates.recipes[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
contract.discovery.candidates.recipes[].summary |
string | yes | ||
contract.discovery.candidates.recipes[].members |
list of object | yes | ||
contract.discovery.candidates.recipes[].members[].name |
string | yes | ||
contract.discovery.candidates.recipes[].members[].parameters |
list of string | yes | ||
contract.discovery.candidates.recipes[].members[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
contract.discovery.candidates.recipes[].members[].summary |
string | yes | ||
contract.discovery.candidates.recipes[].members[].property |
boolean | no | false |
The member is read and not called. |
contract.discovery.candidates.recipes[].rules |
list of string | yes | ||
contract.discovery.choice |
object | yes | The candidate that the agent chose, and why. | |
contract.discovery.choice.name |
string | yes | The selected candidate's exact name. | |
contract.discovery.choice.reason |
string | yes | Why this candidate fits the task. | |
contract.discovery.choice.symbols |
list of string | yes | Indexed callable symbols the code must call, including a recipe's symbols. | |
contract.data |
object | yes | ||
contract.data.file |
string | yes | The data file, relative to the project root. | |
contract.data.target |
string | yes | The column the model predicts. | |
contract.data.rows |
integer | yes | Rows in the file, before the split. | |
contract.split |
object | yes | How the rows were divided once, before any step was fitted. | |
contract.split.seed |
integer | yes | The random_state of the split. | |
contract.split.test_size |
number | yes | The share of the rows held out. | |
contract.split.stratified |
boolean | yes | True when the split keeps the target's proportions. | |
contract.split.train_rows |
integer | yes | ||
contract.split.test_rows |
integer | yes | ||
contract.inputs |
list of object | yes | ||
contract.inputs[].column |
string | yes | ||
contract.inputs[].dtype |
string | yes | ||
contract.steps |
list of object | yes | ||
contract.steps[].name |
string | yes | ||
contract.steps[].estimator |
string | yes | The class name, such as ColumnTransformer. | |
contract.validation |
object | yes | ||
contract.validation.splitter |
object | yes | ||
contract.validation.splitter.class |
string | yes | ||
contract.validation.splitter.args |
object | yes | ||
contract.validation.scoring |
string | yes | The scorer name, such as accuracy. | |
contract.validation.folds |
integer | yes | ||
contract.validation.cv_scores |
list of number | yes | One score per fold. | |
contract.validation.test_score |
number | yes | The score on the held out rows. | |
contract.model |
object | yes | ||
contract.model.file |
string | yes | The saved model, relative to the project root. | |
contract.model.type |
string | yes | The class of the saved object, with its module. | |
contract.model.versions |
object | yes | Python and each library that wrote the file. | |
contract.model.code |
list of string | no | The project files the model imports when it loads, relative to the project root. A Pipeline saves a function such as the one in a FunctionTransformer by its module and name, so the model loads only where that module can be imported. |
Type: object. Required.
lint is the result of check on the training, prediction and step files. dataflow is the result of graph on the file with the feature steps. reproducible compares two runs of the training script with equiv. refit is the result of refitting the saved contract. served is the result of equiv with the training script as old and the loading script as new.
The 133 fields of `proof`
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
proof.discovery |
object or null | no | null |
|
proof.discovery.discovery |
object | yes | ||
proof.discovery.discovery.question |
object | yes | The question that the agent asked find. | |
proof.discovery.discovery.question.task |
string | yes | The task, as it was given to find. | |
proof.discovery.discovery.candidates |
object | yes | Candidates ranked separately within each kind, including entries read through api. | |
proof.discovery.discovery.candidates.symbols |
list of object | no | ||
proof.discovery.discovery.candidates.symbols[].kind |
string | yes | ||
proof.discovery.discovery.candidates.symbols[].package |
string | yes | ||
proof.discovery.discovery.candidates.symbols[].name |
string | yes | ||
proof.discovery.discovery.candidates.symbols[].callable |
"class", "function", "method", "property" or null |
yes | ||
proof.discovery.discovery.candidates.symbols[].parameters |
list of string | yes | ||
proof.discovery.discovery.candidates.symbols[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
proof.discovery.discovery.candidates.symbols[].summary |
string | yes | ||
proof.discovery.discovery.candidates.symbols[].members |
list of object | yes | ||
proof.discovery.discovery.candidates.symbols[].members[].name |
string | yes | ||
proof.discovery.discovery.candidates.symbols[].members[].parameters |
list of string | yes | ||
proof.discovery.discovery.candidates.symbols[].members[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
proof.discovery.discovery.candidates.symbols[].members[].summary |
string | yes | ||
proof.discovery.discovery.candidates.symbols[].members[].property |
boolean | no | false |
The member is read and not called. |
proof.discovery.discovery.candidates.symbols[].rules |
list of string | yes | ||
proof.discovery.discovery.candidates.rules |
list of object | no | ||
proof.discovery.discovery.candidates.rules[].kind |
string | yes | ||
proof.discovery.discovery.candidates.rules[].package |
string | yes | ||
proof.discovery.discovery.candidates.rules[].name |
string | yes | ||
proof.discovery.discovery.candidates.rules[].callable |
"class", "function", "method", "property" or null |
yes | ||
proof.discovery.discovery.candidates.rules[].parameters |
list of string | yes | ||
proof.discovery.discovery.candidates.rules[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
proof.discovery.discovery.candidates.rules[].summary |
string | yes | ||
proof.discovery.discovery.candidates.rules[].members |
list of object | yes | ||
proof.discovery.discovery.candidates.rules[].members[].name |
string | yes | ||
proof.discovery.discovery.candidates.rules[].members[].parameters |
list of string | yes | ||
proof.discovery.discovery.candidates.rules[].members[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
proof.discovery.discovery.candidates.rules[].members[].summary |
string | yes | ||
proof.discovery.discovery.candidates.rules[].members[].property |
boolean | no | false |
The member is read and not called. |
proof.discovery.discovery.candidates.rules[].rules |
list of string | yes | ||
proof.discovery.discovery.candidates.recipes |
list of object | no | ||
proof.discovery.discovery.candidates.recipes[].kind |
string | yes | ||
proof.discovery.discovery.candidates.recipes[].package |
string | yes | ||
proof.discovery.discovery.candidates.recipes[].name |
string | yes | ||
proof.discovery.discovery.candidates.recipes[].callable |
"class", "function", "method", "property" or null |
yes | ||
proof.discovery.discovery.candidates.recipes[].parameters |
list of string | yes | ||
proof.discovery.discovery.candidates.recipes[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
proof.discovery.discovery.candidates.recipes[].summary |
string | yes | ||
proof.discovery.discovery.candidates.recipes[].members |
list of object | yes | ||
proof.discovery.discovery.candidates.recipes[].members[].name |
string | yes | ||
proof.discovery.discovery.candidates.recipes[].members[].parameters |
list of string | yes | ||
proof.discovery.discovery.candidates.recipes[].members[].parameters_written |
list of string | no | The parameter list as the source writes it: each parameter with its type annotation and default, and a * or / that stands alone between parameters. A stub file writes ... for a default, which says only that the parameter is optional. | |
proof.discovery.discovery.candidates.recipes[].members[].summary |
string | yes | ||
proof.discovery.discovery.candidates.recipes[].members[].property |
boolean | no | false |
The member is read and not called. |
proof.discovery.discovery.candidates.recipes[].rules |
list of string | yes | ||
proof.discovery.discovery.choice |
object | yes | The candidate that the agent chose, and why. | |
proof.discovery.discovery.choice.name |
string | yes | The selected candidate's exact name. | |
proof.discovery.discovery.choice.reason |
string | yes | Why this candidate fits the task. | |
proof.discovery.discovery.choice.symbols |
list of string | yes | Indexed callable symbols the code must call, including a recipe's symbols. | |
proof.discovery.paths |
list of string | yes | ||
proof.discovery.gate |
object | yes | One requirement of a pipeline run, and whether the evidence meets it. | |
proof.discovery.gate.name |
string | yes | ||
proof.discovery.gate.stage |
"discovery", "raw data", "profile", "split", "preprocessing", "model", "validation", "proof", "persisted model" |
yes | ||
proof.discovery.gate.evidence |
list of "profile", "check", "graph", "equiv", "contract", "api", "source", "refit" |
yes | The tools whose results decide the gate. | |
proof.discovery.gate.passed |
boolean | yes | ||
proof.discovery.gate.detail |
string | yes | What was found, or each reason the gate failed. | |
proof.lint |
object | yes | check on the training, prediction and step files. |
|
proof.lint.fixed |
integer | yes | Number of fixes written to disk. | |
proof.lint.fixable |
integer | yes | Issues left that the fix tool fixes at the level asked for. | |
proof.lint.issues |
list of object | yes | ||
proof.lint.issues[].package |
string | yes | Module name of the rule's pack, such as pandas. | |
proof.lint.issues[].rule |
string | yes | Rule id, such as chained-assignment. | |
proof.lint.issues[].file |
string | yes | ||
proof.lint.issues[].line |
integer or null | yes | ||
proof.lint.issues[].column |
integer or null | yes | ||
proof.lint.issues[].severity |
"error", "warning", "info", "hint" |
yes | ||
proof.lint.issues[].message |
string | yes | ||
proof.lint.issues[].fix |
"safe", "unsafe" or null |
yes | How safe the rule's fix is, or null for no fix. | |
proof.dataflow |
object | yes | graph on the file with the feature steps. |
|
proof.dataflow.pipelines |
list of object | yes | ||
proof.dataflow.pipelines[].name |
string | yes | ||
proof.dataflow.pipelines[].file |
string | yes | ||
proof.dataflow.pipelines[].line |
integer | yes | ||
proof.dataflow.pipelines[].steps |
list of object | yes | ||
proof.dataflow.pipelines[].steps[].name |
string | yes | The function, or the call of a chain link: .assign(gross). | |
proof.dataflow.pipelines[].steps[].kind |
string | no | "function" |
function, or link for a chain link. |
proof.dataflow.pipelines[].steps[].line |
integer | yes | ||
proof.dataflow.pipelines[].steps[].reads |
list of string | yes | Columns the step reads. | |
proof.dataflow.pipelines[].steps[].columns |
list of string | yes | Columns the step adds. | |
proof.dataflow.pipelines[].steps[].computed |
boolean | yes | True when the step also adds columns named at run time. | |
proof.dataflow.pipelines[].links |
list of object | yes | ||
proof.dataflow.pipelines[].links[].from |
string | yes | ||
proof.dataflow.pipelines[].links[].to |
string | yes | ||
proof.dataflow.pipelines[].links[].columns |
list of string | yes | ||
proof.dataflow.pipelines[].source |
list of string | no | Columns the steps read that no earlier step adds: columns= of the read. | |
proof.dataflow.pipelines[].source_complete |
boolean | no | true |
False when a step also reads columns named at run time. |
proof.dataflow.pipelines[].free |
list of string | yes | Steps that share no column with another step. | |
proof.dataflow.nodes |
list of object | yes | ||
proof.dataflow.nodes[].name |
string | yes | ||
proof.dataflow.nodes[].file |
string | yes | ||
proof.dataflow.nodes[].line |
integer | yes | ||
proof.dataflow.nodes[].function |
string | yes | ||
proof.dataflow.nodes[].inputs |
list of object | yes | ||
proof.dataflow.nodes[].inputs[].name |
string | yes | ||
proof.dataflow.nodes[].inputs[].optional |
boolean | yes | ||
proof.dataflow.nodes[].kind |
string | yes | function, parameterized, extracted or saved. | |
proof.dataflow.nodes[].config |
string or null | yes | The @config.when decorator, when there is one. | |
proof.dataflow.inputs |
list of object | yes | ||
proof.dataflow.inputs[].name |
string | yes | ||
proof.dataflow.inputs[].optional |
boolean | yes | ||
proof.dataflow.inputs[].readers |
list of string | yes | ||
proof.dataflow.unread |
list of string | yes | Nodes that no other node reads. | |
proof.dataflow.problems |
list of object | yes | ||
proof.dataflow.problems[].file |
string | yes | ||
proof.dataflow.problems[].line |
integer | yes | ||
proof.dataflow.problems[].severity |
"error", "warning", "note" |
yes | ||
proof.dataflow.problems[].message |
string | yes | ||
proof.reproducible |
object | yes | equiv of the training script with itself. |
|
proof.reproducible.equivalent |
boolean | yes | ||
proof.reproducible.detail |
string or null | no | null |
How the results differ. |
proof.refit |
object | yes | refit on the saved contract. |
|
proof.refit.rows |
integer | yes | ||
proof.refit.train_rows |
integer | yes | ||
proof.refit.test_rows |
integer | yes | ||
proof.refit.output |
"predict_proba", "decision_function", "predict" |
yes | ||
proof.refit.largest_gap |
number or null | yes | ||
proof.refit.model_matches |
boolean | yes | ||
proof.refit.contract_test_score |
number | yes | ||
proof.refit.refit_test_score |
number | yes | ||
proof.refit.contract_cv_scores |
list of number | yes | ||
proof.refit.refit_cv_scores |
list of number | yes | ||
proof.refit.test_score_matches |
boolean | yes | ||
proof.refit.cv_scores_match |
boolean | yes | ||
proof.refit.counts_match |
boolean | yes | ||
proof.refit.passed |
boolean | yes | ||
proof.served |
object | yes | equiv of the training script with the script that loads the saved model. |
|
proof.served.equivalent |
boolean | yes | ||
proof.served.detail |
string or null | no | null |
How the results differ. |
Result
Section titled “Result”| Field | Type | Required | Default | Description |
|---|---|---|---|---|
passed |
boolean | yes | True when every gate passed. | |
failed |
list of string | yes | The names of the gates that failed. | |
gates |
list of object | yes | ||
gates[].name |
string | yes | ||
gates[].stage |
"discovery", "raw data", "profile", "split", "preprocessing", "model", "validation", "proof", "persisted model" |
yes | ||
gates[].evidence |
list of "profile", "check", "graph", "equiv", "contract", "api", "source", "refit" |
yes | The tools whose results decide the gate. | |
gates[].passed |
boolean | yes | ||
gates[].detail |
string | yes | What was found, or each reason the gate failed. |
Example
Section titled “Example”In samples/pipeline, after train.py wrote its model, the results of the other tools passed
every gate.
{"profile":"(profile of penguins.csv)","contract":"(contract of model/contract.json)","proof":{"discovery":"...","lint":"...","dataflow":"...","reproducible":"...","refit":"...","served":"..."}}{"passed":true,"failed":[],"gates":[{"name":"data-is-profiled","stage":"raw data","evidence":["profile","contract"],"passed":true,"detail":"penguins.csv has 344 rows, and the target species has no nulls."},"..."]}Each entry in gates corresponds to one requirement in the description above; the discovery gates
are judged only when the contract or the proof holds a discovery record.
Errors
Section titled “Errors”A gate that fails is a result, with its name in failed and a detail that says why. Arguments
that do not match the schemas fail before any gate is judged.