Skip to content

Cross Validation and TrainTest #212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
May 28, 2018
Merged

Cross Validation and TrainTest #212

Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
3145971
initial commit.
codemzs May 18, 2018
f3e42ef
changes.
codemzs May 22, 2018
4ce6605
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
codemzs May 22, 2018
c39dfc7
resolve merge conflicts.
codemzs May 22, 2018
6f273a3
Add entry point to combine arrays of metrics data views into one data…
yaeldMS May 22, 2018
3543ddb
Add unit test for the multi-class case
yaeldMS May 22, 2018
d9aa28c
Merge
yaeldMS May 22, 2018
5676a27
Update CSharpApi.cs after merge
yaeldMS May 22, 2018
e930904
train-test big fix.
codemzs May 22, 2018
857d5ab
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
codemzs May 22, 2018
ae376a0
Fix unit test
yaeldMS May 22, 2018
363cad8
Merge branch 'cvmacro' of https://github.com/yaeldekel/machinelearnin…
codemzs May 23, 2018
0ef9ce1
Merge PR# 207.
codemzs May 23, 2018
001fa53
Unit-test.
codemzs May 24, 2018
3f53309
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
codemzs May 24, 2018
372e764
resolve merge conflicts.
codemzs May 24, 2018
95f4e83
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
codemzs May 24, 2018
eaa5545
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
codemzs May 25, 2018
2769d9b
Add Train-Test and address PR feedback.
codemzs May 25, 2018
6371e06
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
codemzs May 25, 2018
1f520a2
clean up.
codemzs May 25, 2018
3991971
more cleanup.
codemzs May 25, 2018
3424775
PR feedback.
codemzs May 25, 2018
542b845
PR feedback.
codemzs May 25, 2018
072e61d
PR feedback.
codemzs May 25, 2018
9d6fb01
PR feedback.
codemzs May 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ZBaselines/Common/EntryPoints/core_ep-list.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
Data.CustomTextLoader Import a dataset from a text file Microsoft.ML.Runtime.EntryPoints.ImportTextData ImportText Microsoft.ML.Runtime.EntryPoints.ImportTextData+Input Microsoft.ML.Runtime.EntryPoints.ImportTextData+Output
Data.DataViewReference Pass dataview from memory to experiment Microsoft.ML.Runtime.EntryPoints.DataViewReference ImportData Microsoft.ML.Runtime.EntryPoints.DataViewReference+Input Microsoft.ML.Runtime.EntryPoints.DataViewReference+Output
Data.IDataViewArrayConverter Create and array variable Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro MakeArray Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIDataViewInput Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIDataViewOutput
Data.PredictorModelArrayConverter Create and array variable Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro MakeArray Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIPredictorModelInput Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIPredictorModelOutput
Data.IDataViewArrayConverter Create an array variable of IDataView Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro MakeArray Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIDataViewInput Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIDataViewOutput
Data.PredictorModelArrayConverter Create an array variable of IPredictorModel Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro MakeArray Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIPredictorModelInput Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayIPredictorModelOutput
Data.TextLoader Import a dataset from a text file Microsoft.ML.Runtime.EntryPoints.ImportTextData TextLoader Microsoft.ML.Runtime.EntryPoints.ImportTextData+LoaderInput Microsoft.ML.Runtime.EntryPoints.ImportTextData+Output
Data.TransformModelArrayConverter Create an array variable of ITransformModel Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro MakeArray Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayITransformModelInput Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+ArrayITransformModelOutput
Models.AnomalyDetectionEvaluator Evaluates an anomaly detection scored dataset. Microsoft.ML.Runtime.Data.Evaluate AnomalyDetection Microsoft.ML.Runtime.Data.AnomalyDetectionMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.BinaryClassificationEvaluator Evaluates a binary classification scored dataset. Microsoft.ML.Runtime.Data.Evaluate Binary Microsoft.ML.Runtime.Data.BinaryClassifierMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.BinaryCrossValidator Cross validation for binary classification Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro CrossValidateBinary Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Expand Down
110 changes: 95 additions & 15 deletions ZBaselines/Common/EntryPoints/core_manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@
},
{
"Name": "Data.IDataViewArrayConverter",
"Desc": "Create and array variable",
"Desc": "Create an array variable of IDataView",
"FriendlyName": null,
"ShortName": null,
"Inputs": [
Expand Down Expand Up @@ -92,7 +92,7 @@
},
{
"Name": "Data.PredictorModelArrayConverter",
"Desc": "Create and array variable",
"Desc": "Create an array variable of IPredictorModel",
"FriendlyName": null,
"ShortName": null,
"Inputs": [
Expand Down Expand Up @@ -469,6 +469,35 @@
"ILearningPipelineLoader"
]
},
{
"Name": "Data.TransformModelArrayConverter",
"Desc": "Create an array variable of ITransformModel",
"FriendlyName": null,
"ShortName": null,
"Inputs": [
{
"Name": "TransformModel",
"Type": {
"Kind": "Array",
"ItemType": "TransformModel"
},
"Desc": "The models",
"Required": true,
"SortOrder": 1.0,
"IsNullable": false
}
],
"Outputs": [
{
"Name": "OutputModel",
"Type": {
"Kind": "Array",
"ItemType": "TransformModel"
},
"Desc": "The model array"
}
]
},
{
"Name": "Models.AnomalyDetectionEvaluator",
"Desc": "Evaluates an anomaly detection scored dataset.",
Expand Down Expand Up @@ -1300,7 +1329,7 @@
"Label"
],
"Required": false,
"SortOrder": 6.0,
"SortOrder": 5.0,
"IsNullable": false,
"Default": "Label"
},
Expand All @@ -1320,7 +1349,7 @@
},
"Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
"Required": true,
"SortOrder": 7.0,
"SortOrder": 6.0,
"IsNullable": false,
"Default": "SignatureBinaryClassifierTrainer"
}
Expand Down Expand Up @@ -1408,12 +1437,31 @@
"Kind": "Struct",
"Fields": [
{
"Name": "Model",
"Name": "PredictorModel",
"Type": "PredictorModel",
"Desc": "The model",
"Required": true,
"Desc": "The predictor model",
"Required": false,
"SortOrder": 1.0,
"IsNullable": false
"IsNullable": false,
"Default": null
},
{
"Name": "TransformModel",
"Type": "TransformModel",
"Desc": "The transform model",
"Required": false,
"SortOrder": 2.0,
"IsNullable": false,
"Default": null
},
{
"Name": "UseTransformModel",
"Type": "Bool",
"Desc": "Indicates to use transform model instead of predictor model.",
"Required": false,
"SortOrder": 3.0,
"IsNullable": false,
"Default": false
}
]
},
Expand All @@ -1430,7 +1478,7 @@
"strat"
],
"Required": false,
"SortOrder": 7.0,
"SortOrder": 6.0,
"IsNullable": false,
"Default": null
},
Expand All @@ -1442,7 +1490,7 @@
"k"
],
"Required": false,
"SortOrder": 8.0,
"SortOrder": 7.0,
"IsNullable": false,
"Default": 2
},
Expand All @@ -1462,7 +1510,7 @@
},
"Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
"Required": true,
"SortOrder": 9.0,
"SortOrder": 8.0,
"IsNullable": false,
"Default": "SignatureBinaryClassifierTrainer"
}
Expand All @@ -1476,6 +1524,14 @@
},
"Desc": "The final model including the trained predictor model and the model from the transforms, provided as the Input.TransformModel."
},
{
"Name": "TransformModel",
"Type": {
"Kind": "Array",
"ItemType": "TransformModel"
},
"Desc": "The final model including the trained predictor model and the model from the transforms, provided as the Input.TransformModel."
Copy link
Member

@sfilipi sfilipi May 26, 2018 •
edited by codemzs
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the description correct? #Resolved

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, so we introduced a second option in CV and TrainTest where the final model is of ITransformModel.

In reply to: 191033404 [](ancestors = 191033404)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the part of the comment that talks about bundling the predictor model in it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please clarify? The comment already talks about predictor being included in the final model.

In reply to: 191049219 [](ancestors = 191049219)

},
{
"Name": "Warnings",
"Type": "DataView",
Expand Down Expand Up @@ -2999,12 +3055,31 @@
"Kind": "Struct",
"Fields": [
{
"Name": "Model",
"Name": "PredictorModel",
"Type": "PredictorModel",
"Desc": "The model",
"Required": true,
"Desc": "The predictor model",
"Required": false,
"SortOrder": 1.0,
"IsNullable": false
"IsNullable": false,
"Default": null
},
{
"Name": "TransformModel",
"Type": "TransformModel",
"Desc": "Transform model",
"Required": false,
"SortOrder": 2.0,
"IsNullable": false,
"Default": null
},
{
"Name": "UseTransformModel",
"Type": "Bool",
"Desc": "Indicates to use transform model instead of predictor model.",
"Required": false,
"SortOrder": 3.0,
"IsNullable": false,
"Default": false
}
]
},
Expand Down Expand Up @@ -3058,6 +3133,11 @@
"Type": "PredictorModel",
"Desc": "The final model including the trained predictor model and the model from the transforms, provided as the Input.TransformModel."
},
{
"Name": "TransformModel",
"Type": "TransformModel",
"Desc": "The final model including the trained predictor model and the model from the transforms, provided as the Input.TransformModel."
},
{
"Name": "Warnings",
"Type": "DataView",
Expand Down
4 changes: 2 additions & 2 deletions src/Microsoft.ML.PipelineInference/PipelinePattern.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ public Experiment CreateTrainTestExperiment(IDataView trainData, IDataView testD
},
Outputs =
{
Model = finalOutput
PredictorModel = finalOutput
},
PipelineId = UniqueId.ToString("N"),
Kind = MacroUtils.TrainerKindApiValue<Models.MacroUtilsTrainerKinds>(trainerKind),
Expand Down Expand Up @@ -189,7 +189,7 @@ public Models.TrainTestEvaluator.Output AddAsTrainTest(Var<IDataView> trainData,
},
Outputs =
{
Model = finalOutput
PredictorModel = finalOutput
},
TrainingData = trainData,
TestingData = testData,
Expand Down
Loading