Skip to content

Commit a434bec

Browse files
yaeldMSeerhardt
yaeldMS
authored and
eerhardt
committed
Update MacroUtils to map trainer kinds to the correct suffix of trainer entry point names (dotnet#113)
* Update suffix of trainer entry point names by trainer kind. * Address PR comments. * Add unit test. * Update C# API * Move unit test to TestAutoInference and fix EntryPointCatalog test. * Trigger build. * Add reference to the test project to make the sweeper entry point visible to EntryPointCatalog test.
1 parent 39d3253 commit a434bec

File tree

9 files changed

+728
-76
lines changed

9 files changed

+728
-76
lines changed

‎ZBaselines/Common/EntryPoints/core_ep-list.tsv

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,13 @@ Models.NaiveCalibrator Apply a Naive calibrator to an input model Microsoft.ML.R
1515
Models.OneVersusAll One-vs-All macro (OVA) Microsoft.ML.Runtime.EntryPoints.OneVersusAllMacro OVA Microsoft.ML.Runtime.EntryPoints.OneVersusAllMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.OneVersusAllMacro+Output]
1616
Models.OvaModelCombiner Combines a sequence of PredictorModels into a single model Microsoft.ML.Runtime.Learners.OvaPredictor CombineOvaModels Microsoft.ML.Runtime.EntryPoints.ModelOperations+CombineOvaPredictorModelsInput Microsoft.ML.Runtime.EntryPoints.ModelOperations+PredictorModelOutput
1717
Models.PAVCalibrator Apply a PAV calibrator to an input model Microsoft.ML.Runtime.Internal.Calibration.Calibrate Pav Microsoft.ML.Runtime.Internal.Calibration.Calibrate+NoArgumentsInput Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CalibratorOutput
18+
Models.PipelineSweeper AutoML pipeline sweeping optimzation macro. Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro PipelineSweep Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+Output]
1819
Models.PlattCalibrator Apply a Platt calibrator to an input model Microsoft.ML.Runtime.Internal.Calibration.Calibrate Platt Microsoft.ML.Runtime.Internal.Calibration.Calibrate+NoArgumentsInput Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CalibratorOutput
1920
Models.QuantileRegressionEvaluator Evaluates a quantile regression scored dataset. Microsoft.ML.Runtime.Data.Evaluate QuantileRegression Microsoft.ML.Runtime.Data.QuantileRegressionMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
2021
Models.RankerEvaluator Evaluates a ranking scored dataset. Microsoft.ML.Runtime.Data.Evaluate Ranking Microsoft.ML.Runtime.Data.RankerMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
2122
Models.RegressionEvaluator Evaluates a regression scored dataset. Microsoft.ML.Runtime.Data.Evaluate Regression Microsoft.ML.Runtime.Data.RegressionMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
2223
Models.Summarizer Summarize a linear regression predictor. Microsoft.ML.Runtime.EntryPoints.SummarizePredictor Summarize Microsoft.ML.Runtime.EntryPoints.SummarizePredictor+Input Microsoft.ML.Runtime.EntryPoints.CommonOutputs+SummaryOutput
24+
Models.SweepResultExtractor Extracts the sweep result. Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro ExtractSweepResult Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+ResultInput Microsoft.ML.Runtime.EntryPoints.PipelineSweeperMacro+Output
2325
Models.TrainTestBinaryEvaluator Train test for binary classification Microsoft.ML.Runtime.EntryPoints.TrainTestBinaryMacro TrainTestBinary Microsoft.ML.Runtime.EntryPoints.TrainTestBinaryMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.TrainTestBinaryMacro+Output]
2426
Models.TrainTestEvaluator General train test for any supported evaluator Microsoft.ML.Runtime.EntryPoints.TrainTestMacro TrainTest Microsoft.ML.Runtime.EntryPoints.TrainTestMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.TrainTestMacro+Output]
2527
Trainers.AveragedPerceptronBinaryClassifier Train a Average perceptron. Microsoft.ML.Runtime.Learners.AveragedPerceptronTrainer TrainBinary Microsoft.ML.Runtime.Learners.AveragedPerceptronTrainer+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+BinaryClassificationOutput

‎ZBaselines/Common/EntryPoints/core_manifest.json

Lines changed: 358 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1723,6 +1723,105 @@
17231723
"ITrainerOutput"
17241724
]
17251725
},
1726+
{
1727+
"Name": "Models.PipelineSweeper",
1728+
"Desc": "AutoML pipeline sweeping optimzation macro.",
1729+
"FriendlyName": null,
1730+
"ShortName": null,
1731+
"Inputs": [
1732+
{
1733+
"Name": "TrainingData",
1734+
"Type": "DataView",
1735+
"Desc": "The data to be used for training.",
1736+
"Aliases": [
1737+
"train"
1738+
],
1739+
"Required": true,
1740+
"SortOrder": 1.0,
1741+
"IsNullable": false
1742+
},
1743+
{
1744+
"Name": "TestingData",
1745+
"Type": "DataView",
1746+
"Desc": "The data to be used for testing.",
1747+
"Aliases": [
1748+
"test"
1749+
],
1750+
"Required": true,
1751+
"SortOrder": 2.0,
1752+
"IsNullable": false
1753+
},
1754+
{
1755+
"Name": "StateArguments",
1756+
"Type": {
1757+
"Kind": "Component",
1758+
"ComponentKind": "AutoMlStateBase"
1759+
},
1760+
"Desc": "The arguments for creating an AutoMlState component.",
1761+
"Aliases": [
1762+
"args"
1763+
],
1764+
"Required": false,
1765+
"SortOrder": 3.0,
1766+
"IsNullable": false,
1767+
"Default": null
1768+
},
1769+
{
1770+
"Name": "State",
1771+
"Type": {
1772+
"Kind": "C# Object",
1773+
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
1774+
},
1775+
"Desc": "The stateful object conducting of the autoML search.",
1776+
"Aliases": [
1777+
"state"
1778+
],
1779+
"Required": false,
1780+
"SortOrder": 3.0,
1781+
"IsNullable": false,
1782+
"Default": null
1783+
},
1784+
{
1785+
"Name": "BatchSize",
1786+
"Type": "Int",
1787+
"Desc": "Number of candidate pipelines to retrieve each round.",
1788+
"Aliases": [
1789+
"bsize"
1790+
],
1791+
"Required": true,
1792+
"SortOrder": 4.0,
1793+
"IsNullable": false,
1794+
"Default": 0
1795+
},
1796+
{
1797+
"Name": "CandidateOutputs",
1798+
"Type": {
1799+
"Kind": "Array",
1800+
"ItemType": "DataView"
1801+
},
1802+
"Desc": "Output datasets from previous iteration of sweep.",
1803+
"Required": false,
1804+
"SortOrder": 7.0,
1805+
"IsNullable": false,
1806+
"Default": null
1807+
}
1808+
],
1809+
"Outputs": [
1810+
{
1811+
"Name": "State",
1812+
"Type": {
1813+
"Kind": "C# Object",
1814+
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
1815+
},
1816+
"Desc": "Stateful autoML object, keeps track of where the search in progress."
1817+
},
1818+
{
1819+
"Name": "Results",
1820+
"Type": "DataView",
1821+
"Desc": "Results of the sweep, including pipelines (as graph strings), IDs, and metric values."
1822+
}
1823+
]
1824+
},
17261825
{
17271826
"Name": "Models.PlattCalibrator",
17281827
"Desc": "Apply a Platt calibrator to an input model",
@@ -2209,6 +2308,44 @@
22092308
}
22102309
]
22112310
},
2311+
{
2312+
"Name": "Models.SweepResultExtractor",
2313+
"Desc": "Extracts the sweep result.",
2314+
"FriendlyName": null,
2315+
"ShortName": null,
2316+
"Inputs": [
2317+
{
2318+
"Name": "State",
2319+
"Type": {
2320+
"Kind": "C# Object",
2321+
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
2322+
},
2323+
"Desc": "The stateful object conducting of the autoML search.",
2324+
"Aliases": [
2325+
"state"
2326+
],
2327+
"Required": false,
2328+
"SortOrder": 1.0,
2329+
"IsNullable": false,
2330+
"Default": null
2331+
}
2332+
],
2333+
"Outputs": [
2334+
{
2335+
"Name": "State",
2336+
"Type": {
2337+
"Kind": "C# Object",
2338+
"ItemType": "Microsoft.ML.Runtime.EntryPoints.IMlState"
2339+
},
2340+
"Desc": "Stateful autoML object, keeps track of where the search in progress."
2341+
},
2342+
{
2343+
"Name": "Results",
2344+
"Type": "DataView",
2345+
"Desc": "Results of the sweep, including pipelines (as graph strings), IDs, and metric values."
2346+
}
2347+
]
2348+
},
22122349
{
22132350
"Name": "Models.TrainTestBinaryEvaluator",
22142351
"Desc": "Train test for binary classification",
@@ -17200,6 +17337,203 @@
1720017337
}
1720117338
],
1720217339
"Components": [
17340+
{
17341+
"Kind": "AutoMlEngine",
17342+
"Components": [
17343+
{
17344+
"Name": "Defaults",
17345+
"Desc": "AutoML engine that returns learners with default settings.",
17346+
"FriendlyName": "Defaults Engine",
17347+
"Settings": []
17348+
},
17349+
{
17350+
"Name": "Rocket",
17351+
"Desc": "AutoML engine that consists of distinct, hierarchical stages of operation.",
17352+
"FriendlyName": "Rocket Engine",
17353+
"Settings": [
17354+
{
17355+
"Name": "TopKLearners",
17356+
"Type": "Int",
17357+
"Desc": "Number of learners to retain for second stage.",
17358+
"Aliases": [
17359+
"topk"
17360+
],
17361+
"Required": false,
17362+
"SortOrder": 1.0,
17363+
"IsNullable": false,
17364+
"Default": 2
17365+
},
17366+
{
17367+
"Name": "SecondRoundTrialsPerLearner",
17368+
"Type": "Int",
17369+
"Desc": "Number of trials for retained second stage learners.",
17370+
"Aliases": [
17371+
"stage2num"
17372+
],
17373+
"Required": false,
17374+
"SortOrder": 2.0,
17375+
"IsNullable": false,
17376+
"Default": 5
17377+
},
17378+
{
17379+
"Name": "RandomInitialization",
17380+
"Type": "Bool",
17381+
"Desc": "Use random initialization only.",
17382+
"Aliases": [
17383+
"randinit"
17384+
],
17385+
"Required": false,
17386+
"SortOrder": 3.0,
17387+
"IsNullable": false,
17388+
"Default": false
17389+
},
17390+
{
17391+
"Name": "NumInitializationPipelines",
17392+
"Type": "Int",
17393+
"Desc": "Number of initilization pipelines, used for random initialization only.",
17394+
"Aliases": [
17395+
"numinitseeds"
17396+
],
17397+
"Required": false,
17398+
"SortOrder": 4.0,
17399+
"IsNullable": false,
17400+
"Default": 20
17401+
}
17402+
]
17403+
},
17404+
{
17405+
"Name": "UniformRandom",
17406+
"Desc": "AutoML engine using uniform random sampling.",
17407+
"FriendlyName": "Uniform Random Engine",
17408+
"Settings": []
17409+
}
17410+
]
17411+
},
17412+
{
17413+
"Kind": "AutoMlStateBase",
17414+
"Components": [
17415+
{
17416+
"Name": "AutoMlState",
17417+
"Desc": "State of an AutoML search and search space.",
17418+
"FriendlyName": "AutoML State",
17419+
"Aliases": [
17420+
"automlst"
17421+
],
17422+
"Settings": [
17423+
{
17424+
"Name": "Metric",
17425+
"Type": {
17426+
"Kind": "Enum",
17427+
"Values": [
17428+
"Auc",
17429+
"AccuracyMicro",
17430+
"AccuracyMacro",
17431+
"L2",
17432+
"F1",
17433+
"AuPrc",
17434+
"TopKAccuracy",
17435+
"Rms",
17436+
"LossFn",
17437+
"RSquared",
17438+
"LogLoss",
17439+
"LogLossReduction",
17440+
"Ndcg",
17441+
"Dcg",
17442+
"PositivePrecision",
17443+
"PositiveRecall",
17444+
"NegativePrecision",
17445+
"NegativeRecall",
17446+
"DrAtK",
17447+
"DrAtPFpr",
17448+
"DrAtNumPos",
17449+
"NumAnomalies",
17450+
"ThreshAtK",
17451+
"ThreshAtP",
17452+
"ThreshAtNumPos",
17453+
"Nmi",
17454+
"AvgMinScore",
17455+
"Dbi"
17456+
]
17457+
},
17458+
"Desc": "Supported metric for evaluator.",
17459+
"Aliases": [
17460+
"metric"
17461+
],
17462+
"Required": true,
17463+
"SortOrder": 150.0,
17464+
"IsNullable": false,
17465+
"Default": "Auc"
17466+
},
17467+
{
17468+
"Name": "Engine",
17469+
"Type": {
17470+
"Kind": "Component",
17471+
"ComponentKind": "AutoMlEngine"
17472+
},
17473+
"Desc": "AutoML engine (pipeline optimizer) that generates next candidates.",
17474+
"Aliases": [
17475+
"engine"
17476+
],
17477+
"Required": true,
17478+
"SortOrder": 150.0,
17479+
"IsNullable": false
17480+
},
17481+
{
17482+
"Name": "TrainerKind",
17483+
"Type": {
17484+
"Kind": "Enum",
17485+
"Values": [
17486+
"SignatureBinaryClassifierTrainer",
17487+
"SignatureMultiClassClassifierTrainer",
17488+
"SignatureRankerTrainer",
17489+
"SignatureRegressorTrainer",
17490+
"SignatureMultiOutputRegressorTrainer",
17491+
"SignatureAnomalyDetectorTrainer",
17492+
"SignatureClusteringTrainer"
17493+
]
17494+
},
17495+
"Desc": "Kind of trainer for task, such as binary classification trainer, multiclass trainer, etc.",
17496+
"Aliases": [
17497+
"tk"
17498+
],
17499+
"Required": true,
17500+
"SortOrder": 150.0,
17501+
"IsNullable": false,
17502+
"Default": "SignatureBinaryClassifierTrainer"
17503+
},
17504+
{
17505+
"Name": "TerminatorArgs",
17506+
"Type": {
17507+
"Kind": "Component",
17508+
"ComponentKind": "SearchTerminator"
17509+
},
17510+
"Desc": "Arguments for creating terminator, which determines when to stop search.",
17511+
"Aliases": [
17512+
"term"
17513+
],
17514+
"Required": true,
17515+
"SortOrder": 150.0,
17516+
"IsNullable": false
17517+
},
17518+
{
17519+
"Name": "RequestedLearners",
17520+
"Type": {
17521+
"Kind": "Array",
17522+
"ItemType": "String"
17523+
},
17524+
"Desc": "Learner set to sweep over (if available).",
17525+
"Aliases": [
17526+
"learners"
17527+
],
17528+
"Required": false,
17529+
"SortOrder": 150.0,
17530+
"IsNullable": false,
17531+
"Default": null
17532+
}
17533+
]
17534+
}
17535+
]
17536+
},
1720317537
{
1720417538
"Kind": "CalibratorTrainer",
1720517539
"Components": [
@@ -21441,6 +21775,30 @@
2144121775
}
2144221776
]
2144321777
},
21778+
{
21779+
"Kind": "SearchTerminator",
21780+
"Components": [
21781+
{
21782+
"Name": "IterationLimited",
21783+
"Desc": "Terminators a sweep based on total number of iterations.",
21784+
"FriendlyName": "Pipeline Sweep Iteration Terminator",
21785+
"Settings": [
21786+
{
21787+
"Name": "FinalHistoryLength",
21788+
"Type": "Int",
21789+
"Desc": "Total number of iterations.",
21790+
"Aliases": [
21791+
"length"
21792+
],
21793+
"Required": true,
21794+
"SortOrder": 150.0,
21795+
"IsNullable": false,
21796+
"Default": 0
21797+
}
21798+
]
21799+
}
21800+
]
21801+
},
2144421802
{
2144521803
"Kind": "StopWordsRemover",
2144621804
"Components": [

0 commit comments

Comments
 (0)