Conversion of prior and random trainers to estimators #876

artidoro · 2018-09-10T22:17:27Z

Fixes #875.

Converted prior and random trainers to estimators.
Allowed feature column to be null for both prior and random estimator.

Zruty0 · 2018-09-10T23:30:46Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

        }

        public override PredictionKind PredictionKind => PredictionKind.BinaryClassification;

        private static readonly TrainerInfo _info = new TrainerInfo(normalization: false, caching: false);
        public override TrainerInfo Info => _info;

-        public RandomTrainer(IHostEnvironment env, Arguments args)


RandomTrainer [](start = 15, length = 13)

You should not remove this constructor. Once you start writing the test you will discover that it is necessary.

Add a second constructor instead. Both here and for Prior #Closed

Zruty0 · 2018-09-10T23:32:19Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

@@ -38,38 +39,37 @@ namespace Microsoft.ML.Runtime.Learners
    /// <summary>
    /// A trainer that trains a predictor that returns random values
    /// </summary>
-    public sealed class RandomTrainer : TrainerBase<RandomPredictor>
+
+    public sealed class RandomTrainer : TrainerEstimatorBase<BinaryPredictionTransformer<RandomPredictor>, RandomPredictor>


TrainerEstimatorBase [](start = 40, length = 20)

It doesn't help you to derive from this base class, and it also hurts: you now have to take label, feature and weight columns, but you don't actually need them at all.

So, don't derive from TrainerEstimatorBase and just implement ITrainerEstimator #Closed

Zruty0 · 2018-09-10T23:34:03Z

using Float = System.Single;

remove this line and replace all usages of Float with float #Closed

Refers to: src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs:5 in dda393a. [](commit_id = dda393a, deletion_comment = False)

Zruty0 · 2018-09-10T23:42:49Z

    private readonly Random _random;

oh look, a System.Random. We probably want to get rid of this in favor of TauswortheHybrid. Check RandomUtils.Create #Closed

Refers to: src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs:99 in dda393a. [](commit_id = dda393a, deletion_comment = False)

Zruty0 · 2018-09-10T23:44:22Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

-        public RandomTrainer(IHostEnvironment env, Arguments args)
-            : base(env, LoadNameValue)
+        protected override SchemaShape.Column[] OutputColumns => throw new NotImplementedException();
+


That is obviously not sufficient. You need to list the columns that you are going to output. #Closed

Same goes for the other trainer

In reply to: 216508453 [](ancestors = 216508453)

Zruty0

🕐

Zruty0 · 2018-09-11T22:33:43Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

+
+            RoleMappedData trainRoles = new RoleMappedData(cachedTrain);
+            var pred = Train(new TrainContext(trainRoles));
+            return new BinaryPredictionTransformer<RandomPredictor>(Host, pred, cachedTrain.Schema, trainRoles.Schema.Feature.Name);


trainRoles.Schema.Feature.Name [](start = 100, length = 30)

this will make features a required column, which we don't want #Closed

Let's follow through and formally say that feature column is also optional in PredictionTransformerBase: right now it is not

In reply to: 216845418 [](ancestors = 216845418)

Zruty0 · 2018-09-11T22:36:06Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

        public override RandomPredictor Train(TrainContext context)
        {
            Host.CheckValue(context, nameof(context));
            return new RandomPredictor(Host, Host.Rand.Next());
        }
+
+        /// <summary>


summary [](start = 13, length = 7)

this is not a good comment.
A good comment has to describe the purpose of the method, and not its implementation.

In this particular case, I think, it's not necessary to have a summary comment at all, given that you're just implementing the interface, and the method is commented there. #Closed

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

Zruty0 · 2018-09-11T22:39:09Z

Still waiting for tests. #Closed

…rors

Zruty0 · 2018-09-12T16:17:44Z

test/Microsoft.ML.Predictor.Tests/TestPredictors.cs

@@ -91,12 +91,36 @@ public IList<TestDataset> GetDatasetsForClassificationWeightingPredictorsTest()
        [TestCategory("Binary")]
        public void BinaryClassifierPerceptronTest()
        {
-            var binaryPredictors = new[] { TestLearners.perceptron };
+            var binaryPredictors = new[] { TestLearners.perceptron};


on}; [](start = 64, length = 4)

undo this change #Closed

Zruty0 · 2018-09-12T16:19:20Z

...BaselineOutput/SingleDebug/RandomPredictor/BinaryRandom-TrainTest-breast-cancer-test-out.txt

+***** Unexpected failure. Please refer to https://aka.ms/MLNetIssue to file an issue with details *****
+***** Error log has been saved to '%Temp%\%ErrorLog%', please refer to https://aka.ms/MLNetIssue to file an issue with details *****
+===== Begin detailed dump =====
+(1) Unexpected exception: Object reference not set to an instance of an object., 'System.NullReferenceException'


Unexpected exception [](start = 4, length = 20)

this is not a correct behavior #Closed

Zruty0 · 2018-09-12T16:20:10Z

test/BaselineOutput/SingleDebug/PriorPredictor/BinaryPrior-TrainTest-breast-cancer-test-out.txt

+***** Unexpected failure. Please refer to https://aka.ms/MLNetIssue to file an issue with details *****
+***** Error log has been saved to '%Temp%\%ErrorLog%', please refer to https://aka.ms/MLNetIssue to file an issue with details *****
+===== Begin detailed dump =====
+(1) Unexpected exception: Could not find file '%Output% 'System.IO.FileNotFoundException'


Unexpected exception [](start = 3, length = 21)

make sure your code doesn't throw exceptions #Closed

Zruty0 · 2018-09-12T16:21:07Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

@@ -96,7 +112,7 @@ private static VersionInfo GetVersionInfo()
        // Keep all the serializable state here.
        private readonly int _seed;
        private readonly object _instanceLock;
-        private readonly Random _random;
+        private readonly TauswortheHybrid _random;


TauswortheHybrid [](start = 25, length = 16)

would IRandom suffice here? #Closed

…es different outputs

Zruty0 · 2018-09-13T01:13:07Z

src/Microsoft.ML.Data/Scorers/PredictionTransformer.cs

@@ -36,17 +36,18 @@ public abstract class PredictionTransformerBase<TModel> : IPredictionTransformer

        public TModel Model { get; }

-        public PredictionTransformerBase(IHost host, TModel model, ISchema trainSchema, string featureColumn)
+        public PredictionTransformerBase(IHost host, TModel model, ISchema trainSchema, string featureColumn = null)


= null [](start = 108, length = 7)

nope, that should still be required param. Just 'null' should now be acceptable. #Closed

Zruty0 · 2018-09-13T01:13:26Z

src/Microsoft.ML.Data/Scorers/PredictionTransformer.cs

            Host = host;
            Host.CheckValue(trainSchema, nameof(trainSchema));

            Model = model;
            FeatureColumn = featureColumn;
-            if (!trainSchema.TryGetColumnIndex(featureColumn, out int col))
+            if (!trainSchema.TryGetColumnIndex(featureColumn, out int col) && (featureColumn != null))


&& [](start = 75, length = 2)

swap the 2 conditions or you get a wrong error :) #Closed

Zruty0 · 2018-09-13T01:13:50Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

@@ -13,6 +11,8 @@
 using Microsoft.ML.Runtime.Model;
 using Microsoft.ML.Runtime.Training;
 using Microsoft.ML.Runtime.Internal.Internallearn;
+using Microsoft.ML.Core.Data;
+using System.Linq;


System [](start = 6, length = 6)

sort namespaces #Closed

Zruty0 · 2018-09-13T01:16:53Z

Now waiting for test containing TestEstimatorCore

In reply to: 420450005 [](ancestors = 420450005)

…to estimators

sfilipi · 2018-09-17T17:17:48Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

@@ -211,12 +236,16 @@ public sealed class Arguments
        public override TrainerInfo Info => _info;

        public PriorTrainer(IHostEnvironment env, Arguments args)


public [](start = 8, length = 6)

private #Closed

Why should I make the constructor private?

In reply to: 218154524 [](ancestors = 218154524)

sfilipi · 2018-09-17T17:18:12Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

@@ -211,12 +236,16 @@ public sealed class Arguments
        public override TrainerInfo Info => _info;

        public PriorTrainer(IHostEnvironment env, Arguments args)
-            : base(env, LoadNameValue)
+            : base(Contracts.CheckRef(env, nameof(env)).Register(LoadNameValue), MakeFeatureColumn(DefaultColumnNames.Features), MakeLabelColumn(DefaultColumnNames.Label), null)
+        {


{ [](start = 8, length = 1)

keep the args check. #Resolved

artidoro · 2018-09-17T17:42:20Z

    public RandomTrainer(IHostEnvironment env, Arguments args)

Same question as below: Why should this be private? Isn't this the only constructor of RandomTrainer?

In reply to: 422096175 [](ancestors = 422096175)

Refers to: src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs:58 in 3dcb705. [](commit_id = 3dcb705, deletion_comment = False)

Zruty0 · 2018-09-17T17:52:02Z

    public RandomTrainer(IHostEnvironment env, Arguments args)

you should actually have a parameterless constructor, and this one needs to be present just for the component catalog, and it can be made private now.

In reply to: 422106631 [](ancestors = 422106631,422096175)

Refers to: src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs:58 in 3dcb705. [](commit_id = 3dcb705, deletion_comment = False)

Ivanidzo4ka · 2018-09-17T18:51:44Z

test/Microsoft.ML.Predictor.Tests/TestTrainerEstimators.cs

+        [Fact]
+        public void TestEstimatorPrior()
+        {
+            using (var env = new TlcEnvironment())


env [](start = 23, length = 3)

Use baseclass Env, no need to create another one. #Resolved

Ivanidzo4ka · 2018-09-17T20:31:51Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

+        {
+        }
+
+        public PriorTrainer(IHost host, SchemaShape.Column feature, SchemaShape.Column label, SchemaShape.Column weight)


public [](start = 8, length = 6)

Can it be private?
I understand what you use TrainerEstimatorBase which requires feature column, but for this class it's pointless, and exposing it to user would be potentially confusing. #Resolved

Ivanidzo4ka · 2018-09-17T20:33:10Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

-                Float lab = default(Float);
-                Float weight = 1;
+                var getWeight = colWeight >= 0 ? cursor.GetGetter<float>(colWeight) : null;
+                float lab = default(float);


float [](start = 36, length = 5)

you can omit it. #Resolved

Ivanidzo4ka

Ivanidzo4ka · 2018-09-17T20:36:22Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

+        /// </summary>
+        /// <param name="host"></param>
+        /// <param name="feature"></param>
+        /// <param name="label"></param>


either omit, or put something into them #Resolved

Ivanidzo4ka · 2018-09-17T20:36:52Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

        {
            dst = PredictCore();
        }

-        private void MapDist(ref VBuffer<Float> src, ref Float score, ref Float prob)
+        private void MapDist(ref VBuffer<float> src, ref float score, ref float prob)
        {
            score = PredictCore();
            prob = (score + 1) / 2;
        }
    }

    // Learns the prior distribution for 0/1 class labels and just outputs that.


// [](start = 4, length = 2)

Make it ///
#Resolved

…to estimators

Zruty0 · 2018-09-17T22:09:50Z

src/Microsoft.ML.Core/Data/RoleMappedSchema.cs

@@ -534,7 +534,7 @@ public RoleMappedData(IDataView data, IEnumerable<KeyValuePair<RoleMappedSchema.
        /// <param name="opt">Whether to consider the column names specified "optional" or not. If <c>false</c> then any non-empty
        /// values for the column names that does not appear in <paramref name="data"/>'s schema will result in an exception being thrown,
        /// but if <c>true</c> such values will be ignored</param>
-        public RoleMappedData(IDataView data, string label, string feature,
+        public RoleMappedData(IDataView data, string label, string feature = null,


= null [](start = 74, length = 7)

undo this #Resolved

I will handle this in my code then! Sorry about that

In reply to: 218244002 [](ancestors = 218244002)

Zruty0 · 2018-09-17T22:12:23Z

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs

+        /// <summary>
+        /// Initializes PriorTrainer object.
+        /// </summary>
+        public PriorTrainer(IHost host, SchemaShape.Column label, SchemaShape.Column weight)


SchemaShape.Column label [](start = 40, length = 24)

The constructor signature should be
PriorTrainer(IHostEnvironment env, string labelColumn, string weightColumn = null) #Resolved

Ok, I made those Strings instead of SchemShape.Column.
Thank you!

In reply to: 218244513 [](ancestors = 218244513)

sfilipi · 2018-09-17T22:32:19Z

test/Microsoft.ML.Tests/TrainerEstimators/PriorRandomTests.cs

+
+        private static SchemaShape.Column MakeLabelColumn(string labelColumn)
+            => new SchemaShape.Column(labelColumn, SchemaShape.Column.VectorKind.Scalar, NumberType.R4, false);
+


don't think you're using this #Resolved

sfilipi

sfilipi

Ivanidzo4ka · 2018-09-18T17:18:13Z

test/Microsoft.ML.Tests/TrainerEstimators/PriorRandomTests.cs

+            var pipe = new RandomTrainer(Env);
+
+            // Test only that the schema propagation works.
+            // REVIEW: the save/load is not preserving the full state of the random predictor. This is unfortunate, but we don't care too much at this point.


Is it still the case?
If yes, can you write down what "full" state are we missing?

This comment was written by Pete. I don't know exactly what "full state" refers to. I am going to search more in depth to understand what is causing the problem and file a separate issue if it is needed. The behavior that we observed is that the data produced by the model that was saved + loaded back is different from the data produced by the original model.

In reply to: 218520614 [](ancestors = 218520614)

Ivanidzo4ka

work in progress on prior and random estimators

dda393a

artidoro requested a review from Zruty0 September 10, 2018 22:17

Zruty0 reviewed Sep 10, 2018

View reviewed changes

Zruty0 suggested changes Sep 10, 2018

View reviewed changes

artidoro changed the title ~~WIP: Convertion of prior and random trainers to estimators~~ WIP: Conversion of prior and random trainers to estimators Sep 11, 2018

working on conversion of random and prior estimators, fixed constructors

602a581

Zruty0 reviewed Sep 11, 2018

View reviewed changes

src/Microsoft.ML.StandardLearners/Standard/Simple/SimpleTrainers.cs Outdated Show resolved Hide resolved

artidoro added 2 commits September 11, 2018 18:00

got rid of Float, changed random generation function, fixed syntax er…

6adb80b

…rors

working on tests, still not passing random

a04ba3f

Zruty0 reviewed Sep 12, 2018

View reviewed changes

artidoro added 2 commits September 12, 2018 13:52

working on basic tests for random and prior estimators, random produc…

e083f1b

…es different outputs

fixed random predictor test

f07205f

Zruty0 reviewed Sep 13, 2018

View reviewed changes

artidoro added 4 commits September 14, 2018 11:54

working on testestimatorcore

e3e5d90

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

7211756

…to estimators

fixed tests

f05fe55

added output files for release tests

0221dfb

sfilipi reviewed Sep 17, 2018

View reviewed changes

Ivanidzo4ka reviewed Sep 17, 2018

View reviewed changes

artidoro added 3 commits September 17, 2018 13:14

fixed review comments and added constructors

df28a35

no need to create new env

f9dc9d1

unused reference

14c47c0

Ivanidzo4ka reviewed Sep 17, 2018

View reviewed changes

Ivanidzo4ka approved these changes Sep 17, 2018

View reviewed changes

Ivanidzo4ka reviewed Sep 17, 2018

View reviewed changes

artidoro added 2 commits September 17, 2018 13:45

Merge branch 'master' of https://github.com/dotnet/machinelearning in…

d4c8f31

…to estimators

removed dependency on feature column for prior estimator

77392ee

Zruty0 reviewed Sep 17, 2018

View reviewed changes

artidoro added 2 commits September 17, 2018 15:14

fixed review comments

886d514

fixed bug

005e53f

sfilipi reviewed Sep 17, 2018

View reviewed changes

sfilipi approved these changes Sep 17, 2018

View reviewed changes

keeping constructors public and fix review comments

c661496

sfilipi approved these changes Sep 17, 2018

View reviewed changes

Ivanidzo4ka reviewed Sep 18, 2018

View reviewed changes

Ivanidzo4ka approved these changes Sep 18, 2018

View reviewed changes

retriggering build

2042edd

artidoro force-pushed the estimators branch from 67211cc to 2042edd Compare September 18, 2018 23:14

Zruty0 merged commit 7b1c7d7 into dotnet:master Sep 19, 2018

artidoro deleted the estimators branch January 5, 2019 00:02

ghost locked as resolved and limited conversation to collaborators Mar 29, 2022

		@@ -211,12 +236,16 @@ public sealed class Arguments
		public override TrainerInfo Info => _info;

		public PriorTrainer(IHostEnvironment env, Arguments args)


		private static SchemaShape.Column MakeLabelColumn(string labelColumn)
		=> new SchemaShape.Column(labelColumn, SchemaShape.Column.VectorKind.Scalar, NumberType.R4, false);

Conversion of prior and random trainers to estimators #876

Conversion of prior and random trainers to estimators #876

Conversation

artidoro commented Sep 10, 2018 • edited Loading

Zruty0 Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 commented Sep 10, 2018 • edited Loading

Zruty0 commented Sep 10, 2018 • edited Loading

Zruty0 Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 10, 2018

Choose a reason for hiding this comment

Zruty0 left a comment

Choose a reason for hiding this comment

Zruty0 Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 12, 2018

Choose a reason for hiding this comment

Zruty0 Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 commented Sep 11, 2018 • edited Loading

Zruty0 Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 12, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Zruty0 commented Sep 13, 2018

sfilipi Sep 17, 2018 • edited by Zruty0 Loading

Choose a reason for hiding this comment

artidoro Sep 17, 2018

Choose a reason for hiding this comment

sfilipi Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

artidoro commented Sep 17, 2018 • edited Loading

Zruty0 commented Sep 17, 2018

Ivanidzo4ka Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Zruty0 Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

artidoro Sep 17, 2018

Choose a reason for hiding this comment

Zruty0 Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

artidoro Sep 17, 2018 • edited Loading

Choose a reason for hiding this comment

sfilipi Sep 17, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

sfilipi left a comment

Choose a reason for hiding this comment

Ivanidzo4ka Sep 18, 2018

Choose a reason for hiding this comment

artidoro Sep 18, 2018

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

artidoro commented Sep 10, 2018 •

edited

Loading

Zruty0 Sep 10, 2018 •

edited

Loading

Zruty0 Sep 10, 2018 •

edited

Loading

Zruty0 commented Sep 10, 2018 •

edited

Loading

Zruty0 commented Sep 10, 2018 •

edited

Loading

Zruty0 Sep 10, 2018 •

edited

Loading

Zruty0 Sep 11, 2018 •

edited

Loading

Zruty0 Sep 11, 2018 •

edited

Loading

Zruty0 commented Sep 11, 2018 •

edited

Loading

Zruty0 Sep 12, 2018 •

edited

Loading

Zruty0 Sep 12, 2018 •

edited

Loading

Zruty0 Sep 12, 2018 •

edited

Loading

Zruty0 Sep 12, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

Zruty0 Sep 13, 2018 •

edited

Loading

sfilipi Sep 17, 2018 •

edited by Zruty0

Loading

sfilipi Sep 17, 2018 •

edited by artidoro

Loading

artidoro commented Sep 17, 2018 •

edited

Loading

Ivanidzo4ka Sep 17, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Sep 17, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Sep 17, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Sep 17, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Sep 17, 2018 •

edited by artidoro

Loading

Zruty0 Sep 17, 2018 •

edited by artidoro

Loading

Zruty0 Sep 17, 2018 •

edited by artidoro

Loading

artidoro Sep 17, 2018 •

edited

Loading

sfilipi Sep 17, 2018 •

edited by artidoro

Loading