Clean up of TextLoader constructor #1784

artidoro · 2018-11-29T22:23:12Z

Hid the constructor of TextLoader that takes Arguments, and exposed HasHeader and SeparatorChars as non-advanced parameters.
Made Create methods internal and modified the code accordingly.
Added comments for the public facing constructor that was retained.

Ivanidzo4ka · 2018-11-29T22:29:42Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoader.cs

+        /// <param name="env">The environment to use.</param>
+        /// <param name="columns">Defines a mapping between input columns in the file and IDataView columns.</param>
+        /// <param name="hasHeader">Whether the file has a header.</param>
+        /// <param name="separatorChars">Defines the characters used as separators between data points in a row. By default the tab character is taken as separator.</param>


By default the tab character [](start = 112, length = 29)

this statement and char[] separatorChars = null a bit weird.
I know what latter down the line we probably check for null in separators, and use tab as default, but still. #Resolved

Yes it is not ideal, that's why I have an explanation in the documentation above. But if you have a better idea, I would be happy to take it!

In reply to: 237683995 [](ancestors = 237683995)

eerhardt · 2018-11-30T21:24:53Z

test/Microsoft.ML.Tests/Scenarios/Api/CookbookSamples/CookbookSamplesDynamicApi.cs

                    // We read the first 11 values as a single float vector.
                    new TextLoader.Column("FeatureVector", DataKind.R4, 0, 10),

                    // Separately, read the target variable.
                    new TextLoader.Column("Target", DataKind.R4, 11),
                },
                // First line of the file is a header, not a data row.
-                HasHeader = true,
+                true,


I'd still qualify hasHeader: here. #Resolved

Apply this everywhere - especially in docs/samples/etc.

In reply to: 238006767 [](ancestors = 238006767)

eerhardt · 2018-11-30T21:25:46Z

test/Microsoft.ML.Tests/Scenarios/Api/CookbookSamples/CookbookSamplesDynamicApi.cs

                // Default separator is tab, but we need a semicolon.
-                Separator = ";"
-            });
+                new[] { ';' }


Isn't the single separator the more common case? Maybe the "simple" constructor just takes a single character separator. And the "advanced" case can support multiple separators. #Resolved

eerhardt · 2018-11-30T21:34:49Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoaderSaverCatalog.cs

-        /// <param name="catalog">The catalog.</param>
-        /// <param name="args">The arguments to text reader, describing the data schema.</param>
-        /// <param name="dataSample">The optional location of a data sample.</param>
-        public static TextLoader TextReader(this DataOperations catalog,


Yesterday in the API design review, we decided against this approach. See the notes from the review here: https://github.com/dotnet/apireviews/pull/81/files

But basically, the pattern decided will be:

Make a simple constructor/factory that has the most common parameters.

If there are advanced parameters that we don't want exposed in Get a working build #1, then make another constructor/factory that takes the only the nested Arguments class (to be renamed to "Options").

We are going to move away from the Action<Arguments> advancedSettings approach. One main reason is because there can be conflicts between the "simple" parameters and the "advanced" parameters - and which one should win? Another reasoning is that it is simpler and understandable to construct and pass an object to a method.

I'd say, for this change, let's not move away from where we are going. You don't need to rename Arguments to Options. But let's leave this overload, and remove the "advancedSettings" parameter below instead. #Resolved

I have left the overload as you suggested and removed the advanced arguments parameter from the constructors.

In reply to: 238009119 [](ancestors = 238009119)

artidoro · 2018-11-30T21:45:22Z

Thanks @eerhardt! I'll update accordingly #Resolved

eerhardt · 2018-11-30T22:09:19Z

Sorry for the "late breaking" change here, but I thought it would be good to note where we landed yesterday, and not have to redo some of this work. #Resolved

artidoro · 2018-12-03T18:57:37Z

I am doing the changes and it looks a lot better, it's good to see that!

In reply to: 443354726 [](ancestors = 443354726)

Ivanidzo4ka · 2018-12-05T17:19:22Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoaderSaverCatalog.cs

-        /// <param name="advancedSettings">The delegate to set additional settings</param>
-        /// <param name="path">The path to the file</param>
+        /// <param name="hasHeader">Whether the file has a header.</param>
+        /// <param name="separatorChar"> The character used as separator between data points in a row. By default the tab character is used as separator.</param>


[](start = 40, length = 1)

extra space, here and in constructor above. #Resolved

Ivanidzo4ka · 2018-12-05T17:23:54Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoaderSaverCatalog.cs

+
+            var env = catalog.GetEnvironment();
+
+            // REVIEW: it is almost always a mistake to have a 'trainable' text loader here.


// REVIEW: it is almost always a mistake to have a 'trainable' text loader here [](start = 12, length = 79)

Did it work well if you specify header to true and didn't pass dataSample? #Resolved

Yes, check this test public void CustomTransformer()

In reply to: 239161201 [](ancestors = 239161201)

Ivanidzo4ka · 2018-12-05T17:26:47Z

src/Microsoft.ML.Data/Utilities/ModelFileUtils.cs

@@ -283,8 +283,7 @@ internal static void SaveRoleMappings(IHostEnvironment env, IChannel ch, RoleMap
            {
                // REVIEW: Should really validate the schema here, and consider
                // ignoring this stream if it isn't as expected.
-                var loader = TextLoader.ReadFile(env, new TextLoader.Arguments(),
-                    new RepositoryStreamWrapper(rep, DirTrainingInfo, RoleMappingFile));
+                var loader = TextLoader.ReadFile(env, new RepositoryStreamWrapper(rep, DirTrainingInfo, RoleMappingFile), new TextLoader.Arguments());


, new TextLoader.Arguments() [](start = 120, length = 28)

can we change method to have arg default = null, and if it's null to use new TextLoader.Arguments()? #Resolved

Sure, I can do that. Do you think I should change that in the constructor of TextLoader too?

In reply to: 239162303 [](ancestors = 239162303)

Yes I think it makes sense

In reply to: 239258487 [](ancestors = 239258487,239162303)

Ivanidzo4ka · 2018-12-05T17:27:51Z

test/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs

@@ -64,30 +64,29 @@ public void TrainSentiment()
        {
            var env = new MLContext(seed: 1);
            // Pipeline
-            var loader = TextLoader.ReadFile(env,
-                new TextLoader.Arguments()
+            var arguemnts = new TextLoader.Arguments()


arguemnts [](start = 16, length = 9)

msitype #Resolved

Ivanidzo4ka · 2018-12-05T17:30:18Z

src/Microsoft.ML.Transforms/TermLookupTransformer.cs

@@ -369,7 +364,11 @@ public TermLookupTransformer(IHostEnvironment env, IDataView input, IDataView lo
                var txtArgs = new TextLoader.Arguments();
                bool parsed = CmdParser.ParseArguments(host, "col=Term:TX:0 col=Value:TX:1", txtArgs);


you don't need this anymore #Resolved

Ivanidzo4ka · 2018-12-05T17:31:16Z

test/Microsoft.ML.Predictor.Tests/TestPredictors.cs

@@ -595,7 +595,7 @@ public void RankingLightGBMTest()
        public void TestTreeEnsembleCombiner()
        {
            var dataPath = GetDataPath("breast-cancer.txt");
-            var dataView = TextLoader.Create(Env, new TextLoader.Arguments(), new MultiFileSource(dataPath));
+            var dataView = TextLoader.ReadFile(Env, new MultiFileSource(dataPath), new TextLoader.Arguments());


ReadFile [](start = 38, length = 8)

can you use ReadTextFile instead? #ByDesign

Isn't text implied by the name of TextLoader? #Resolved

Ivanidzo4ka · 2018-12-05T17:31:29Z

test/Microsoft.ML.Predictor.Tests/TestPredictors.cs

@@ -617,7 +617,7 @@ public void TestTreeEnsembleCombiner()
        public void TestTreeEnsembleCombinerWithCategoricalSplits()
        {
            var dataPath = GetDataPath("adult.tiny.with-schema.txt");
-            var dataView = TextLoader.Create(Env, new TextLoader.Arguments(), new MultiFileSource(dataPath));
+            var dataView = TextLoader.ReadFile(Env, new MultiFileSource(dataPath), new TextLoader.Arguments());


ReadFile [](start = 38, length = 8)

ReadTextFile #ByDesign

across whole file

In reply to: 239164058 [](ancestors = 239164058)

Isn't text implied by the name of TextLoader? #Resolved

Ivanidzo4ka · 2018-12-05T17:32:09Z

test/Microsoft.ML.TestFramework/DataPipe/TestDataPipeBase.cs

@@ -438,7 +438,7 @@ protected void VerifyArgParsing(IHostEnvironment env, string[] strs)

            // Note that we don't pass in "args", but pass in a default args so we test
            // the auto-schema parsing.
-            var loadedData = TextLoader.ReadFile(env, new TextLoader.Arguments(), new MultiFileSource(pathData));
+            var loadedData = TextLoader.ReadFile(env, new MultiFileSource(pathData), new TextLoader.Arguments());


ReadFile [](start = 40, length = 8)

ReadTextFile? #ByDesign

Isn't text implied by the name of TextLoader? #Resolved

I agree with Eric, since it's a method of TextLoader it should already implied that it loads a text file. #Resolved

Ivanidzo4ka · 2018-12-05T17:32:58Z

test/Microsoft.ML.Tests/Scenarios/Api/CookbookSamples/CookbookSamplesDynamicApi.cs

-            var reader = mlContext.Data.TextReader(new TextLoader.Arguments
-            {
-                Column = new[] {
+            var reader = mlContext.Data.TextReader(new[] {


TextReader [](start = 40, length = 10)

please update https://github.com/dotnet/machinelearning/blob/master/docs/code/MlNetCookBook.md with your changes. #Resolved

Ivanidzo4ka

eerhardt · 2018-12-06T00:29:33Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoaderSaverCatalog.cs

-        /// <param name="args">The arguments to text reader, describing the data schema.</param>
+        /// <param name="columns">The columns of the schema.</param>
+        /// <param name="hasHeader">Whether the file has a header.</param>
+        /// <param name="separatorChar">The character used as separator between data points in a row. By default the tab character is used as separator.</param>
        /// <param name="dataSample">The optional location of a data sample.</param>
        public static TextLoader TextReader(this DataOperations catalog,


I think we want these named CreateTextReader. See https://github.com/dotnet/apireviews/pull/81/files #Resolved

eerhardt · 2018-12-06T00:31:10Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoader.cs

+        /// <param name="hasHeader">Whether the file has a header.</param>
+        /// <param name="separatorChar"> The character used as separator between data points in a row. By default the tab character is used as separator.</param>
+        /// <param name="fileSource">Specifies a file from which to read.</param>
+        public static IDataView ReadFile(IHostEnvironment env, IMultiStreamSource fileSource, Column[] columns, bool hasHeader = false, char separatorChar = '\t')


Why do we have both IDataView ReadFromTextFile(this DataOperations catalog, and these methods? I think we should only have 1. #Resolved

artidoro · 2018-12-06T18:41:49Z

@eerhardt I just removed the ReadFile method from TextLoader, as you suggested. #Resolved

artidoro · 2018-12-06T18:51:58Z

I am actually updating the cookbook again, to reflect that and the new name for the MlContext extension.

In reply to: 444982206 [](ancestors = 444982206)

eerhardt · 2018-12-07T15:53:02Z

src/Microsoft.ML.Data/DataLoadSave/Text/TextLoader.cs

        {
-            var result = new Arguments { Column = columns };
-            advancedSettings?.Invoke(result);
+            separatorChars = separatorChars ?? new[] { '\t' };


(nit) separatorChars can never be null, right? This is a private method and only called in 1 spot that ensures it won't be null.

Maybe just add an Assert it won't be null and you can remove the null check here. #Resolved

eerhardt · 2018-12-07T15:59:31Z

src/Microsoft.ML.Data/Utilities/ModelFileUtils.cs

-                var loader = TextLoader.ReadFile(env, new TextLoader.Arguments(),
-                    new RepositoryStreamWrapper(rep, DirTrainingInfo, RoleMappingFile));
+                var loader = new TextLoader(env, dataSample: new RepositoryStreamWrapper(rep, DirTrainingInfo, RoleMappingFile))
+                    .Read(new RepositoryStreamWrapper(rep, DirTrainingInfo, RoleMappingFile));


We shouldn't create two instances of new RepositoryStreamWrapper(rep, DirTrainingInfo, RoleMappingFile). #Resolved

Maybe it may be valuable to still have an internal TextLoader.ReadFile helper method for our internal code.

In reply to: 239856914 [](ancestors = 239856914)

eerhardt · 2018-12-07T16:01:05Z

src/Microsoft.ML.Transforms/TermLookupTransformer.cs

+                        new TextLoader.Column("Term", DataKind.TX, 0),
+                        new TextLoader.Column("Value", DataKind.TX, 1)
+                    },
+                    dataSample: new MultiFileSource(filename)


Same comment here about creating duplicate objects - new MultiFileSource(filename) #Resolved

eerhardt

Looks good. Just a couple minor comments to clean up.

artidoro requested review from Ivanidzo4ka, TomFinley and sfilipi November 29, 2018 22:23

Ivanidzo4ka reviewed Nov 29, 2018

View reviewed changes

eerhardt reviewed Nov 30, 2018

View reviewed changes

Ivanidzo4ka reviewed Dec 5, 2018

View reviewed changes

Ivanidzo4ka approved these changes Dec 5, 2018

View reviewed changes

artidoro force-pushed the textloader branch from 7a156c8 to 490e03f Compare December 5, 2018 23:46

eerhardt reviewed Dec 6, 2018

View reviewed changes

artidoro added 6 commits December 6, 2018 10:43

textloader refactoring of constructor

a6e4fd8

added comments and fixed bugs

75254c5

made create methods internal and adjusted code accordingly

2986558

kept overload with Arguments and removed advanced settings

f04cd68

review comments

d90c116

removed ReadFile, only kept MlContext ReadFromTextFile

a52849c

artidoro force-pushed the textloader branch from 9d182f3 to a52849c Compare December 6, 2018 18:44

artidoro added 2 commits December 6, 2018 10:57

updated Cookbook.md, and renamed static extension

358a709

fixed bug

52c91a2

eerhardt reviewed Dec 7, 2018

View reviewed changes

eerhardt approved these changes Dec 7, 2018

View reviewed changes

review comments, added back the readfile method but made it internal

8643afe

artidoro self-assigned this Dec 7, 2018

artidoro added the API Issues pertaining the friendly API label Dec 7, 2018

artidoro merged commit 14c7a47 into dotnet:master Dec 7, 2018

artidoro mentioned this pull request Dec 21, 2018

Improve constructor for TextLoader #1288

Closed

artidoro deleted the textloader branch January 5, 2019 00:01

najeeb-kazmi mentioned this pull request Jan 7, 2019

Rename mlContext.Data.TextReader() to mlContext.Data.CreateTextLoader() #1690

Closed

ghost locked as resolved and limited conversation to collaborators Mar 26, 2022


		var env = catalog.GetEnvironment();

		// REVIEW: it is almost always a mistake to have a 'trainable' text loader here.

		@@ -369,7 +364,11 @@ public TermLookupTransformer(IHostEnvironment env, IDataView input, IDataView lo
		var txtArgs = new TextLoader.Arguments();
		bool parsed = CmdParser.ParseArguments(host, "col=Term:TX:0 col=Value:TX:1", txtArgs);

Clean up of TextLoader constructor #1784

Clean up of TextLoader constructor #1784

Conversation

artidoro commented Nov 29, 2018

Ivanidzo4ka Nov 29, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt Nov 30, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt Nov 30, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

eerhardt Nov 30, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artidoro commented Nov 30, 2018 • edited Loading

eerhardt commented Nov 30, 2018 • edited by artidoro Loading

artidoro commented Dec 3, 2018

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

eerhardt Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

eerhardt Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

artidoro Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

Ivanidzo4ka Dec 5, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

eerhardt Dec 6, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

eerhardt Dec 6, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

artidoro commented Dec 6, 2018 • edited Loading

artidoro commented Dec 6, 2018

eerhardt Dec 7, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

eerhardt Dec 7, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eerhardt Dec 7, 2018 • edited by artidoro Loading

Choose a reason for hiding this comment

eerhardt left a comment

Choose a reason for hiding this comment

Ivanidzo4ka Nov 29, 2018 •

edited by artidoro

Loading

eerhardt Nov 30, 2018 •

edited by artidoro

Loading

eerhardt Nov 30, 2018 •

edited by artidoro

Loading

eerhardt Nov 30, 2018 •

edited by artidoro

Loading

artidoro commented Nov 30, 2018 •

edited

Loading

eerhardt commented Nov 30, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

eerhardt Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

eerhardt Dec 5, 2018 •

edited by artidoro

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

eerhardt Dec 5, 2018 •

edited by artidoro

Loading

artidoro Dec 5, 2018 •

edited

Loading

Ivanidzo4ka Dec 5, 2018 •

edited by artidoro

Loading

eerhardt Dec 6, 2018 •

edited by artidoro

Loading

eerhardt Dec 6, 2018 •

edited by artidoro

Loading

artidoro commented Dec 6, 2018 •

edited

Loading

eerhardt Dec 7, 2018 •

edited by artidoro

Loading

eerhardt Dec 7, 2018 •

edited by artidoro

Loading

eerhardt Dec 7, 2018 •

edited by artidoro

Loading