Multi column MapKeyToValue and MapValueToKey #3187

sfilipi · 2019-04-03T06:31:10Z

Towards #1209 more samples for MapKeyToValue and MapValueToKey

codecov · 2019-04-03T07:12:31Z

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3187      +/-   ##
==========================================
- Coverage   72.54%   72.53%   -0.01%     
==========================================
  Files         807      807              
  Lines      144774   144774              
  Branches    16208    16208              
==========================================
- Hits       105021   105017       -4     
- Misses      35339    35343       +4     
  Partials     4414     4414

Flag	Coverage Δ
#Debug	`72.53% <ø> (-0.01%)`	⬇️
#production	`68.12% <ø> (-0.01%)`	⬇️
#test	`88.82% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs	`44.87% <ø> (ø)`	⬆️
src/Microsoft.ML.Maml/MAML.cs	`24.75% <0%> (-1.46%)`	⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs	`89.26% <0%> (-0.63%)`	⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs	`86.26% <0%> (+0.15%)`	⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs	`85.11% <0%> (+0.4%)`	⬆️

codecov · 2019-04-03T07:13:06Z

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3187      +/-   ##
==========================================
+ Coverage    72.6%   72.62%   +0.01%     
==========================================
  Files         807      807              
  Lines      145077   145080       +3     
  Branches    16213    16213              
==========================================
+ Hits       105337   105366      +29     
+ Misses      35322    35296      -26     
  Partials     4418     4418

Flag	Coverage Δ
#Debug	`72.62% <ø> (+0.01%)`	⬆️
#production	`68.17% <ø> (+0.02%)`	⬆️
#test	`88.92% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs	`64.07% <ø> (ø)`	⬆️
src/Microsoft.ML.Maml/MAML.cs	`24.75% <0%> (-1.46%)`	⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs	`78.87% <0%> (ø)`	⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs	`70.39% <0%> (ø)`	⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs	`86.21% <0%> (ø)`	⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs	`97.31% <0%> (+0.05%)`	⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs	`84.9% <0%> (+0.2%)`	⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs	`91.12% <0%> (+1.02%)`	⬆️
...LogisticRegression/MulticlassLogisticRegression.cs	`67.61% <0%> (+1.74%)`	⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs	`92.34% <0%> (+3.27%)`	⬆️
... and 1 more

abgoswam · 2019-04-03T16:38:52Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs

+
+            // at this point, the Label colum is tranformed from strings, to DataViewKeyType and
+            // the transformation has added the PredictedLabel column, with 
+            var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[] 


newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.

In reply to: 271831815 [](ancestors = 271831815)

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question

In reply to: 271866086 [](ancestors = 271866086,271831815)

abgoswam · 2019-04-03T16:58:59Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyInputOutputPair.cs

+                new LookupMap { Key = "6-11yrs" },
+                new LookupMap { Key = "25+yrs"  }
+
+            };


}; [](start = 12, length = 2)

whitespace #Resolved

abgoswam · 2019-04-03T17:06:00Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -100,8 +100,9 @@ internal static TypeConvertingEstimator ConvertType(this TransformsCatalog.Conve
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
-        ///  [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
-        /// ]]></format>
+        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]


KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

abgoswam · 2019-04-03T17:07:54Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -173,7 +181,7 @@ public static KeyToValueMappingEstimator MapKeyToValue(this TransformsCatalog.Co
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
-        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
+        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]


ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

abgoswam · 2019-04-03T17:08:13Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -173,7 +181,7 @@ public static KeyToValueMappingEstimator MapKeyToValue(this TransformsCatalog.Co
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
-        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
+        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]


MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

abgoswam · 2019-04-03T17:08:43Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

+        /// <example>
+        /// <format type="text/markdown">
+        /// <![CDATA[
+        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]


ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved

abgoswam · 2019-04-03T17:51:23Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs

+
+namespace Microsoft.ML.Samples.Dynamic
+{
+    public class MapKeyToValueInputOutputPair


MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved

abgoswam · 2019-04-03T17:52:13Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyInputOutputPair.cs

+
+namespace Microsoft.ML.Samples.Dynamic
+{
+    public static class MapValueToKeyInputOutputPair


MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

abgoswam · 2019-04-03T20:48:13Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -173,7 +181,7 @@ public static KeyToValueMappingEstimator MapKeyToValue(this TransformsCatalog.Co
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
-        /// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
+        /// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]


Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

abgoswam · 2019-04-03T20:52:01Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

+        /// <example>
+        /// <format type="text/markdown">
+        /// <![CDATA[
+        /// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]


KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

i had them on the same file, at some point :)

In reply to: 271927396 [](ancestors = 271927396)

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs

In reply to: 271930498 [](ancestors = 271930498,271927396)

abgoswam · 2019-04-03T21:15:40Z

src/Microsoft.ML.Data/Transforms/ConversionsExtensionsCatalog.cs

@@ -100,8 +100,9 @@ internal static TypeConvertingEstimator ConvertType(this TransformsCatalog.Conve
        /// <example>
        /// <format type="text/markdown">
        /// <![CDATA[
-        ///  [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
-        /// ]]></format>
+        /// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]


KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

https://github.com/dotnet/machinelearning/blob/master/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs

In reply to: 271935868 [](ancestors = 271935868)

updating it.

In reply to: 271959976 [](ancestors = 271959976,271935868)

abgoswam

shmoradims · 2019-04-04T00:05:23Z

...osoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/StochasticDualCoordinateAscent.cs


-namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification


.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

I thought every class had its own distinctive name?

In reply to: 271977575 [](ancestors = 271977575)

ic, we call them the same between tasks. Reverting.

In reply to: 272288927 [](ancestors = 272288927,271977575)

shmoradims · 2019-04-04T00:07:22Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueMultiColumn.cs

+            var mlContext = new MLContext(seed: 0);
+
+            // Create a list of data examples.
+            var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);


can we have inline data like the other sample? #Resolved

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.

In reply to: 271977941 [](ancestors = 271977941)

Gani's PR is not checked in. I can switch the Generate after he checks in.

In reply to: 272332287 [](ancestors = 272332287,271977941)

shmoradims · 2019-04-04T00:12:22Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyMultiColumn.cs

+
+            // Get a small dataset as an IEnumerable.
+            var rawData = new[] {
+                new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },


6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.

In reply to: 271978738 [](ancestors = 271978738)

Addresssing Abhishek's comments

abgoswam · 2019-04-05T15:59:35Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyMultiColumn.cs

+
+            // TransformedData obtained post-transformation.
+            //
+            //  StudyTime   StudyTimeCategory   DevelopmentTime    DevelopmentTimeCategory


DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

abgoswam · 2019-04-05T16:00:17Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyMultiColumn.cs

+            // This will contain the newly created columns.
+            features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);
+
+            Console.WriteLine($" StudyTime   StudyTimeCategory   DevelopmentTime    DevelopmentTimeCategory");


DevelopmentTime [](start = 65, length = 15)

Course #Resolved

shmoradims · 2019-04-05T21:54:30Z

docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyMultiColumn.cs

+            // This will contain the newly created columns.
+            features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);
+
+            Console.WriteLine($" StudyTime   StudyTimeCategory   DevelopmentTime    DevelopmentTimeCategory");


DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

shmoradims

* Multi column MapKeyToValue and MapValueToKey

sfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019

sfilipi requested review from shmoradims, singlis, abgoswam, Ivanidzo4ka and zeahmed April 3, 2019 06:31

sfilipi self-assigned this Apr 3, 2019

abgoswam reviewed Apr 3, 2019

View reviewed changes

abgoswam approved these changes Apr 3, 2019

View reviewed changes

shmoradims reviewed Apr 4, 2019

View reviewed changes

sfilipi force-pushed the moreConversionSamples branch from 4171b5b to a5758c3 Compare April 4, 2019 18:12

sfilipi added 5 commits April 5, 2019 08:32

Multi column MapKeyToValue and MapValueToKey

0cb5c91

renaming per the new convention

5d0ad95

Addresssing Abhishek's comments

fixing mismatch

5bf777a

fix reference from xml

1e7cd06

reverting name

bff9d9a

sfilipi added 3 commits April 5, 2019 08:32

addressing PR issue, and merge.

aa091bf

PR comments

aa526aa

changing namespace and removing the train/test split

4e6c4af

sfilipi force-pushed the moreConversionSamples branch from a5758c3 to 4e6c4af Compare April 5, 2019 15:45

abgoswam reviewed Apr 5, 2019

View reviewed changes

shmoradims reviewed Apr 5, 2019

View reviewed changes

shmoradims approved these changes Apr 5, 2019

View reviewed changes

PR review comments

e111f07

sfilipi merged commit 7a0d395 into dotnet:master Apr 7, 2019

sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11

sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019

Multi column MapKeyToValue and MapValueToKey (dotnet#3187)

c0f80a9

* Multi column MapKeyToValue and MapValueToKey

sfilipi mentioned this pull request Apr 9, 2019

Release cherry picks for Conversions samples and removing gplv2 files #3264

Closed

codemzs mentioned this pull request Apr 12, 2019

Standalone app to run all samples to catch run time exceptions #3308

Closed

ghost locked as resolved and limited conversation to collaborators Mar 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi column MapKeyToValue and MapValueToKey #3187

Multi column MapKeyToValue and MapValueToKey #3187

sfilipi commented Apr 3, 2019

codecov bot commented Apr 3, 2019

codecov bot commented Apr 3, 2019 •

edited

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

sfilipi Apr 3, 2019

abgoswam Apr 3, 2019

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

sfilipi Apr 3, 2019

abgoswam Apr 3, 2019

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

sfilipi Apr 3, 2019

sfilipi Apr 3, 2019

abgoswam left a comment

shmoradims Apr 4, 2019 •

edited by sfilipi

Loading

sfilipi Apr 4, 2019

sfilipi Apr 4, 2019

shmoradims Apr 4, 2019 •

edited by sfilipi

Loading

sfilipi Apr 4, 2019

sfilipi Apr 5, 2019

shmoradims Apr 4, 2019 •

edited by sfilipi

Loading

sfilipi Apr 4, 2019

abgoswam Apr 5, 2019 •

edited by sfilipi

Loading

abgoswam Apr 5, 2019 •

edited by sfilipi

Loading

shmoradims Apr 5, 2019 •

edited by sfilipi

Loading

shmoradims left a comment


		namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

Multi column MapKeyToValue and MapValueToKey #3187

Multi column MapKeyToValue and MapValueToKey #3187

Conversation

sfilipi commented Apr 3, 2019

codecov bot commented Apr 3, 2019

Codecov Report

codecov bot commented Apr 3, 2019 • edited Loading

Codecov Report

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam Apr 3, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam left a comment

Choose a reason for hiding this comment

shmoradims Apr 4, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Apr 4, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmoradims Apr 4, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abgoswam Apr 5, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

abgoswam Apr 5, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims Apr 5, 2019 • edited by sfilipi Loading

Choose a reason for hiding this comment

shmoradims left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 3, 2019 •

edited

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

abgoswam Apr 3, 2019 •

edited by sfilipi

Loading

shmoradims Apr 4, 2019 •

edited by sfilipi

Loading

shmoradims Apr 4, 2019 •

edited by sfilipi

Loading

shmoradims Apr 4, 2019 •

edited by sfilipi

Loading

abgoswam Apr 5, 2019 •

edited by sfilipi

Loading

abgoswam Apr 5, 2019 •

edited by sfilipi

Loading

shmoradims Apr 5, 2019 •

edited by sfilipi

Loading