Error due to ShuffleTransform in pipeline.

[zeahmed]

Need to convert ShuffleTransform into Transformer/Estimator design. Currently, when it is used in TensorFlowTransform to enable shuffling of data during training, it gives error regarding ShuffleTransform is not RowToRowMapper.

Update

It seems like the transform works fine during training. The error is somewhere when creating prediction engine.

The following is the exact location where assertion fails during creation of prediction engine.

machinelearning/src/Microsoft.ML.Data/DataLoadSave/TransformWrapper.cs

Line 133 in 8ca1c93

_host.Assert(xf is IRowToRowMapper);

Here is the test failure log

[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8050239]         E:\TLC_git\machinelearning\test\Microsoft.ML.TestFramework\GlobalBase.cs(70,0): at Microsoft.ML.Runtime.Internal.Internallearn.Test.GlobalBase.AssertHandler(String msg, IExceptionContext ectx)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8060923]         E:\TLC_git\machinelearning\src\Microsoft.ML.Core\Utilities\Contracts.cs(777,0): at Microsoft.ML.Runtime.Contracts.DbgFailCore(String msg, IExceptionContext ctx)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8064869]         E:\TLC_git\machinelearning\src\Microsoft.ML.Core\Utilities\Contracts.cs(786,0): at Microsoft.ML.Runtime.Contracts.DbgFail(IExceptionContext ctx)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8067575]         E:\TLC_git\machinelearning\src\Microsoft.ML.Core\Utilities\Contracts.cs(841,0): at Microsoft.ML.Runtime.Contracts.Assert(IExceptionContext ctx, Boolean f)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8070263]         E:\TLC_git\machinelearning\src\Microsoft.ML.Data\DataLoadSave\TransformWrapper.cs(133,0): at Microsoft.ML.Runtime.Data.TransformWrapper.GetRowToRowMapper(ISchema inputSchema)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8073429]         E:\TLC_git\machinelearning\src\Microsoft.ML.Api\PredictionEngine.cs(190,0): at Microsoft.ML.Runtime.Api.PredictionEngine`2..ctor(IHostEnvironment env, Func`2 makeMapper, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8076450]         E:\TLC_git\machinelearning\src\Microsoft.ML.Api\PredictionEngine.cs(172,0): at Microsoft.ML.Runtime.Api.PredictionEngine`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8078831]         E:\TLC_git\machinelearning\src\Microsoft.ML.Api\PredictionEngine.cs(166,0): at Microsoft.ML.Runtime.Api.PredictionEngine`2..ctor(IHostEnvironment env, IDataView dataPipe, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8081564]         E:\TLC_git\machinelearning\src\Microsoft.ML.Api\ComponentCreation.cs(190,0): at Microsoft.ML.Runtime.Api.ComponentCreation.CreatePredictionEngine[TSrc,TDst](IHostEnvironment env, IDataView dataPipe, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8084805]         E:\TLC_git\machinelearning\test\Microsoft.ML.Tests\ScenariosWithDirectInstantiation\TensorflowTests.cs(438,0): at Microsoft.ML.Scenarios.ScenariosTests.ExecuteTFTransformMNISTLRTrainingTest(Boolean shuffle, Nullable`1 shuffleSeed, Double expectedMicroAccuracy, Double expectedMacroAccruacy)
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8087841]         E:\TLC_git\machinelearning\test\Microsoft.ML.Tests\ScenariosWithDirectInstantiation\TensorflowTests.cs(365,0): at Microsoft.ML.Scenarios.ScenariosTests.TensorFlowTransformMNISTLRTrainingTest()
[10/9/2018 4:18:27 PM Informational] [xUnit.net 00:00:10.8586866]   Finished:    Microsoft.ML.Tests

[Zruty0]

It should not give this error during training, only during scoring, right?
And is it necessary to have the shuffle transform when you are done training?

[TomFinley]

Hi @zeahmed , different objection... Shuffle transform simply cannot be a row to row mapper. The two notions are incompatible. One is based on the idea that it is applicable to situations where an operation can be applied as an operation of one row to another row. One has to do with the permutation of the rows themselves. So, cannot be done, sorry. Perhaps if you were to describe your scenario, we could say what the actual problem is?

[zeahmed]

@Zruty0, I will copy exact exception. It is applied during training. I have a parameter to set shuffle on/off during training. Its a good practice to shuffle the training data before each iteration to reduce variance and skip local minimum.

@TomFinley, yes the title got wrong. It should be "Convert ShuffleTransform to Transformer/Estimator". The idea is to have option for user to shuffle the data on each iteration i.e. whenever I open a new cursor on dataview I should get the data in different ordering. I have done it previously with direct instantiation. Since the design has changed I think it needs conversion or there should be some other way to used it.

Let me know if you guys know the other way to use it?

[TomFinley]

Ah. Thanks @zeahmed ! Then I have another objection, these things should just not be estimators/transformers. I feel pretty strongly about this: See #933 . This has been the cause of a large amount of suffering in the past. It should still exist, but purely an operation over IDataView, not something that becomes part of a data model (which a transform is).

[zeahmed]

So the question is; does this or other similar transforms need conversion? if not are they supposed to work as-is? or this is the work in progress?