Skip to content

Fixing ONNX test #3253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Apr 18, 2019
Merged

Fixing ONNX test #3253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
925bdf0
The test pipeline for consuming an ONNX model would fail due to the
singlis Apr 9, 2019
f278dd1
When adding an ONNX transform to an ML.NET pipeline, an exception would
singlis Apr 10, 2019
973531f
Merge branch 'master' of https://github.com/dotnet/machinelearning
singlis Apr 10, 2019
0c84021
Merge remote-tracking branch 'origin/master' into singlis/onnx
singlis Apr 10, 2019
9c19726
Reverted back to using VectorScoreColumn
singlis Apr 10, 2019
962e8b4
Updating comment to reflect the current onnx behavior
singlis Apr 10, 2019
3d0a318
Merge branch 'master' of https://github.com/dotnet/machinelearning
singlis Apr 16, 2019
8706388
Merge remote-tracking branch 'origin/master' into singlis/onnx
singlis Apr 16, 2019
8c9c177
Merge branch 'master' of https://github.com/dotnet/machinelearning
singlis Apr 17, 2019
ab6055c
Merge remote-tracking branch 'origin/master' into singlis/onnx
singlis Apr 17, 2019
155212b
Adding in a check for variable vector
singlis Apr 17, 2019
e215b33
Moved the check for variable vector
singlis Apr 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/Microsoft.ML.OnnxTransformer/OnnxTransform.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,7 @@ public Mapper(OnnxTransformer parent, DataViewSchema inputSchema) :
var col = inputSchema.GetColumnOrNull(_parent.Inputs[i]);
if (!col.HasValue)
throw Host.ExceptSchemaMismatch( nameof(inputSchema),"input", _parent.Inputs[i]);

_inputColIndices[i] = col.Value.Index;

var type = inputSchema[_inputColIndices[i]].Type;
Expand Down Expand Up @@ -564,7 +565,7 @@ public override SchemaShape GetOutputSchema(SchemaShape inputSchema)
var input = Transformer.Inputs[i];
if (!inputSchema.TryFindColumn(input, out var col))
throw Host.ExceptSchemaMismatch(nameof(inputSchema), "input", input);
if (!(col.Kind == SchemaShape.Column.VectorKind.VariableVector || col.Kind == SchemaShape.Column.VectorKind.Vector))
if (col.Kind == SchemaShape.Column.VectorKind.VariableVector)
throw Host.ExceptSchemaMismatch(nameof(inputSchema), "input", input, "vector", col.GetTypeString());

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if input col.Kind == VariableVector, how is that case handled? If not handled, should throw exception.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so prior to this change, we allowed VariableVector in that it would not throw -- this change now allows for vector and simple types. I would prefer to have a test that uses VariableVector, but I have yet to find something that provides a VariableVector output AND can be saved to ONNX.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jignparm from our conversation, you mentioned that we should not have allowed VariableVector to begin with and should fix as part of this change. I have updated the code to reflect that, but we should have a test that confirms if this works correctly. I have created the following issue for tracking #3375

var inputsInfo = Transformer.Model.ModelInfo.InputsInfo;
Expand Down
20 changes: 12 additions & 8 deletions test/Microsoft.ML.Functional.Tests/ONNX.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
// See the LICENSE file in the project root for more information.

using System.IO;
using Microsoft.ML.Data;
using Microsoft.ML.Functional.Tests.Datasets;
using Microsoft.ML.RunTests;
using Microsoft.ML.TestFramework;
Expand Down Expand Up @@ -48,18 +49,21 @@ public void SaveOnnxModelLoadAndScoreFastTree()
mlContext.Model.ConvertToOnnx(model, data, file);

// Load the model as a transform.
var onnxEstimator = mlContext.Transforms.ApplyOnnxModel(modelPath);
// Note that when saving an ML.NET model as an ONNX model, the column types and column names will
// change. The name changes as ONNX doesn't not allow the same name for an input and output within the ONNX model.
// Therefore names maintained but have a number appended to the end of the name. In this case, Score0 is the output
// of the ONNX model. We are renaming Score0 to Score using Copy Columns.
// ONNX also uses tensors and will return an output of a tensor with the dimension of [1,1] for a single float.
// Therefore the VectorScoreColumn class (which contains a float [] field called Score) is used for the return
// type on the Prediction engine.
// See #2980 and #2981 for more information.
var onnxEstimator = mlContext.Transforms.ApplyOnnxModel(modelPath)
.Append(mlContext.Transforms.CopyColumns("Score", "Score0"));
var onnxModel = onnxEstimator.Fit(data);

// TODO #2980: ONNX outputs don't match the outputs of the model, so we must hand-correct this for now.
// TODO #2981: ONNX models cannot be fit as part of a pipeline, so we must use a workaround like this.
var onnxWorkaroundPipeline = onnxModel.Append(
mlContext.Transforms.CopyColumns("Score", "Score0").Fit(onnxModel.Transform(data)));

// Create prediction engine and test predictions.
var originalPredictionEngine = mlContext.Model.CreatePredictionEngine<HousingRegression, ScoreColumn>(model);
// TODO #2982: ONNX produces vector types and not the original output type.
var onnxPredictionEngine = mlContext.Model.CreatePredictionEngine<HousingRegression, VectorScoreColumn>(onnxWorkaroundPipeline);
var onnxPredictionEngine = mlContext.Model.CreatePredictionEngine<HousingRegression, VectorScoreColumn>(onnxModel);

// Take a handful of examples out of the dataset and compute predictions.
var dataEnumerator = mlContext.Data.CreateEnumerable<HousingRegression>(mlContext.Data.TakeRows(data, 5), false);
Expand Down