Help to consume simple linear integration TensorFlow model needed

[evgeny1984]

System information

• OS version/distro: Windows 10.0.19041 64 Bit
• .NET Version (eg., dotnet --info): Version: 5.0.100

Issue

Please help me by providing the solution that I can solve this problem.

• What did you do?
  I created a simple Python script in Jupyter Notebook for the data set consisting of 8 rows. Successfuly trained and evaluated the model and then exported it. The model can be loaded but I cannot configure the pipeline to make prediction.
• What happened?
  Now I try to consume the model with ML.NET Console dotnet core 3.1 application.
• What did you expect?
  Cannot consume the model, because not able to create the proper pipeline.
  Cannot access the properties in GetModelShema as in the microsoft tutorial, because they dont exist. (how can I define them in my model ("Features"m "Prediction/Softmax")) that they are labeled and recognized?

Source code / logs

project&jupyter_notebook.zip

[antoniovs1029]

Hi,

So I've ran the C# code you have, and I don't think there's any problem with the .GetModelSchema() method, which is meant to return the nodes found on your TF model. The schema returned by it is as follows:

Where the saved_model.pb you provided has the same nodes:

After quickly reading your jupyter notebok, it seems to me you're simply training a simple regression model that maps 1 input number to one output number, so I'm surprised by all the nodes found on your .pb model. More importantly, several of the nodes seem to hold information about training (like all the "Adam" nodes for learning_rate, decay, etc...).

So, ML.NET uses Tensorflow.NET to load and run TF models. And Tensorflow.NET requires the TF models to be frozen (link to docs) so I would suggest you trying to freeze the model before saving it. Here's a tutorial on how to freeze keras models:
https://www.dlology.com/blog/how-to-convert-trained-keras-model-to-tensorflow-and-make-prediction/

I'd suspect this is necessary to get a meaningful graph that doesn't have information about training, only for inferencing.

When you have a clearer graph, then you'd be able to identify which one is your output variable and use that on ML.NET.

BTW, in ML.NET you don't need the tensorflow model to have a "Features" or "Prediction/Softmax" node. I'd think you're taking those names from ModelBuilder or one of our samples on tensorflow, but you don't need one. Once you've done the steps above I hope you're able to identify the desired output variable, and its name, and then use that name on ML.NET.

Please, let us know if this fixes your problem.

[evgeny1984]

Hi @antoniovs1029,

thank you for the reply, I will try to implement it and will write you back this week.

With kind regards Evgeny

[evgeny1984]

Hi @antoniovs1029,

it seems, that the tutorial you provided is outdated, cause I cannot execute all steps while getting the error.

`AttributeErrorTraceback (most recent call last)
in
33
34
---> 35 frozen_graph = freeze_session(K.get_session(),
36 output_names=[out.op.name for out in model.outputs])

AttributeError: module 'keras.backend' has no attribute 'get_session'`

The version they are used is under TF 2.

[evgeny1984]

Ok, I have created 2 files: pb and pbtxt by using this function:

freeze_graph.zip

Here is the output:

Then I put these files into the project folder. When I try to load the tensorflow model in VS 2019 I get the exception:

"Could not find SavedModel .pb or .pbtxt at supplied export directory path: C:\Users<User>\Documents\machinelearning\TensorFlow.Net.Demo\bin\Debug\netcoreapp3.1\Data\regression".

The graph can be shown in www.netron.app and looks better as before.

All links you have provided cannot be applied since I use the TF v.2

Please help me to solve this problem.

Thank you for your help in advance.

[antoniovs1029]

The model seems much clearer now, so I think that not freezing it before was the problem.

"Could not find SavedModel .pb or .pbtxt at supplied export directory path: C:\Users\Evgeny Volynsky\Documents\machinelearning\TensorFlow.Net.Demo\bin\Debug\netcoreapp3.1\Data\regression".

I don't think you need to load both the .pb and .pbtxt, you simply need to load the .pb file the same way you were loading it before, but this time the Schema should be much simpler. Are you loading it the same way as before? Can you provide a full stack trace? since I don't know where that error message could be coming from.

[evgeny1984]

Hi @antoniovs1029,
yes it looks more clear now 💯

I didn't change anything in code I provided previously to you. The model path just directs to the folder, where the both new frozen_model files are located. Also they are set to "copy if newer" option, so that they are definitly under the provided folder.

Here is the StackTrace of the TensorFlowException:
at Tensorflow.Status.Check(Boolean throwException)
at Tensorflow.Session.LoadFromSavedModel(String path)
at Microsoft.ML.TensorFlow.TensorFlowUtils.LoadTFSession(IHostEnvironment env, String exportDirSavedModel)
at Microsoft.ML.TensorFlow.TensorFlowUtils.GetSession(IHostEnvironment env, String modelPath, Boolean metaGraph)
at Microsoft.ML.TensorFlow.TensorFlowUtils.LoadTensorFlowModel(IHostEnvironment env, String modelPath)
at Microsoft.ML.TensorflowCatalog.LoadTensorFlowModel(ModelOperationsCatalog catalog, String modelLocation)
at TensorFlow.Net.Demo.Program.LoadTensorFlowModel(MLContext mlContext, String modelPath) in C:\Users<User>\Documents\machinelearning\TensorFlow.Net.Demo\Program.cs:line 50

UPDATE:

I can load now the model, in the previous version I just could provide the folder path, now it must be path + model name. Thzen it works as expected.

I will try to configure the pipeline and will report the results or prolems. :D

[evgeny1984]

Now I am stucked at the pipeline. Can you help me to construct the pipeline please.
Here is the updated project.
TensorFlow.Net.Demo.zip

Thank you in advance.

[antoniovs1029]

To correctly run your tensorflow model you need to do 2 main things.

1. Change your pipeline to the pipeline below. Since you're not loading a "Features" column, the Concatenate transform you had isn't necessary and it would cause problems since you didn't have a Features column. Since the output of your tensorflow model is called "Identity" add a CopyColumn transform to rename it to "DeliveryDelay" (which is the name you've defined as your output on your custom class.

     var pipeline = tensorFlowModel.ScoreTensorFlowModel("Identity", "NumberOfStops")
                .Append(mlContext.Transforms.CopyColumns("DeliveryDelay", "Identity"));

2. Change the input and output types of your tensorflow models to vector types. In ML.NET the input and outputs of a tensorflow model need to be VectorType. Since your input and output is a single number, you can simply add a vector type of dimensions "(1)" (1-D of length 1). Since you're loading data with TextLoader, using LoadColumn(2,2) is also necessary to make it load a vector that uses the columns 2 to 2 (i.e. a vector of size 1, with only the element found on column 2). Making this change also means you need to change the type of the member variable from float to float[].

    public class DeliveryDelay
    {
        [LoadColumn(0)]
        public string Month;

        [LoadColumn(1)]
        public float ProductId;

        [LoadColumn(2, 2), VectorType(1)]
        public float[] NumberOfStops;

        [LoadColumn(3)]
        public float Delay;
    }

In DeliveryDelayPrediction class:

        [VectorType(1)]
        public float[] DeliveryDelay;

Bellow I attach the full Program.cs I used, since I might have changed other things. Notice that I disabled the call to the Evaluate method, and only call the TestSinglePrediction() method. This is because the evaluate method expects the output column to be a Single (i.e. a real number) whereas, as mentioned, your tensorflow model outputs a vector of size 1. If you want to use the evaluate method then you'd need to extract the element inside the vector and put it inside another column of type Single. But doing this is a little more involved, and since I'm not sure it's necessary for you I didn't do it. In case you'd need to do it, please refer to this comment (or see this other comment for a more advanced scenario), the idea is to use a CustomMappingTransformer to extract the element from inside the tensorflow output array.

**Click to see Code**

using System;
using System.IO;
using System.Collections.Generic;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Transforms;
using TensorFlow.Net.Demo.Mappings;
using Tensorflow;

namespace TensorFlow.Net.Demo
{
public class Program
{
public const int FeatureLength = 1;
private static readonly string _modelPath = Path.Combine(Environment.CurrentDirectory, "Data/regression/frozen_graph.pb");
static readonly string _trainDataPath = Path.Combine(Environment.CurrentDirectory, "Data/regression", "dataset-train.csv");
static readonly string _testDataPath = @"C:\Users\anvelazq\Downloads\TensorFlow.Net.Demo\TensorFlow.Net.Demo\Data\regression\dataset-test.csv";

    static void Main(string[] args)
    {
        // Initialize the context
        MLContext mlContext = new MLContext();

        // Load the model
        var tensorModel = LoadTensorFlowModel(mlContext, _modelPath);
        // Construct a pipeline
        var model = CreateMLPipeline(mlContext, tensorModel);

        // Evaluate the model
        //Evaluate(mlContext, model);

        var toPredict = new DeliveryDelay()
        {
            NumberOfStops = new float[] { 8f },
            Delay = 20
        };

        // Make the prediction
        TestSinglePrediction(mlContext, model, toPredict);

        Console.ReadKey();
    }

    public static TensorFlowModel LoadTensorFlowModel(MLContext mlContext, string modelPath)
    {
        TensorFlowModel tensorFlowModel = null;

        try
        {
            tensorFlowModel = mlContext.Model.LoadTensorFlowModel(modelPath);
        }
        catch (TensorflowException ex)
        {

            throw ex;
        }
        DataViewSchema schema = tensorFlowModel.GetModelSchema();
        Console.WriteLine(" =============== TensorFlow Model Schema =============== ");
        var featuresType = (VectorDataViewType)schema["NumberOfStops"].Type;
        Console.WriteLine($"Name: NumberOfStops, Type: {featuresType.ItemType.RawType}, Size: ({featuresType.Dimensions[0]})");
        var predictionType = (VectorDataViewType)schema["Identity"].Type;
        Console.WriteLine($"Name: Identity, Type: {predictionType.ItemType.RawType}, Size: ({predictionType.Dimensions[0]})");

        return tensorFlowModel;
    }

    public static ITransformer CreateMLPipeline(MLContext mlContext, TensorFlowModel tensorFlowModel)
    {
        // Loads the data
        IDataView dataView = mlContext.Data.LoadFromEnumerable<DeliveryDelay>(new List<DeliveryDelay>());

        // Extracts and transforms the data
        var pipeline = tensorFlowModel.ScoreTensorFlowModel("Identity", "NumberOfStops")
            .Append(mlContext.Transforms.CopyColumns("DeliveryDelay", "Identity"));

        // Trains the model
        var model = pipeline.Fit(dataView);

        return model;
    }

    private static void Evaluate(MLContext mlContext, ITransformer model)
    {
        // Loads the test dataset
        IDataView dataView = mlContext.Data.LoadFromTextFile<DeliveryDelay>(_testDataPath, hasHeader: true, separatorChar: ',');

        // Creates the regression evaluator
        var predictions = model.Transform(dataView);

        // Evaluates the model and creates metrics
        var metrics = mlContext.Regression.Evaluate(predictions, "Delay", "DeliveryDelay");

        // Displays the metrics
        Console.WriteLine();
        Console.WriteLine($"*************************************************");
        Console.WriteLine($"*       Model quality metrics evaluation         ");
        Console.WriteLine($"*------------------------------------------------");
        Console.WriteLine($"*       RSquared Score:      {metrics.RSquared:0.##}");
        Console.WriteLine($"*       Root Mean Squared Error:      {metrics.RootMeanSquaredError:#.##}");
    }

    private static void TestSinglePrediction(MLContext mlContext, ITransformer model, DeliveryDelay toPredict)
    {
        // Creates a single comment of test data
        var predictionFunction = mlContext.Model.CreatePredictionEngine<DeliveryDelay, DeliveryDelayPrediction>(model);

        // Predicts subscribers gained based on test data
        var prediction = predictionFunction.Predict(toPredict);

        // Displays the predicted results
        Console.WriteLine($"**********************************************************************");
        Console.WriteLine($"Predicted delay: {prediction.DeliveryDelay[0]:0.####}, actual delay: 20");
        Console.WriteLine($"**********************************************************************");
    }
}

}

[evgeny1984]

Hi @antoniovs1029,

thank you for the provided solution. The evaluation function is also important, since I want to demonstrate that the same accuracy is reached as in the Jupyter notebook by using the same model.

If it is possible, could you provide the snippet for the evaluation function too.

Thank you in advance.

[antoniovs1029]

Hi.

Bellow is the code to use the Evaluate method with your tensorflow model. By the way, it seems you're doing this for a demo, I'm just curious: where are you presenting this demo? Thanks.

So, to extract the value of the output array of your tensorflow model, simply define the following to be used by the CustomMappingTransformer:

        public class ExtractInput
        {
            [ColumnName("DeliveryDelay")]
            [VectorType(1)]
            public float[] array;
        }

        public class ExtractOutput
        {
            [ColumnName("output")]
            public float extractedValue;
        }

        public static void ExtractMapping(ExtractInput input, ExtractOutput output)
        {
            // The input array is length 1, extract its only value
            output.extractedValue = input.array[0];
        }

and add the CustomMappingTransformer to your pipeline:

    .Append(mlContext.Transforms.CustomMapping<ExtractInput, ExtractOutput>(ExtractMapping, contractName: null));

Then on the .Evaluate call provide the column names. Notice that the score column is named "output" because that's how I declared the ColumnName attribute on the ExtractOutput class above.

 var metrics = mlContext.Regression.Evaluate(predictions, labelColumnName:"Delay", scoreColumnName:"output");

The code above could have been written in multiple ways, please refer to the CustomMapping samples and APIs to know how to use it in the link here. Thanks.

Click to see full code

using System;
using System.IO;
using System.Collections.Generic;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Transforms;
using TensorFlow.Net.Demo.Mappings;
using Tensorflow;

namespace TensorFlow.Net.Demo
{
public class Program
{
public const int FeatureLength = 1;
private static readonly string _modelPath = Path.Combine(Environment.CurrentDirectory, "Data/regression/frozen_graph.pb");
static readonly string _trainDataPath = Path.Combine(Environment.CurrentDirectory, "Data/regression", "dataset-train.csv");
static readonly string _testDataPath = @"C:\Users\anvelazq\Downloads\TensorFlow.Net.Demo\TensorFlow.Net.Demo\Data\regression\dataset-test.csv";

    static void Main(string[] args)
    {
        // Initialize the context
        MLContext mlContext = new MLContext();

        // Load the model
        var tensorModel = LoadTensorFlowModel(mlContext, _modelPath);
        // Construct a pipeline
        var model = CreateMLPipeline(mlContext, tensorModel);

        // Evaluate the model
        Evaluate(mlContext, model);

        var toPredict = new DeliveryDelay()
        {
            NumberOfStops = new float[] { 8f },
            Delay = 20
        };

        // Make the prediction
        TestSinglePrediction(mlContext, model, toPredict);

        Console.ReadKey();
    }

    public static TensorFlowModel LoadTensorFlowModel(MLContext mlContext, string modelPath)
    {
        TensorFlowModel tensorFlowModel = null;

        try
        {
            tensorFlowModel = mlContext.Model.LoadTensorFlowModel(modelPath);
        }
        catch (TensorflowException ex)
        {

            throw ex;
        }
        DataViewSchema schema = tensorFlowModel.GetModelSchema();
        Console.WriteLine(" =============== TensorFlow Model Schema =============== ");
        var featuresType = (VectorDataViewType)schema["NumberOfStops"].Type;
        Console.WriteLine($"Name: NumberOfStops, Type: {featuresType.ItemType.RawType}, Size: ({featuresType.Dimensions[0]})");
        var predictionType = (VectorDataViewType)schema["Identity"].Type;
        Console.WriteLine($"Name: Identity, Type: {predictionType.ItemType.RawType}, Size: ({predictionType.Dimensions[0]})");

        return tensorFlowModel;
    }

    public static ITransformer CreateMLPipeline(MLContext mlContext, TensorFlowModel tensorFlowModel)
    {
        // Loads the data
        IDataView dataView = mlContext.Data.LoadFromEnumerable<DeliveryDelay>(new List<DeliveryDelay>());

        // Extracts and transforms the data
        var pipeline = tensorFlowModel.ScoreTensorFlowModel("Identity", "NumberOfStops")
            .Append(mlContext.Transforms.CopyColumns("DeliveryDelay", "Identity"))
            .Append(mlContext.Transforms.CustomMapping<ExtractInput, ExtractOutput>(ExtractMapping, contractName: null));

        // Trains the model
        var model = pipeline.Fit(dataView);

        return model;
    }

    private static void Evaluate(MLContext mlContext, ITransformer model)
    {
        // Loads the test dataset
        IDataView dataView = mlContext.Data.LoadFromTextFile<DeliveryDelay>(_testDataPath, hasHeader: true, separatorChar: ';');

        // Creates the regression evaluator
        var predictions = model.Transform(dataView);
        var p = predictions.Preview();

        // Evaluates the model and creates metrics
        var metrics = mlContext.Regression.Evaluate(predictions, labelColumnName:"Delay", scoreColumnName:"output");

        // Displays the metrics
        Console.WriteLine();
        Console.WriteLine($"*************************************************");
        Console.WriteLine($"*       Model quality metrics evaluation         ");
        Console.WriteLine($"*------------------------------------------------");
        Console.WriteLine($"*       RSquared Score:      {metrics.RSquared:0.##}");
        Console.WriteLine($"*       Root Mean Squared Error:      {metrics.RootMeanSquaredError:#.##}");
    }

    private static void TestSinglePrediction(MLContext mlContext, ITransformer model, DeliveryDelay toPredict)
    {
        // Creates a single comment of test data
        var predictionFunction = mlContext.Model.CreatePredictionEngine<DeliveryDelay, DeliveryDelayPrediction>(model);

        // Predicts subscribers gained based on test data
        var prediction = predictionFunction.Predict(toPredict);

        // Displays the predicted results
        Console.WriteLine($"**********************************************************************");
        Console.WriteLine($"Predicted delay: {prediction.DeliveryDelay[0]:0.####}, actual delay: 20");
        Console.WriteLine($"**********************************************************************");
    }

    public class ExtractInput
    {
        [ColumnName("DeliveryDelay")]
        [VectorType(1)]
        public float[] array;
    }

    public class ExtractOutput
    {
        [ColumnName("output")]
        public float extractedValue;
    }

    public static void ExtractMapping(ExtractInput input, ExtractOutput output)
    {
        // The input array is length 1, extract its only value
        output.extractedValue = input.array[0];
    }
}

}

[evgeny1984]

Hi @antoniovs1029 ,

thank you a lot for your help.

I am a little bit surprised, that is it not trivial to implement the so simple use case. Can you provide any tips how and where I can find all the most important stuff like documentation to make all these configuration steps without help. I find a few of examples but not a real documentation.

P.S. I have next monday the demonstration of ML basics in my work. We have a very focussed dotnet landscape with microservice architecture, so my task is to evaluate different constellations for the ML tools and frameworks.

My idea is to combine the power of the tensorflow with the ML.NET, so that we can fit this apporach to the existing it landscape. I am very excited, how quick I got help from you but have some thoughts about the complexity of the implementation of this approach, since it seems not a very trivial task.

Have a nice weekend.

[evgeny1984]

Sorry for the following question, I got the wrong evaluation results somehow:

The RSquaredScore should be in {-infinity;1} but is -8 (-infinity)! The model accuracy but is by 0.9999...
So something is wrong configured.

UPDATE:

The separator char was wrong by reading the test data, so the predictions got the wrong values. Now it works as expected, thanky ou one more time!

[antoniovs1029]

Great to hear it's all working now, will close this issue then.