Pytorch_Snippets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "#### PyTorch has emerged as a major contender in the race to be the king of deep learning frameworks.\n",
    "\n",
    "In this notebook I will go over some regular snippets and techniques of it."
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Compute basic gradients from the sample tensors using PyTorch\n",
    "\n",
    "#### First some basics of Pytorch here\n",
    "\n",
    "**Autograd**: This class is an engine to calculate derivatives (Jacobian-vector product to be more precise). It records a graph of all the operations performed on a gradient enabled tensor and creates an acyclic graph called the dynamic computational graph. The leaves of this graph are input tensors and the roots are output tensors. Gradients are calculated by tracing the graph from the root to the leaf and multiplying every gradient in the way using the chain rule.\n",
    "\n",
    "A Variable class wraps a tensor. You can access this tensor by calling `.data` attribute of a Variable.\n",
    "\n",
    "The Variable also stores the gradient of a scalar quantity (say, loss) with respect to the parameter it holds. This gradient can be accessed by calling the `.grad` attribute. This is basically the gradient computed up to this particular node, and the gradient of the every subsequent node, can be computed by multiplying the edge weight with the gradient computed at the node just before it.\n",
    "\n",
    "The third attribute a Variable holds is a grad_fn, a Function object which created the variable.\n",
    "\n",
    "**Variable**: The Variable, just like a Tensor is a class that is used to hold data. It differs, however, in the way it’s meant to be used. Variables are specifically tailored to hold values which change during training of a neural network, i.e. the learnable paramaters of our network. Tensors on the other hand are used to store values that are not to be learned. For example, a Tensor maybe used to store the values of the loss generated by each example.\n",
    "\n",
    "Every **variable** object has several members one of them is **grad**:\n",
    "\n",
    "**grad**: grad holds the value of gradient. If requires_grad is False it will hold a None value. Even if requires_grad is True, it will hold a None value unless .backward() function is called from some other node. For example, if you call out.backward() for some variable out that involved x in its calculations then x.grad will hold ∂out/∂x.\n",
    "\n",
    "**Backward() function**\n",
    "Backward is the function which actually calculates the gradient by passing it’s argument (1x1 unit tensor by default) through the backward graph all the way up to every leaf node traceable from the calling root tensor. The calculated gradients are then stored in .grad of every leaf node. Remember, the backward graph is already made dynamically during the forward pass. Backward function only calculates the gradient using the already made graph and stores them in leaf nodes."
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch as tch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "from sklearn.datasets import make_blobs\n",
    "from matplotlib import pyplot\n",
    "\n",
    "def forward(x):\n",
    "    return x * w\n",
    "\n",
    "w = Variable(torch.Tensor([1.0]), requires_grad=True)\n",
    "# . On setting .requires_grad = True they start forming a backward graph\n",
    "# that tracks every operation applied on them to calculate the gradients\n",
    "# using something called a dynamic computation graph (DCG)\n",
    "# When you finish your computation you can call .backward() and have\n",
    "# all the gradients computed automatically. The gradient for this tensor\n",
    "# will be accumulated into .grad attribute.\n",
    "\n",
    "# Now create an array of data.\n",
    "# By PyTorch’s design, gradients can only be calculated\n",
    "# for floating point tensors which is why I’ve created a float type\n",
    "# array before making it a gradient enabled PyTorch tensor\n",
    "x_data = [11.0, 22.0, 33.0]\n",
    "y_data = [21.0, 14.0, 64.0]\n",
    "\n",
    "def loss_function(x, y):\n",
    "    y_pred = forward(x)\n",
    "    return (y_pred - y) * (y_pred - y)\n",
    "\n",
    "\n",
    "# Now running the training loop\n",
    "for epoch in range(10):\n",
    "    for x_val, y_val in zip(x_data, y_data):\n",
    "        l = loss_function(x_val, y_val)\n",
    "        l.backward()\n",
    "        print(\"\\tgrad: \", x_val, y_val, w.grad.data[0])\n",
    "        w.data = w.data - 0.01 * w.grad\n",
    "\n",
    "        # Manually set the gradient to zero after updating weights\n",
    "        w.grad.data.zero_()\n",
    "\n",
    "        print('progress: ', epoch, l.data[0])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\tgrad:  11.0 21.0 tensor(-220.)\n",
      "progress:  0 tensor(100.)\n",
      "\tgrad:  22.0 14.0 tensor(2481.6001)\n",
      "progress:  0 tensor(3180.9602)\n",
      "\tgrad:  33.0 64.0 tensor(-51303.6484)\n",
      "progress:  0 tensor(604238.8125)\n",
      "\tgrad:  11.0 21.0 tensor(118461.7578)\n",
      "progress:  1 tensor(28994192.)\n",
      "\tgrad:  22.0 14.0 tensor(-671630.6875)\n",
      "progress:  1 tensor(2.3300e+08)\n",
      "\tgrad:  33.0 64.0 tensor(13114108.)\n",
      "progress:  1 tensor(3.9481e+10)\n",
      "\tgrad:  11.0 21.0 tensor(-30279010.)\n",
      "progress:  2 tensor(1.8943e+12)\n",
      "\tgrad:  22.0 14.0 tensor(1.7199e+08)\n",
      "progress:  2 tensor(1.5279e+13)\n",
      "\tgrad:  33.0 64.0 tensor(-3.3589e+09)\n",
      "progress:  2 tensor(2.5900e+15)\n",
      "\tgrad:  11.0 21.0 tensor(7.7553e+09)\n",
      "progress:  3 tensor(1.2427e+17)\n",
      "\tgrad:  22.0 14.0 tensor(-4.4050e+10)\n",
      "progress:  3 tensor(1.0023e+18)\n",
      "\tgrad:  33.0 64.0 tensor(8.6030e+11)\n",
      "progress:  3 tensor(1.6991e+20)\n",
      "\tgrad:  11.0 21.0 tensor(-1.9863e+12)\n",
      "progress:  4 tensor(8.1519e+21)\n",
      "\tgrad:  22.0 14.0 tensor(1.1282e+13)\n",
      "progress:  4 tensor(6.5750e+22)\n",
      "\tgrad:  33.0 64.0 tensor(-2.2034e+14)\n",
      "progress:  4 tensor(1.1146e+25)\n",
      "\tgrad:  11.0 21.0 tensor(5.0875e+14)\n",
      "progress:  5 tensor(5.3477e+26)\n",
      "\tgrad:  22.0 14.0 tensor(-2.8897e+15)\n",
      "progress:  5 tensor(4.3132e+27)\n",
      "\tgrad:  33.0 64.0 tensor(5.6436e+16)\n",
      "progress:  5 tensor(7.3118e+29)\n",
      "\tgrad:  11.0 21.0 tensor(-1.3030e+17)\n",
      "progress:  6 tensor(3.5081e+31)\n",
      "\tgrad:  22.0 14.0 tensor(7.4013e+17)\n",
      "progress:  6 tensor(2.8295e+32)\n",
      "\tgrad:  33.0 64.0 tensor(-1.4455e+19)\n",
      "progress:  6 tensor(4.7966e+34)\n",
      "\tgrad:  11.0 21.0 tensor(3.3374e+19)\n",
      "progress:  7 tensor(2.3013e+36)\n",
      "\tgrad:  22.0 14.0 tensor(-1.8957e+20)\n",
      "progress:  7 tensor(1.8562e+37)\n",
      "\tgrad:  33.0 64.0 tensor(3.7022e+21)\n",
      "progress:  7 tensor(inf)\n",
      "\tgrad:  11.0 21.0 tensor(-8.5480e+21)\n",
      "progress:  8 tensor(inf)\n",
      "\tgrad:  22.0 14.0 tensor(4.8553e+22)\n",
      "progress:  8 tensor(inf)\n",
      "\tgrad:  33.0 64.0 tensor(-9.4824e+23)\n",
      "progress:  8 tensor(inf)\n",
      "\tgrad:  11.0 21.0 tensor(2.1894e+24)\n",
      "progress:  9 tensor(inf)\n",
      "\tgrad:  22.0 14.0 tensor(-1.2436e+25)\n",
      "progress:  9 tensor(inf)\n",
      "\tgrad:  33.0 64.0 tensor(2.4287e+26)\n",
      "progress:  9 tensor(inf)\n"
     ]
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Weight initialization is an important task in training a neural network,\n",
    "whether its a convolutional neural network\n",
    "(CNN), a deep neural network (DNN), and a recurrent neural network\n",
    "(RNN). Lets some examples of initializing the weights.\n",
    "\n",
    "\n",
    "Weight initialization can be done by using various methods, including\n",
    "random weight initialization.\n",
    "Weight initialization based on a distribution\n",
    "is done using\n",
    "- Uniform distribution,\n",
    "- Bernoulli distribution,\n",
    "- Multinomial distribution, and normal distribution.\n",
    "\n",
    "To execute a neural network, a set of initial weights needs to be passed to\n",
    "the backpropagation layer to compute the loss function (and hence, the\n",
    "accuracy can be calculated). The selection of a method depends on the\n",
    "data type, the task, and the optimization required for the model."
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bernoulli Distribution is a random experiment that has only two outcomes (usually called a “Success” or a “Failure”). It is best used when we have two outcomes of a given event. Its considered as the discrete\n",
    "probability distribution, which has two possible outcomes. If the event happens, then the value is 1, and if the event does not happen, then the value is 0.\n",
    "\n",
    "For discrete probability distribution, we calculate probability mass\n",
    "function instead of probability density function. The probability mass\n",
    "function looks like the following formula.\n",
    "\n",
    "![](https://i.imgur.com/bz2dWtc.png)\n",
    "\n",
    "From the Bernoulli distribution, we create sample tensors by considering the uniform distribution of size 4 and 4 in a matrix format, as follows.\n",
    "\n",
    " Specifically, `torch.bernoulli()` samples from the distribution and returns a binary value (i.e. either 0 or 1). Here, it returns 1 with probability p and return 0 with probability 1-p.\n",
    "\n",
    "```python\n",
    "torch.bernoulli(input, *, generator=None, out=None)\n",
    "```\n",
    "It draws binary random numbers (0 or 1) from a Bernoulli distribution.\n",
    "\n",
    "Syntax\n",
    "\n",
    "```python\n",
    "torch.bernoulli(input, *, generator=None, out=None) → Tensor\n",
    "```\n",
    "\n",
    "Parameters :\n",
    "\n",
    "input (Tensor) – the input tensor of probability values for the Bernoulli distribution\n",
    "\n",
    "generator (torch.Generator, optional) – a pseudorandom number generator for sampling\n",
    "\n",
    "out (Tensor, optional) – out tensor only has values 0 or 1 and is of the same shape as input.\n",
    "\n",
    "The input tensor should be a tensor containing probabilities to be used for drawing the binary random number. Hence, all values in input have to be in the range:\n",
    "\n",
    "0 <= input_i <=1\n"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "torch.bernoulli(torch.Tensor(4, 4).uniform_(0, 1))\n"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 1.],\n",
       "        [1., 0., 0., 1.],\n",
       "        [0., 1., 0., 1.],\n",
       "        [1., 1., 1., 0.]])"
      ]
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Generation of sample random values from a multinomial distribution\n",
    "\n",
    "Note the syntax of multinomial function from official doc\n",
    "\n",
    "```python\n",
    "torch.multinomial(input, num_samples, replacement=False, *, generator=None, out=None) → LongTensor\n",
    "```\n",
    "Returns a tensor where each row contains num_samples indices sampled from the multinomial probability distribution located in the corresponding row of tensor input.\n",
    "\n"
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "sample_tensor = torch.Tensor([10, 10, 13, 10, 34,45,65,67,87,89,87,34])\n",
    "torch.multinomial(torch.tensor([10., 10., 13., 10., 34., 45., 65., 67., 87., 89., 87., 34.]), 3)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([8, 7, 9])"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Sampling from multinomial distribution with a replacement returns the tensors’ index values."
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "torch.multinomial(torch.tensor([10., 10., 13., 10., 34., 45., 65., 67., 87., 89., 87., 34.]), 5, replacement=True)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([5, 5, 7, 7, 7])"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "And now, the weight initialization from the normal distribution, which is also a method\n",
    "that is used in fitting a neural network, fitting a deep neural network, and\n",
    "CNN and RNN. Let’s have a look at the process of creating a set of random\n",
    "weights generated from a normal distribution.\n",
    "\n",
    "Syntax\n",
    "\n",
    "```python\n",
    "torch.normal(mean, std, *, generator=None, out=None) → Tensor\n",
    "```\n",
    "Returns a tensor of random numbers drawn from separate normal distributions whose mean and standard deviation are given.\n",
    "\n",
    "The mean is a tensor with the mean of each output element’s normal distribution\n",
    "\n",
    "The std is a tensor with the standard deviation of each output element’s normal distribution\n",
    "\n",
    "The shapes of mean and std don’t need to match, but the total number of elements in each tensor need to be the same."
   ],
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "torch.normal(mean=torch.arange(1., 11), std=torch.arange(1, 0, -0.1))"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([0.3873, 1.8538, 2.5595, 3.3766, 5.2688, 7.0845, 7.6231, 8.0542, 8.7442,\n",
       "        9.9076])"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "torch.normal(mean=0.5, std=torch.arange(1.,6.))"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([ 3.2883, -0.2210,  1.6501,  1.4141, -4.3036])"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "torch.normal(mean=0.5, std=torch.arange(0.2, 0.6))\n"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([0.5485])"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Variable in PyTorch and its defined? What is a random variable in PyTorch?\n",
    "\n",
    "In PyTorch, the algorithms are represented as a computational graph.\n",
    "\n",
    "A variable is considered as a representation around the tensor object,\n",
    "corresponding gradients (slope of the function), and a reference to the function from where it was\n",
    "created. \n",
    "\n",
    "The slope of the function can be computed by the derivative of the\n",
    "function with respect to the parameters that are present in the function.\n",
    "\n",
    "Basically, a PyTorch variable is a node in a computational graph, which\n",
    "stores data and gradients. When training a neural network model, after\n",
    "each iteration, we need to compute the gradient of the loss function with\n",
    "respect to the parameters of the model, such as weights and biases. After\n",
    "that, we usually update the weights using the gradient descent algorithm.\n",
    "\n",
    "Below Figure explains how the linear regression equation is deployed under\n",
    "the hood using a neural network model in the PyTorch framework.\n",
    "In a computational graph structure, the sequencing and ordering\n",
    "of tasks is very important. The one-dimensional tensors are X, Y, W,\n",
    "and alpha. The direction of the arrows change when we\n",
    "implement backpropagation to update the weights to match with Y, so that\n",
    "the error or loss function between Y and predicted Y can be minimized.\n",
    "\n",
    "\n",
    "![Imgur](https://imgur.com/6JOtOGb.png)\n",
    "\n",
    "#### Lets see and example\n",
    "\n",
    "An example of how a variable is used to create a computational graph is\n",
    "displayed in the following script. There are three variable objects around\n",
    "tensors— x1, x2, and x3—with random points generated from a = 12 and\n",
    "b = 23. The graph computation involves only multiplication and addition,\n",
    "and the final result with the gradient is shown.\n",
    "\n",
    "The partial derivative of the loss function with respect to the weights\n",
    "and biases in a neural network model is achieved in PyTorch using the\n",
    "Autograd module. Variables are specifically designed to hold the changed\n",
    "values while running a backpropagation in a neural network model when\n",
    "the parameters of the model change. The variable type is just a wrapper\n",
    "around the tensor. It has three properties: data, grad, and function.\n",
    "\n",
    "\n",
    "\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "from torch.autograd import Variable\n",
    "Variable(torch.ones(2,2), requires_grad=True)\n"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1.],\n",
       "        [1., 1.]], requires_grad=True)"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "a, b = 12, 23\n",
    "x1 = Variable(torch.randn(a, b), requires_grad=True )\n",
    "x2 = Variable(torch.randn(a,b), requires_grad=True)\n",
    "x3 = Variable(torch.randn(a,b), requires_grad=True)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "c = x1 * x2\n",
    "d = a + x3\n",
    "e = torch.sum(d)\n",
    "\n",
    "e.backward()\n",
    "\n",
    "print(e)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor(3296.0845, grad_fn=<SumBackward0>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "x1.data"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[ 8.1828e-01, -1.6748e-01,  3.3572e-01,  1.3247e+00, -1.1390e-01,\n",
       "         -1.1533e+00,  1.8396e-02,  1.1846e+00, -5.4557e-01,  2.6307e-01,\n",
       "         -7.5235e-01, -1.0258e+00,  1.7264e+00,  3.1132e-01,  1.4500e+00,\n",
       "          1.0881e+00,  6.6679e-01, -7.3055e-01,  1.0475e+00, -1.4899e+00,\n",
       "          1.7605e+00,  3.0716e+00,  1.1979e+00],\n",
       "        [-7.5272e-01, -1.6185e-01,  3.1758e-01, -1.1183e+00, -8.4021e-02,\n",
       "         -1.5449e+00,  4.1204e-01,  4.0797e-01, -9.3155e-01,  3.6896e-01,\n",
       "          6.0915e-01, -2.7323e-01,  1.5830e+00,  3.6732e-01, -4.4652e-02,\n",
       "         -4.2154e-01,  5.9620e-01,  3.7424e-01,  5.9423e-01,  9.7377e-01,\n",
       "         -2.4661e+00,  1.5546e+00,  1.2334e+00],\n",
       "        [ 8.0186e-02,  2.4171e+00, -1.8963e+00, -6.1453e-01,  1.7587e-02,\n",
       "         -2.2381e-01,  7.1807e-01,  4.1588e-01, -9.7389e-02, -7.9841e-02,\n",
       "         -2.5912e-02, -7.5603e-01, -1.2318e+00, -9.9687e-01, -6.7872e-01,\n",
       "         -1.1399e+00,  1.3165e+00,  7.0379e-01,  4.6876e-01, -3.2940e-01,\n",
       "          1.8421e+00, -9.1479e-01,  9.8207e-01],\n",
       "        [-1.4315e+00, -1.1209e+00, -1.1372e+00,  1.0578e-01,  2.8298e-01,\n",
       "          1.8813e+00, -1.9740e+00,  1.0221e+00,  2.5351e-02,  8.8454e-01,\n",
       "         -3.1019e-02, -1.2024e+00, -7.6022e-01,  2.3599e+00,  4.3342e-01,\n",
       "         -1.4911e+00, -1.8171e+00,  3.1752e+00, -8.4787e-01, -2.6176e-01,\n",
       "         -2.4772e+00, -2.1579e-01, -1.8123e+00],\n",
       "        [ 1.6255e+00, -9.7396e-01, -2.8700e-01, -7.3277e-01, -7.4025e-01,\n",
       "          1.7996e+00, -5.6025e-01, -1.1269e+00,  1.3040e-01,  1.5784e+00,\n",
       "          7.2693e-01, -1.1640e+00,  2.5077e+00, -1.1092e+00,  1.4315e+00,\n",
       "         -1.6928e-01, -1.0006e+00,  2.1357e-01, -1.8996e+00, -1.6179e+00,\n",
       "         -5.3331e-01, -9.6995e-02, -5.5277e-01],\n",
       "        [-1.2942e-01,  1.7531e+00,  1.2613e+00, -6.1684e-01, -1.0448e+00,\n",
       "          3.8583e-01,  5.5967e-01, -7.6490e-01, -1.3691e+00, -7.4254e-01,\n",
       "         -6.1801e-01, -2.4623e+00,  5.4438e-01,  1.6291e-02,  9.7013e-02,\n",
       "         -4.2504e-01,  4.5972e-02, -8.8440e-01,  8.1780e-02,  4.5479e-01,\n",
       "          1.0666e+00, -1.6575e+00,  5.2121e-01],\n",
       "        [-4.9489e-01, -1.3061e-01,  1.7024e+00,  1.1070e+00, -1.9368e-01,\n",
       "         -3.0838e-01,  7.2037e-01, -9.7681e-01, -4.0122e-01,  1.7588e-01,\n",
       "         -1.3683e+00, -5.7922e-01,  5.8105e-01, -7.1818e-01,  9.1150e-01,\n",
       "         -1.0377e-02,  4.6405e-01, -1.2826e+00,  4.0026e-01,  8.8992e-02,\n",
       "         -6.1300e-01, -7.1794e-01, -1.3024e+00],\n",
       "        [-5.5622e-01, -1.7814e+00,  1.3475e-01, -1.3978e+00,  1.0538e+00,\n",
       "          1.0623e+00,  6.0301e-01,  8.1401e-01, -8.8276e-02, -3.4456e-01,\n",
       "         -4.0565e-01, -6.1802e-01,  1.9864e+00,  1.4386e+00,  1.5970e+00,\n",
       "          3.5818e-01,  3.2612e-01,  2.5245e+00, -1.1329e+00, -2.8670e-01,\n",
       "         -4.8489e-01,  1.3820e+00, -2.2591e+00],\n",
       "        [ 2.5130e-01, -4.7077e-01, -7.7623e-01,  3.9318e-01, -4.8960e-01,\n",
       "         -1.5893e-01,  1.4466e+00,  4.5993e-01,  1.2436e+00, -6.4161e-01,\n",
       "         -9.6691e-01,  7.3241e-01, -4.6972e-01,  1.0425e+00,  4.7464e-01,\n",
       "          4.3148e-01,  5.8403e-01, -2.1586e-01,  5.6033e-01, -4.8158e-01,\n",
       "          1.5954e+00, -1.4141e+00, -6.5668e-01],\n",
       "        [ 1.1142e-01, -1.7094e+00,  4.1432e-01,  2.1122e+00,  1.2153e-01,\n",
       "         -1.9289e+00,  8.9772e-01, -4.6843e-01,  1.9718e-01,  5.8592e-01,\n",
       "         -1.5820e-01,  6.2766e-01,  1.1660e+00, -1.1501e+00,  2.0902e-03,\n",
       "          6.1637e-01, -4.1892e-01, -1.4610e-01, -9.3883e-01,  2.8957e-01,\n",
       "          5.3938e-01,  2.1542e-01,  6.2522e-02],\n",
       "        [-8.8307e-01,  1.4768e+00,  3.8939e-01, -1.8815e-01,  5.1347e-01,\n",
       "         -1.6767e+00,  6.1305e-01, -1.0179e+00, -1.1121e+00,  1.0612e+00,\n",
       "         -5.0803e-01, -7.6460e-01,  2.5374e-02,  7.9666e-01, -1.9969e-01,\n",
       "         -1.1243e+00, -6.5737e-01,  2.4809e-01, -7.4305e-01,  2.6273e-01,\n",
       "         -2.2055e+00, -7.8564e-01,  1.4449e+00],\n",
       "        [-8.6263e-02, -5.1073e-01,  3.3203e-01, -4.1298e-01, -8.8586e-01,\n",
       "          8.9832e-01,  1.4073e+00,  1.4606e+00, -1.5882e+00, -2.7847e-01,\n",
       "         -9.4072e-01,  4.0043e-01,  3.5120e-01,  9.6688e-01, -4.2043e-01,\n",
       "         -3.3148e-01, -1.4899e+00,  1.3092e+00, -1.0466e-01, -3.8151e-01,\n",
       "          2.0022e-01,  7.1774e-01, -2.9872e-01]])"
      ]
     },
     "metadata": {},
     "execution_count": 11
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "x2.data"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[-8.0376e-01,  1.4647e-01,  2.6804e-01,  3.5481e-01,  1.5795e-02,\n",
       "          1.3872e-01,  3.4475e-01, -2.7923e-01,  9.9045e-01,  6.9297e-01,\n",
       "         -5.5273e-01,  1.0695e+00,  1.4228e-01,  2.7997e-01, -3.2946e-01,\n",
       "         -5.1557e-02, -6.0712e-01,  5.7552e-02, -2.2274e+00,  1.4848e+00,\n",
       "         -2.7376e-01,  2.7362e-02,  1.5364e+00],\n",
       "        [ 2.2755e-01,  7.2207e-01,  1.6892e+00,  9.3856e-01,  1.5081e+00,\n",
       "         -2.8657e-02,  3.5874e-01, -4.7104e-01, -6.1006e-01, -1.4048e+00,\n",
       "          4.8653e-01, -2.2149e-01,  1.8077e+00, -4.4378e-01, -9.5723e-02,\n",
       "         -3.5231e-01,  1.0047e+00,  9.0307e-01, -1.5678e+00,  4.6605e-01,\n",
       "          4.7620e-01,  3.9073e-01,  9.4730e-03],\n",
       "        [-1.0358e+00,  1.7686e+00,  2.0033e+00,  4.9726e-01, -1.3719e+00,\n",
       "          1.0420e+00,  1.6000e-01,  9.5809e-01, -5.9137e-01, -4.2875e-01,\n",
       "         -1.3759e+00,  1.9779e+00, -5.9183e-01,  7.4635e-01, -8.1359e-01,\n",
       "         -8.0251e-01,  2.4896e-01, -3.3909e-01,  2.0032e-01,  1.3580e+00,\n",
       "          6.5844e-01,  9.2425e-02,  1.6641e+00],\n",
       "        [ 1.0686e-01,  9.8976e-03,  1.7733e+00, -6.7655e-01,  2.0650e+00,\n",
       "         -3.9291e-01, -2.9600e-01, -1.1717e+00, -1.4352e+00,  2.8235e-01,\n",
       "         -3.7990e-01, -1.6375e+00, -3.4651e-01,  2.9772e-01,  1.5205e+00,\n",
       "          2.5014e-01, -1.9056e+00, -4.3440e-01, -9.9309e-02,  7.0252e-01,\n",
       "         -3.3539e-01, -1.3643e+00,  6.7900e-02],\n",
       "        [ 6.8059e-01, -6.5804e-01,  1.0675e+00,  8.9371e-02, -9.1871e-01,\n",
       "         -7.3087e-02, -7.4175e-01,  3.5131e-01, -6.3622e-01, -1.3534e+00,\n",
       "          1.7203e+00,  7.6398e-01, -6.6748e-01, -2.6534e-01, -4.7715e-02,\n",
       "          1.5770e-01,  2.9351e-01,  2.1106e-01,  1.6469e+00, -6.8995e-02,\n",
       "          4.8197e-01,  1.4299e+00,  1.1817e+00],\n",
       "        [-2.0124e+00, -3.0986e-01, -2.8832e-01, -1.5645e+00, -9.6728e-01,\n",
       "          4.4763e-01, -2.6425e-02,  2.1292e-01,  2.0410e-01,  1.5156e+00,\n",
       "          1.0822e+00,  6.8500e-01, -3.0196e-01, -3.7130e-01, -6.9885e-01,\n",
       "          3.3726e-01,  4.0180e-01, -2.9727e-01, -5.8624e-01, -5.3661e-01,\n",
       "          9.2177e-01,  7.8870e-01, -1.0353e-01],\n",
       "        [-8.6743e-01, -9.6013e-01,  3.7820e-01, -1.8773e+00,  8.1681e-01,\n",
       "         -1.8189e-01, -3.9152e-01, -7.2111e-01,  1.0947e-01, -4.6188e-01,\n",
       "         -1.4131e+00, -9.3863e-01, -7.6708e-01, -1.7881e-01, -2.5826e-03,\n",
       "         -1.8603e-01,  1.9843e+00,  8.4100e-01, -1.1480e+00, -1.3222e+00,\n",
       "          1.5693e+00, -3.9685e-02,  1.8003e+00],\n",
       "        [ 1.3068e-01,  1.7193e+00,  9.4066e-01, -7.6710e-01, -1.0162e+00,\n",
       "         -2.1806e+00, -1.3724e-01, -5.4335e-01,  2.6989e-01,  7.6332e-01,\n",
       "          1.5141e+00,  7.9552e-01, -1.4277e-01,  1.1525e-01, -5.2724e-01,\n",
       "         -2.7524e-01,  4.4419e-01, -2.2286e-01,  2.8208e-01,  3.1420e-01,\n",
       "         -4.8821e-01, -1.4537e+00,  8.6589e-01],\n",
       "        [-7.2660e-01,  6.2819e-01,  7.8414e-01, -3.2921e-02,  1.1608e+00,\n",
       "          7.0336e-01,  3.6440e-01,  1.1498e+00,  2.0715e-01,  1.0674e+00,\n",
       "          9.4139e-02, -1.3580e+00, -9.9186e-01,  4.5268e-01, -2.0779e-02,\n",
       "         -7.1860e-01, -2.4092e+00,  4.9897e-01,  3.8530e-01,  1.6127e-01,\n",
       "         -8.5063e-01, -4.0928e-01,  2.3132e-01],\n",
       "        [-1.3484e+00, -2.2708e+00, -8.9870e-02, -1.1237e+00, -1.1731e+00,\n",
       "          1.2649e+00, -2.1773e-03, -1.1954e+00, -1.0829e+00, -7.3585e-01,\n",
       "          8.9904e-01, -2.2153e+00,  1.2470e+00, -1.2273e-01,  1.6576e+00,\n",
       "         -1.0838e+00,  7.8243e-01, -4.9396e-01, -3.9647e-01,  2.7506e-01,\n",
       "         -9.1782e-01,  1.7463e+00, -1.8463e+00],\n",
       "        [ 1.1476e+00, -8.0517e-02, -4.2227e-01,  1.9925e-01,  2.1168e-01,\n",
       "         -2.2342e+00, -1.6919e+00, -1.5482e+00,  6.0199e-02,  1.1141e+00,\n",
       "         -1.8963e+00, -1.1360e+00, -8.3956e-01, -2.9780e-01, -1.8273e-01,\n",
       "         -9.0401e-01, -1.1187e+00, -2.8199e-01,  1.5620e+00,  3.1030e-01,\n",
       "         -6.1423e-01,  1.9003e+00, -1.0622e+00],\n",
       "        [ 4.3430e-01,  6.3485e-01, -1.1233e+00, -1.5373e+00, -1.2517e-01,\n",
       "         -9.2152e-01,  6.8428e-01,  1.3999e+00, -4.5760e-01,  7.9467e-02,\n",
       "          2.5068e-01, -7.4106e-02,  3.8985e-01,  1.1055e-01,  1.6945e+00,\n",
       "         -1.3396e+00, -3.6785e-01,  1.5331e+00,  2.6652e-01, -1.9801e+00,\n",
       "          6.5811e-01, -1.3713e+00, -1.6273e+00]])"
      ]
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "x3.data"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[-4.8534e-01, -1.1212e+00,  2.4476e-01, -6.7335e-01,  2.9018e-01,\n",
       "         -7.7006e-01,  3.8578e-01, -3.5645e-01, -4.8541e-01, -4.1493e-01,\n",
       "         -5.3967e-01, -4.7828e-01,  8.5377e-01, -2.0661e-01,  1.3739e+00,\n",
       "          1.9713e+00, -1.1111e-01, -1.5334e+00,  1.8913e-01,  5.4865e-01,\n",
       "         -1.9779e+00, -6.4604e-01, -1.4073e+00],\n",
       "        [-5.0758e-01,  1.5364e+00,  2.5612e+00, -3.0844e-01,  8.1289e-02,\n",
       "         -1.3138e+00, -9.6680e-01,  4.3937e-01,  1.0111e+00,  1.2071e+00,\n",
       "         -5.6007e-01,  1.5521e+00,  2.8156e-01, -6.9735e-01,  5.7842e-01,\n",
       "          2.5003e-01,  7.7192e-01,  3.7598e-01,  4.9031e-01,  1.4758e+00,\n",
       "          1.0504e+00,  1.7253e+00, -6.4296e-01],\n",
       "        [-8.5327e-01, -4.7621e-01,  8.7918e-01, -1.1781e+00,  1.7463e-01,\n",
       "         -5.0065e-02, -1.9840e+00,  2.8272e-01,  3.6420e-01,  6.0833e-01,\n",
       "          1.4822e+00,  1.0106e+00, -1.3307e+00,  1.3743e+00, -5.1769e-01,\n",
       "          1.1176e-01, -1.1042e+00,  2.8250e-01, -1.1586e-01,  1.9232e+00,\n",
       "          2.0642e+00, -8.5339e-01,  4.5129e-01],\n",
       "        [ 5.0091e-01, -2.9587e-01, -5.0268e-01,  2.3585e-01, -1.7318e+00,\n",
       "         -1.4916e+00,  9.7555e-01,  8.4020e-01, -1.0281e+00, -1.4706e+00,\n",
       "         -6.9310e-01,  6.4572e-01, -6.9677e-01,  6.4532e-01, -5.0184e-02,\n",
       "          1.2586e+00,  3.4544e-01, -6.0064e-01,  5.7394e-01, -9.4235e-01,\n",
       "          7.7327e-01,  4.6138e-01,  2.5522e-01],\n",
       "        [ 7.3013e-01, -2.0802e-01, -4.9584e-01, -6.3639e-02, -2.9576e-01,\n",
       "         -2.0428e+00, -2.5473e+00,  1.0109e+00,  1.4240e+00, -4.4904e-01,\n",
       "         -3.8008e-01, -6.6679e-01,  4.9162e-02, -1.0929e+00,  1.4119e+00,\n",
       "         -4.3276e-01, -4.1313e-01, -9.8708e-01,  1.3931e+00,  1.0837e-01,\n",
       "          3.7227e-01, -3.5198e-01,  1.2714e+00],\n",
       "        [-2.5435e+00, -1.8978e+00,  1.0293e+00,  3.3196e-01,  1.8786e+00,\n",
       "          1.0300e+00, -4.1165e-01, -3.0275e-01,  1.3217e-01, -6.3597e-01,\n",
       "          1.0461e-01, -2.5519e+00,  7.2257e-01, -1.3368e-01,  2.2063e-03,\n",
       "         -6.8396e-01,  1.3606e+00, -6.9864e-01,  4.1690e-02,  1.0745e+00,\n",
       "         -1.8555e+00,  3.8607e-01,  7.4189e-01],\n",
       "        [ 7.7834e-01,  4.8460e-01, -2.5899e-01,  9.6962e-02, -3.3163e-02,\n",
       "         -1.3124e+00,  6.1929e-02, -6.0028e-01,  4.9112e-01,  8.4886e-01,\n",
       "         -2.9387e-02,  2.1024e-01,  1.0770e+00,  4.3588e-01, -1.7062e+00,\n",
       "         -1.0853e+00, -8.1065e-01, -6.3258e-01, -2.9816e-01, -2.6045e-01,\n",
       "          6.8552e-03, -4.7307e-01,  1.8071e-01],\n",
       "        [-1.6683e-01, -1.4248e+00,  9.3615e-01,  1.1049e+00,  5.5291e-01,\n",
       "         -3.5216e-01,  7.7819e-01,  7.2578e-01, -5.2688e-01,  1.6756e+00,\n",
       "         -1.1414e+00, -1.9124e+00, -1.4818e+00,  4.9635e-01, -8.0127e-01,\n",
       "          1.8658e-02,  4.7293e-01, -1.2909e+00,  1.0216e-02, -7.6093e-01,\n",
       "          3.8187e-01,  1.4918e+00,  8.1025e-02],\n",
       "        [ 8.3364e-02,  9.3273e-01,  9.5534e-01,  7.1530e-01,  6.9008e-01,\n",
       "         -3.4732e-01,  1.0381e+00, -6.3553e-02, -4.5357e-01, -1.2467e+00,\n",
       "         -1.2313e+00, -1.5801e+00, -5.9285e-01,  4.3600e-01, -1.0970e+00,\n",
       "         -1.2442e+00,  4.0403e-02, -1.3635e-01,  2.5329e-01,  8.6790e-01,\n",
       "          9.1903e-01, -7.1955e-01,  5.5651e-01],\n",
       "        [-1.3606e+00,  2.8303e-01, -1.4269e+00, -7.9313e-01,  1.0162e+00,\n",
       "          2.4351e-01, -6.3081e-03,  1.2944e+00, -2.0339e+00, -1.5895e+00,\n",
       "          8.9820e-01,  1.5409e+00, -2.5772e-03,  1.5352e+00,  1.0076e+00,\n",
       "          3.6794e-01,  1.4828e-01, -2.3440e+00, -3.3338e-01,  8.1280e-02,\n",
       "          1.2755e+00,  2.0565e-01,  7.1317e-02],\n",
       "        [-2.5576e-01,  7.3566e-01, -7.8445e-02, -4.9739e-01,  6.7564e-01,\n",
       "          8.1005e-03, -1.0023e+00, -3.5937e-01, -8.4077e-01, -1.2648e+00,\n",
       "          5.5050e-01, -1.2325e+00,  6.0418e-01,  1.1521e+00, -1.7205e+00,\n",
       "         -9.4743e-01, -7.5345e-01,  8.0838e-01,  1.1715e-01,  1.1421e-01,\n",
       "         -1.0499e+00, -1.0765e+00, -4.2897e-01],\n",
       "        [-2.4112e+00, -7.3537e-01, -2.1970e-01, -4.0713e-01,  6.3874e-02,\n",
       "          1.1040e+00, -4.1973e-01,  2.7643e-02, -1.3527e+00,  1.0129e+00,\n",
       "          4.2519e-02,  9.3525e-01, -1.4373e+00,  1.2800e+00,  1.0214e+00,\n",
       "          3.4062e-02, -1.2973e+00, -6.8308e-01, -4.5150e-01, -7.4374e-01,\n",
       "          2.9035e-01,  1.6340e+00, -1.4126e+00]])"
      ]
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# How do we set up a loss function and optimize it ? \n",
    "\n",
    "Choosing the right loss function increases the chances of model convergence. \n",
    "\n",
    "we use another tensor as the update variable, and introduce\n",
    "the tensors to the sample model and compute the error or loss. Then we\n",
    "compute the rate of change in the loss function to measure the choice of\n",
    "loss function in model convergence.\n",
    "\n",
    "In the following example, t_c and t_u are two tensors. This can be\n",
    "constructed from any NumPy array.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "torch.__version__"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'1.7.1'"
      ]
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "torch.tensor"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<function _VariableFunctionsClass.tensor>"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The sample model is just a linear equation to make the calculation\n",
    "happen and the loss function defined if the mean square error (MSE)\n",
    "shown next. For now, this is just a simple linear equation\n",
    "computation.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "#height of people\n",
    "t_c = torch.tensor([58.0, 59.0, 60.0, 61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0, 70.0, 71.0, 72.0])\n",
    "\n",
    "#weight of people\n",
    "t_u = torch.tensor([115.0, 117.0, 120.0, 123.0, 126.0, 129.0, 132.0, 135.0, 139.0, 142.0, 146.0, 150.0, 154.0, 159.0,164.0])\n",
    "\n"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let’s now define the model. The w parameter is the weight tensor,\n",
    "which is multiplied with the t_u tensor. The result is added with a constant\n",
    "tensor, b, and the loss function chosen is a custom-built one; it is also available in PyTorch. \n",
    "\n",
    "In the following example, t_u is the tensor used, t_p\n",
    "is the tensor predicted, and t_c is the precomputed tensor, with which the\n",
    "predicted tensor needs to be compared to calculate the loss function.\n",
    "\n",
    "The formula $$w * t_u + b$$ is the linear equation representation of a\n",
    "tensor-based computation.\n",
    "\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "def model(t_u, w, b):\n",
    "    return w * t_u + b\n",
    "  \n",
    "def loss_fn(t_p, t_c):\n",
    "    squared_diffs = (t_p - t_c)**2\n",
    "    return squared_diffs.mean()\n",
    "  \n",
    "w = torch.ones(1)\n",
    "b = torch.zeros(1)\n",
    "\n",
    "t_p = model(t_u, w, b)\n",
    "t_p"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([115., 117., 120., 123., 126., 129., 132., 135., 139., 142., 146., 150.,\n",
       "        154., 159., 164.])"
      ]
     },
     "metadata": {},
     "execution_count": 17
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "loss = loss_fn(t_p, t_c)\n",
    "loss"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor(5259.7334)"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The initial loss value is 5259.7334, which is too high because of the\n",
    "initial round of weights chosen. The error in the first round of iteration\n",
    "is backpropagated to reduce the errors in the second round, for which\n",
    "the initial set of weights needs to be updated. Therefore, the rate of\n",
    "change in the loss function is essential in updating the weights in the\n",
    "estimation process.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "delta = 0.1\n",
    "\n",
    "loss_rate_of_change_w = (loss_fn(model(t_u, \n",
    "                                       w + delta, b), \n",
    "                                 t_c) - loss_fn(model(t_u, w - delta, b), \n",
    "                                                t_c)) / (2.0 * delta)"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "learning_rate = 1e-2\n",
    "\n",
    "w = w - learning_rate * loss_rate_of_change_w"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "There are two parameters to update the rate of loss function: the\n",
    "learning rate at the current iteration and the learning rate at the previous\n",
    "iteration. If the delta between the two iterations exceeds a certain\n",
    "threshold, then the weight tensor needs to be updated, else model\n",
    "convergence could happen. The preceding script shows the delta and\n",
    "learning rate values. Currently, these are static values that the user has the\n",
    "option to change.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "loss_rate_of_change_b = (loss_fn(model(t_u, w, b + delta), t_c) - \n",
    "                         loss_fn(model(t_u, w, b - delta), t_c)) / (2.0 * delta)\n",
    "\n",
    "b = b - learning_rate * loss_rate_of_change_b\n",
    "\n",
    "b"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([544.])"
      ]
     },
     "metadata": {},
     "execution_count": 21
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "This is how a simple mean square loss function works in a two-­\n",
    "dimensional tensor example, with a tensor size of 10,5.\n",
    "Let’s look at the following example. The MSELoss function is within the\n",
    "neural network module of PyTorch.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "from torch import nn\n",
    "loss = nn.MSELoss()\n",
    "input = torch.randn(10, 5, requires_grad=True)\n",
    "target = torch.randn(10, 5)\n",
    "output = loss(input, target)\n",
    "output.backward()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "When we look at the gradient calculation that is used for\n",
    "backpropagation, it is shown as MSELoss.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "source": [
    "output.grad_fn"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<MseLossBackward at 0x7f8e0049afa0>"
      ]
     },
     "metadata": {},
     "execution_count": 23
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Tensor differentiation and its relevance in computational graph execution using the PyTorch framework\n",
    "\n",
    "The computational graph network is represented by nodes and connected\n",
    "through functions. There are two different kinds of nodes: dependent and\n",
    "independent. Dependent nodes are waiting for results from other nodes\n",
    "to process the input. Independent nodes are connected and are either\n",
    "constants or the results. Tensor differentiation is an efficient method to\n",
    "perform computation in a computational graph environment.\n",
    "\n",
    "In a computational graph, tensor differentiation is very effective because\n",
    "the tensors can be computed as parallel nodes, multiprocess nodes, or\n",
    "multithreading nodes. The major deep learning and neural computation\n",
    "frameworks include this tensor differentiation.\n",
    "Autograd is the function that helps perform tensor differentiation,\n",
    "which means calculating the gradients or slope of the error function,\n",
    "and backpropagating errors through the neural network to fine-tune the\n",
    "weights and biases. Through the learning rate and iteration, it tries to\n",
    "reduce the error value or loss function.\n",
    "To apply tensor differentiation, the nn.backward() method needs to\n",
    "be applied. Let’s take an example and see how the error gradients are\n",
    "backpropagated. To update the curve of the loss function, or to find where\n",
    "the shape of the loss function is minimum and in which direction it is\n",
    "moving, a derivative calculation is required. Tensor differentiation is a way\n",
    "to compute the slope of the function in a computational graph.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "source": [
    "# Make a sample tensor x, for which automatic gradient calculation needs to happen.\n",
    "x = Variable(torch.ones(4, 4) * 12.5, requires_grad=True)\n",
    "x"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[12.5000, 12.5000, 12.5000, 12.5000],\n",
       "        [12.5000, 12.5000, 12.5000, 12.5000],\n",
       "        [12.5000, 12.5000, 12.5000, 12.5000],\n",
       "        [12.5000, 12.5000, 12.5000, 12.5000]], requires_grad=True)"
      ]
     },
     "metadata": {},
     "execution_count": 24
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "source": [
    "#  Create a linear function fn that is created using the x variable.\n",
    "fn = 2 * (x * x) + 5 * x + 6\n",
    "\n",
    "\n",
    "#  Using the backward function, we can perform a backpropagation calculation. \n",
    "fn.backward(torch.ones(4,4))\n",
    "\n",
    "# The .grad() function holds the final output from the tensor differentiation.\n",
    "x.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[55., 55., 55., 55.],\n",
       "        [55., 55., 55., 55.],\n",
       "        [55., 55., 55., 55.],\n",
       "        [55., 55., 55., 55.]])"
      ]
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Define a feed forward neural network with a toy dataset\n",
    "\n",
    "The toy dataset, with 5,000 samples each having 32 features, is divided\n",
    "into 80% train and 20% test. Let’s create a class that defines the neural\n",
    "network using PyTorch’s NN module.\n",
    "\n",
    "Feed-forward neural networks were the earliest implementations within\n",
    "deep learning. These networks are called feed-forward because the\n",
    "information within them moves only in one direction (forward)—that is,\n",
    "from the input nodes (units) towards the output units.\n",
    "\n",
    "We will now implement a simple network using PyTorch. Defining the creation of a neural network for the purpose of this exercise."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "source": [
    "# Creating a toy dataset\n",
    "\n",
    "samples = 5000\n",
    "\n",
    "#Let’s divide the toy dataset into training (80%) and rest for validation.\n",
    "train_split = int(samples*0.8)\n",
    "\n",
    "#Create a dummy classification dataset\n",
    "X, y = make_blobs(n_samples=samples, centers=2, n_features=64, cluster_std=10, random_state=2020)\n",
    "y = y.reshape(-1,1)\n",
    "\n",
    "#Convert the numpy datasets to Torch Tensors\n",
    "X,y = tch.from_numpy(X),tch.from_numpy(y)\n",
    "X,y =X.float(),y.float()\n",
    "\n",
    "#Split the datasets inot train and test(validation)\n",
    "X_train, x_test = X[:train_split], X[train_split:]\n",
    "Y_train, y_test = y[:train_split], y[train_split:]\n",
    "\n",
    "#Print shapes of each dataset\n",
    "print(\"X_train.shape:\",X_train.shape)\n",
    "print(\"x_test.shape:\",x_test.shape)\n",
    "print(\"Y_train.shape:\",Y_train.shape)\n",
    "print(\"y_test.shape:\",y_test.shape)\n",
    "print(\"X.dtype\",X.dtype)\n",
    "print(\"y.dtype\",y.dtype)\n",
    "\n"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "X_train.shape: torch.Size([4000, 64])\n",
      "x_test.shape: torch.Size([1000, 64])\n",
      "Y_train.shape: torch.Size([4000, 1])\n",
      "y_test.shape: torch.Size([1000, 1])\n",
      "X.dtype torch.float32\n",
      "y.dtype torch.float32\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "source": [
    "#Define a neural network with 3 hidden layers and 1 output layer\n",
    "#Hidden Layers will have 64,256 and 1024 neurons\n",
    "#Output layers will have 1 neuron\n",
    "\n",
    "class NeuralNetwork(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        tch.manual_seed(2020)\n",
    "        self.fc1 = nn.Linear(64, 256)\n",
    "        self.relu1 = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(256, 1024)\n",
    "        self.relu2 = nn.ReLU()\n",
    "        self.out = nn.Linear(1024, 1)\n",
    "        self.final = nn.Sigmoid()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        op = self.fc1(x)\n",
    "        op = self.relu1(op)        \n",
    "        op = self.fc2(op)\n",
    "        op = self.relu2(op)\n",
    "        op = self.out(op)\n",
    "        y = self.final(op)\n",
    "        return y\n"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The torch.nn module provides the essential means to define and train\n",
    "neural networks. It contains all the necessary building blocks for creating\n",
    "neural networks of various kinds, sizes, and complexity. We will create\n",
    "a class for our neural network by inheriting this module and create an\n",
    "initializing method as well as a forward pass method.\n",
    "\n",
    "\n",
    "The __init__ method creates the different pieces of the network\n",
    "and keeps it ready for us every time we create an object with this class.\n",
    "Essentially, we used the initialization method to create the hidden layers,\n",
    "the output layer, and the activation for each layer. \n",
    "\n",
    "The `nn.Linear(64,256)` function creates a layer with 64 input features and 256 output features.\n",
    "The next layer, naturally, will have 256 input features, and so on. The `nn.ReLU()` and `nn.Sigmoid()` functions add the activation function when\n",
    "connected to a layer. Each of the individual components created within the\n",
    "initialization function is connected in the `forward()` method.\n",
    "\n",
    "\n",
    "In the forward method, we connect the individual components of\n",
    "the neural network. The first hidden layer, fc1, accepts input data and\n",
    "produces 256 outputs for the next layer. The fc1 layer is passed to the\n",
    "relu1 activation layer, which then passes the activated output to the next\n",
    "layer, fc2, which repeats the same process, to create the final output layer,\n",
    "which has the sigmoid activation function (since our toy dataset is crafted\n",
    "for binary classification).\n",
    "\n",
    "\n",
    "On creating an object of the class NeuralNetwork, and calling the\n",
    "forward method, we get outputs from the network, which are computed\n",
    "by multiplying the input matrix with a randomly initialized weight matrix\n",
    "passed through an activation function and repeated for the number of\n",
    "hidden layers until the final output layer. At first, the network would\n",
    "obviously generate junk outputs—i.e., predictions (which would add no\n",
    "value to our classification problem, at least not now).\n",
    "\n",
    "\n",
    "# Defining the Loss, Optimizer, and Training Function for the Neural Network\n",
    "\n",
    "To get more accurate predictions for our given problem, we would\n",
    "need to train the network—i.e., to backpropagate the loss and update the\n",
    "weights with respect to the loss function. Fortunately, PyTorch provides\n",
    "these essential building blocks in an extremely easy to use and intuitive\n",
    "way. \n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "source": [
    "#Define function for training a network\n",
    "def train_network(model,optimizer,loss_function, num_epochs,batch_size,X_train,Y_train):\n",
    "    #Explicitly start model training\n",
    "    model.train()\n",
    "\n",
    "    loss_across_epochs = []\n",
    "    for epoch in range(num_epochs):\n",
    "        train_loss= 0.0\n",
    "\n",
    "\n",
    "        for i in range(0,X_train.shape[0],batch_size):\n",
    "\n",
    "            #Extract train batch from X and Y\n",
    "            input_data = X_train[i:min(X_train.shape[0],i+batch_size)]\n",
    "            labels = Y_train[i:min(X_train.shape[0],i+batch_size)]\n",
    "\n",
    "            #set the gradients to zero before starting to do backpropragation \n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            #Forward pass\n",
    "            output_data  = model(input_data)\n",
    "\n",
    "            #Caculate loss\n",
    "            loss = loss_function(output_data, labels)\n",
    "\n",
    "            #Backpropogate\n",
    "            loss.backward()\n",
    "\n",
    "            #Update weights\n",
    "            optimizer.step()\n",
    "\n",
    "            train_loss += loss.item() * batch_size\n",
    "\n",
    "        print(\"Epoch: {} - Loss:{:.4f}\".format(epoch+1,train_loss ))\n",
    "        loss_across_epochs.extend([train_loss])        \n",
    "\n",
    "    #Predict\n",
    "    y_test_pred = model(x_test)\n",
    "    a =np.where(y_test_pred>0.5,1,0)\n",
    "    return(loss_across_epochs)\n",
    "###------------END OF FUNCTION--------------\n",
    "\n",
    "#Create an object of the Neural Network class\n",
    "model = NeuralNetwork()\n",
    "\n",
    "#Define loss function\n",
    "loss_function = nn.BCELoss()  #Binary Crosss Entropy Loss\n",
    "\n",
    "#Define Optimizer\n",
    "adam_optimizer = tch.optim.Adam(model.parameters(),lr= 0.001)\n",
    "\n",
    "#Define epochs and batch size\n",
    "num_epochs = 10\n",
    "batch_size=16\n",
    "\n",
    "\n",
    "#Calling the function for training and pass model, optimizer, loss and related paramters\n",
    "adam_loss = train_network(model,adam_optimizer \\\n",
    ",loss_function,num_epochs,batch_size,X_train,Y_train)\n",
    "\n"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Epoch: 1 - Loss:107.9976\n",
      "Epoch: 2 - Loss:8.7378\n",
      "Epoch: 3 - Loss:8.2710\n",
      "Epoch: 4 - Loss:0.8969\n",
      "Epoch: 5 - Loss:0.2221\n",
      "Epoch: 6 - Loss:0.0017\n",
      "Epoch: 7 - Loss:0.0016\n",
      "Epoch: 8 - Loss:0.0014\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let’s look at the individual components we defined leveraging PyTorch’s readily provided building\n",
    "blocks. We need to define a loss function that will be used to measure\n",
    "the difference between our predictions and actual labels. PyTorch\n",
    "provides a comprehensive list of loss functions for different outcomes.\n",
    "These loss functions are available under torch.nn.*. Examples include\n",
    "MSELoss (mean squared error loss), CrossEntropyLoss (for multi-class\n",
    "classification), and BCELoss (binary cross-entropy loss), which is used\n",
    "for binary classification. For our use case, we will leverage binary cross-\n",
    "entropy loss.\n",
    "\n",
    "```py\n",
    "This is defined as loss_function = torch.nn.BCELoss().\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "# Gradient-Based Optimization Techniques\n",
    "\n",
    "### Gradient Descent with Momentum\n",
    "\n",
    "Gradient descent with momentum leverages the past\n",
    "gradients to calculate an exponentially weighted average of the gradients\n",
    "to further smoothen the parameter updates.\n",
    "\n",
    "![Imgur](https://imgur.com/ZNSJbau.png)\n",
    "\n",
    "The update process can be simplified using the following equations.\n",
    "First, we compute an exponentially weighted average of the past gradients\n",
    "as νt, where \n",
    "\n",
    "$$νt = γνt − 1 + η∇ΘJ(Θ)$$\n",
    "\n",
    "and \n",
    "\n",
    "$$Θ = Θ - νt.$$\n",
    "\n",
    "\n",
    "The γ here is a hyperparameter that takes values between 0 and 1.\n",
    "Next, we use this exponentially weighted average in the updates of weights\n",
    "instead of the gradients directly.\n",
    "\n",
    "By leveraging the exponentially weighted averages of the gradients,\n",
    "instead of directly using the gradients, the incremental steps are smoother and\n",
    "faster and thus overcome the problems with oscillating around the minima.\n",
    "\n",
    "\n",
    "\n",
    "### RMSprop\n",
    "\n",
    "At the core, RMSprop computes the moving\n",
    "average of the squared gradients for each weight and divides the gradient\n",
    "by the square root of the mean square. This complex process should help\n",
    "in decoding the name root mean square prop. Leveraging exponential\n",
    "average here helps in giving recent updates more preferences than less\n",
    "recent ones.\n",
    "The RMSprop can be represented as follows:\n",
    "\n",
    "![Imgur](https://imgur.com/MaLGGt9.png)\n",
    "\n",
    "\n",
    "where η – is a hyperparameter that defines the initial learning rate, and\n",
    "gt is the gradient at time t for a parameter/weight w in Θ. We add ∈ to the\n",
    "denominator to avoid divide by zero situations.\n",
    "\n",
    "\n",
    "### Adam\n",
    "\n",
    "A simplified name for adaptive moment estimation, Adam is the most\n",
    "popular choice recently for optimizers in deep learning. In a simple way,\n",
    "Adam combines the best of RMSprop and stochastic gradient descent\n",
    "with momentum. From RMSprop, it borrows the idea of using squared\n",
    "gradients to scale the learning rate, and it takes the idea of moving\n",
    "averages of the gradient instead of directly using the gradient when\n",
    "compared to SGD with momentum.\n",
    "Here, for each weight w in Θ, we have\n",
    "\n",
    "![Imgur](https://imgur.com/7pt6keW.png)\n",
    "\n",
    "The preceding three types of optimization algorithms represent just a\n",
    "few from the breadth of available options for different types of use cases\n",
    "within deep learning. \n",
    "\n",
    "---\n",
    "\n",
    "# Training Model with Various Optimizers\n",
    "\n",
    "Next, we define an optimizer for our network.\n",
    "\n",
    "Pytorch provides a comprehensive list of optimizers that can be used for building various\n",
    "kinds of neural networks. All optimizers are organized under torch.\n",
    "optim.* (e.g., `torch.optim.SGD`, for SGD optimizer). For our use case,\n",
    "we are using the Adam optimizer (the most recommended optimizer for\n",
    "the majority of use cases). While defining the optimizer, we also need\n",
    "to define the parameters for which the gradient needs to be computed\n",
    "during backpropagation. For the neural network, this list would be all\n",
    "the weights in the feed-forward network. We can easily denote the entire\n",
    "list of model weights to the optimizer by using `model.parameters()`\n",
    "within the definition of the optimizer. We can then additionally define\n",
    "hyperparameters for the selected optimizer. By default, PyTorch provides\n",
    "fairly good values for all necessary hyperparameters. However, we can\n",
    "further override them to tailor optimizers for our use case.\n",
    "\n",
    "```py\n",
    "adam_optimizer = tch.optim.Adam(model.parameters(),lr= 0.001)\n",
    "```\n",
    "\n",
    "Lastly, we need to define the batch size and the number of epochs\n",
    "required to train our model. Batch size refers to the number of samples\n",
    "within a batch in a mini-batch update. One forward and backward pass\n",
    "for all the batches that cover all samples once is called an epoch. Finally,\n",
    "we pass all these constructs to our function to train our model. Let’s take a\n",
    "detailed look at the constructs within the function.\n",
    "\n",
    "\n",
    "In our training function, we define a structure to train our network with\n",
    "the provided optimizer, loss function, model object, and training data over\n",
    "batches for the defined number of epochs. First, we initiate our model for\n",
    "training mode with model.train(). Setting the model object to train mode\n",
    "explicitly is essential; the same would be essential while leveraging the\n",
    "model for evaluation—i.e., explicitly setting the model to evaluate mode\n",
    "with model.eval(). This ensures that the model is aware of the time when\n",
    "it is expected to update the parameters and when to not. In the preceding\n",
    "example, we did not add the evaluation loop because it is a tiny toy dataset.\n",
    "\n",
    "\n",
    "We will train the network over mini-batches. The for loop divides the\n",
    "training data into batches with our defined size. The training data, along\n",
    "with the corresponding labels, is extracted for a batch using the following\n",
    "code:\n",
    "\n",
    "\n",
    "\n",
    "```py\n",
    "input_data = X_train[i:min(X_train.shape[0],i+batch_size)]\n",
    "labels = Y_train[i:min(X_train.shape[0],i+batch_size)]\n",
    "\n",
    "```\n",
    "\n",
    "We then need to set the gradients to zero before starting to do\n",
    "backpropagation using optimizer.zero_grad(). Missing this step will\n",
    "accumulate the gradients on subsequent backward passes and lead to\n",
    "undesirable effects. This behavior is by design in PyTorch. Then, we\n",
    "calculate the forward pass using output_data = model(input_data).\n",
    "The forward pass is the execution of the forward() function in our class\n",
    "definition. It connects the different layers we defined for the network,\n",
    "which finally outputs the prediction for each sample. \n",
    "\n",
    "Once we have the predictions, we can calculate its deviation from the actual label using the\n",
    "loss function—i.e., \n",
    "\n",
    "$$loss = loss_function(output_data, labels)$$\n",
    "\n",
    "\n",
    "To backpropagate our loss, PyTorch provides a built-in module that\n",
    "does the heavy lifting for computing gradients for the loss with respect to\n",
    "the weights. We simply call the loss.backward() method, and the entire\n",
    "backpropagation is taken care of. \n",
    "\n",
    "Once the gradients are computed, it is time to update our model weights. This is done in the step\n",
    "optimizer.step(). The optimizer step is aware of the parameters that\n",
    "need to be updated with the gradient, as we provided them while defining\n",
    "our optimizer. Calling the optimizer.step() function updates the weights\n",
    "for the network, automatically taking into account the hyperparameters\n",
    "defined within the optimizer—in our case, the learning rate.\n",
    "We repeat this process over batches for the entire training sample. The\n",
    "training process is repeated for multiple epochs, and with each iteration\n",
    "we expect the losses to reduce and the weights to align in order to achieve\n",
    "better accuracy for predictions.\n",
    "\n",
    "Below code ses different optimizers to illustrate the training process\n",
    "for the preceding neural network. Since the network was trained for a toy\n",
    "dataset, we will plot the total losses after each epoch for different optimizers,\n",
    "instead of plotting the validation accuracy. We can study the outputs i.e. loss\n",
    "across epochs for each optimization variant showcased in the plot at the bottom.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "\n",
    "#Define loss function\n",
    "loss_function = nn.BCELoss()  #Binary Crosss Entropy Loss\n",
    "num_epochs = 10\n",
    "batch_size=16\n",
    "\n",
    "#Define a model object from the class defined earlier\n",
    "model = NeuralNetwork()\n",
    "\n",
    "#Train network using RMSProp optimizer\n",
    "rmsprp_optimizer = tch.optim.RMSprop(model.parameters()\n",
    ", lr=0.01, alpha=0.9\n",
    ", eps=1e-08, weight_decay=0.1\n",
    ", momentum=0.1, centered=True)\n",
    "print(\"RMSProp...\")\n",
    "\n",
    "rmsprop_loss = train_network(model,rmsprp_optimizer,loss_function\n",
    ",num_epochs,batch_size,X_train,Y_train)\n",
    "\n",
    "\n",
    "#Train network using Adam optimizer\n",
    "\n",
    "model = NeuralNetwork()\n",
    "adam_optimizer = tch.optim.Adam(model.parameters(),lr= 0.001)\n",
    "print(\"Adam...\")\n",
    "adam_loss = train_network(model,adam_optimizer,loss_function\n",
    ",num_epochs,batch_size,X_train,Y_train)\n",
    "\n",
    "#Train network using SGD optimizer\n",
    "\n",
    "model = NeuralNetwork()\n",
    "sgd_optimizer = tch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)\n",
    "print(\"SGD...\")\n",
    "sgd_loss = train_network(model,sgd_optimizer,loss_function\n",
    ",num_epochs,batch_size,X_train,Y_train) \n",
    "\n",
    "#Plot the losses for each optimizer across epochs\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "epochs = range(0,10)\n",
    "\n",
    "ax = plt.subplot(111)\n",
    "ax.plot(adam_loss,label=\"ADAM\")\n",
    "ax.plot(sgd_loss,label=\"SGD\")\n",
    "ax.plot(rmsprop_loss,label=\"RMSProp\")\n",
    "ax.legend()\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Overall Loss\")\n",
    "plt.title(\"Loss across epochs for different optimizers\")\n",
    "plt.show()\n",
    "\n"
   ],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}