Deep Learning with Torch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning with Torch: the 60-minute blitz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Goal of this talk\n",
    "* Understand torch and the neural networks package at a high-level.\n",
    "* Train a small neural network on CPU and GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is Torch?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Torch is a scientific computing framework based on Lua[JIT] with strong CPU and CUDA backends.\n",
    "\n",
    "Strong points of Torch:\n",
    "\n",
    "* Efficient Tensor library (like NumPy) with an efficient CUDA backend\n",
    "* Neural Networks package -- build arbitrary acyclic computation graphs with automatic differentiation\n",
    "   * also with fast CUDA and CPU backends\n",
    "* Good community and industry support - several hundred community-built and maintained packages.\n",
    "* Easy to use Multi-GPU support and parallelizing neural networks\n",
    "\n",
    "http://torch.ch  \n",
    "https://github.com/torch/torch7/wiki/Cheatsheet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before getting started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Based on Lua and runs on Lua-JIT (Just-in-time compiler) which is fast\n",
    "* Lua is pretty close to javascript.\n",
    "   * variables are global by default, unless `local` keyword is used\n",
    "* Only has one data structure built-in, a table: `{}`. Doubles as a hash-table and an array.\n",
    "* 1-based indexing.\n",
    "* `foo:bar()` is the same as `foo.bar(foo)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started\n",
    "#### Strings, numbers, tables - a tiny introduction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = 'hello'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b[1] = a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b[2] = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i=1,#b do -- the # operator is the length operator in Lua\n",
    "    print(b[i]) \n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = torch.Tensor(5,3) -- construct a 5x3 matrix, uninitialized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = torch.rand(5,3)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "b=torch.rand(3,4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- matrix-matrix multiplication: syntax 1\n",
    "a*b "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- matrix-matrix multiplication: syntax 2\n",
    "torch.mm(a,b) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- matrix-matrix multiplication: syntax 3\n",
    "c=torch.Tensor(5,4)\n",
    "c:mm(a,b) -- store the result of a*b in c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### CUDA Tensors\n",
    "Tensors can be moved onto GPU using the :cuda function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "require 'cutorch';\n",
    "a = a:cuda()\n",
    "b = b:cuda()\n",
    "c = c:cuda()\n",
    "c:mm(a,b) -- done on GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Add two tensors\n",
    "https://github.com/torch/torch7/blob/master/doc/maths.md#res-torchaddres-tensor1-tensor2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "function addTensors(a,b)\n",
    "    return a -- FIX ME\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "a = torch.ones(5,2)\n",
    "b = torch.Tensor(2,5):fill(4)\n",
    "print(addTensors(a,b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Networks\n",
    "Neural networks in Torch can be constructed using the `nn` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "require 'nn';"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Modules` are the bricks used to build neural networks. Each are themselves neural networks, but can be combined with other networks using `containers` to create complex neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, look at this network that classfies digit images:\n",
    "![LeNet](http://fastml.com/images/cifar/lenet5.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a simple feed-forward network.   \n",
    "It takes the input, feeds it through several layers one after the other, and then finally gives the output.\n",
    "\n",
    "Such a network container is `nn.Sequential` which feeds the input through several layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "net = nn.Sequential()\n",
    "net:add(nn.SpatialConvolution(1, 6, 5, 5)) -- 1 input image channel, 6 output channels, 5x5 convolution kernel\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.SpatialMaxPooling(2,2,2,2))     -- A max-pooling operation that looks at 2x2 windows and finds the max.\n",
    "net:add(nn.SpatialConvolution(6, 16, 5, 5))\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.SpatialMaxPooling(2,2,2,2))\n",
    "net:add(nn.View(16*5*5))                    -- reshapes from a 3D tensor of 16x5x5 into 1D tensor of 16*5*5\n",
    "net:add(nn.Linear(16*5*5, 120))             -- fully connected layer (matrix multiplication between input and weights)\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.Linear(120, 84))\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.Linear(84, 10))                   -- 10 is the number of outputs of the network (in this case, 10 digits)\n",
    "net:add(nn.LogSoftMax())                     -- converts the output to a log-probability. Useful for classification problems\n",
    "\n",
    "print('Lenet5\\n' .. net:__tostring());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other examples of nn containers are shown in the figure below:\n",
    "![containers](https://raw.githubusercontent.com/soumith/ex/gh-pages/assets/nn_containers.png)\n",
    "\n",
    "Every neural network module in torch has automatic differentiation.\n",
    "It has a `:forward(input)` function that computes the output for a given input, flowing the input through the network.\n",
    "and it has a `:backward(input, gradient)` function that will differentiate each neuron in the network w.r.t. the gradient that is passed in. This is done via the chain rule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "input = torch.rand(1,32,32) -- pass a random tensor as input to the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "output = net:forward(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "net:zeroGradParameters() -- zero the internal gradient buffers of the network (will come to this later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gradInput = net:backward(input, torch.rand(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(#gradInput)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Criterion: Defining a loss function\n",
    "When you want a model to learn to do something, you give it feedback on how well it is doing. This function that computes an objective measure of the model's performance is called a __loss function__.\n",
    "\n",
    "A typical loss function takes in the model's output and the groundtruth and computes a value that quantifies the model's performance.\n",
    "\n",
    "The model then corrects itself to have a smaller loss.\n",
    "\n",
    "In torch, loss functions are implemented just like neural network modules, and have automatic differentiation.  \n",
    "They have two functions - `forward(input, target)`, `backward(input, target)`\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "criterion = nn.ClassNLLCriterion() -- a negative log-likelihood criterion for multi-class classification\n",
    "criterion:forward(output, 3) -- let's say the groundtruth was class number: 3\n",
    "gradients = criterion:backward(output, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gradInput = net:backward(input, gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Review of what you learnt so far\n",
    "* Network can have many layers of computation\n",
    "* Network takes an input and produces an output in the :forward pass\n",
    "* Criterion computes the loss of the network, and its gradients w.r.t. the output of the network.\n",
    "* Network takes an (input, gradients) pair in its backward pass and calculates the gradients w.r.t. each layer (and neuron) in the network.\n",
    "\n",
    "##### Missing details\n",
    "> A neural network layer can have learnable parameters or not.\n",
    "\n",
    "A convolution layer learns its convolution kernels to adapt to the input data and the problem being solved.  \n",
    "A max-pooling layer has no learnable parameters. It only finds the max of local windows.\n",
    "\n",
    "A layer in torch which has learnable weights, will typically have fields .weight (and optionally, .bias)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "m = nn.SpatialConvolution(1,3,2,2) -- learn 3 2x2 kernels\n",
    "print(m.weight) -- initially, the weights are randomly initialized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(m.bias) -- The operation in a convolution layer is: output = convolution(input,weight) + bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also two other important fields in a learnable layer. The gradWeight and gradBias.\n",
    "The gradWeight accumulates the gradients w.r.t. each weight in the layer, and the gradBias, w.r.t. each bias in the layer.\n",
    "\n",
    "#### Training the network\n",
    "\n",
    "For the network to adjust itself, it typically does this operation (if you do Stochastic Gradient Descent):\n",
    "> weight = weight + learningRate * gradWeight [equation 1]\n",
    "\n",
    "This update over time will adjust the network weights such that the output loss is decreasing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now it is time to discuss one missing piece. Who visits each layer in your neural network and updates the weight according to Equation 1?\n",
    "\n",
    "There are multiple answers, but we will use the simplest answer.  \n",
    "We shall use the simple SGD trainer shipped with the neural network module: [__nn.StochasticGradient__](https://github.com/torch/nn/blob/master/doc/training.md#stochasticgradientmodule-criterion).\n",
    "\n",
    "It has a function :train(dataset) that takes a given dataset and simply trains your network by showing different samples from your dataset to the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What about data?\n",
    "Generally, when you have to deal with image, text, audio or video data, you can use standard functions like: [__image.load__](https://github.com/torch/image#res-imageloadfilename-depth-tensortype) or [__audio.load__](https://github.com/soumith/lua---audio#usage) to load your data into a _torch.Tensor_ or a Lua table, as convenient.\n",
    "\n",
    "Let us now use some simple data to train our network.\n",
    "\n",
    "We shall use the CIFAR-10 dataset, which has the classes: 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'.  \n",
    "The images in CIFAR-10 are of size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.\n",
    "![CIFAR-10 image](https://raw.githubusercontent.com/soumith/ex/gh-pages/assets/cifar10.png)\n",
    "\n",
    "The dataset has 50,000 training images and 10,000 test images in total.\n",
    "\n",
    "__We now have 5 steps left to do in training our first torch neural network__\n",
    "1. Load and normalize data\n",
    "2. Define Neural Network\n",
    "3. Define Loss function\n",
    "4. Train network on training data\n",
    "5. Test network on test data.\n",
    "\n",
    "__1. Load and normalize data__\n",
    "\n",
    "Today, in the interest of time, we prepared the data before-hand into a 4D torch ByteTensor of size 50000x3x32x32 (training) and 10000x3x32x32 (testing)\n",
    "Let us load the data and inspect it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "require 'paths'\n",
    "if (not paths.filep(\"cifar10torchsmall.zip\")) then\n",
    "    os.execute('wget -c https://s3.amazonaws.com/torch7/data/cifar10torchsmall.zip')\n",
    "    os.execute('unzip cifar10torchsmall.zip')\n",
    "end\n",
    "trainset = torch.load('cifar10-train.t7')\n",
    "testset = torch.load('cifar10-test.t7')\n",
    "classes = {'airplane', 'automobile', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(#trainset.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For fun, let us display an image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "itorch.image(trainset.data[100]) -- display the 100-th image in dataset\n",
    "print(classes[trainset.label[100]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to prepare the dataset to be used with __nn.StochasticGradient__, a couple of things have to be done according to its [documentation](https://github.com/torch/nn/blob/master/doc/training.md#traindataset).\n",
    "1. The dataset has to have a :size() function.\n",
    "2. The dataset has to have a [i] index operator, so that dataset[i] returns the ith sample in the datset.\n",
    "\n",
    "Both can be done quickly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- ignore setmetatable for now, it is a feature beyond the scope of this tutorial. It sets the index operator.\n",
    "setmetatable(trainset, \n",
    "    {__index = function(t, i) \n",
    "                    return {t.data[i], t.label[i]} \n",
    "                end}\n",
    ");\n",
    "trainset.data = trainset.data:double() -- convert the data from a ByteTensor to a DoubleTensor.\n",
    "\n",
    "function trainset:size() \n",
    "    return self.data:size(1) \n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(trainset:size()) -- just to test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(trainset[33]) -- load sample number 33.\n",
    "itorch.image(trainset[33][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__One of the most important things you can do in conditioning your data (in general in data-science or machine learning) is to make your data to have a mean of 0.0 and standard-deviation of 1.0.__\n",
    "\n",
    "Let us do that as a final step of our data processing.\n",
    "\n",
    "To do this, we introduce you to the tensor indexing operator.\n",
    "It is shown by example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "redChannel = trainset.data[{ {}, {1}, {}, {}  }] -- this picks {all images, 1st channel, all vertical pixels, all horizontal pixels}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(#redChannel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this indexing operator, you initally start with ___[{ }]___. You can pick all elements in a dimension using ___{}___ or pick a particular element using ___{i}___ where ___i___ is the element index. You can also pick a range of elements using ___{i1, i2}___, for example ___{3,5}___ gives us the 3,4,5 elements.\n",
    "\n",
    "__Exercise: Select the 150th to 300th data elements of the data__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- TODO: fill"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving back to mean-subtraction and standard-deviation based scaling, doing this operation is simple, using the indexing operator that we learnt above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mean = {} -- store the mean, to normalize the test set in the future\n",
    "stdv  = {} -- store the standard-deviation for the future\n",
    "for i=1,3 do -- over each image channel\n",
    "    mean[i] = trainset.data[{ {}, {i}, {}, {}  }]:mean() -- mean estimation\n",
    "    print('Channel ' .. i .. ', Mean: ' .. mean[i])\n",
    "    trainset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction\n",
    "    \n",
    "    stdv[i] = trainset.data[{ {}, {i}, {}, {}  }]:std() -- std estimation\n",
    "    print('Channel ' .. i .. ', Standard Deviation: ' .. stdv[i])\n",
    "    trainset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you notice, our training data is now normalized and ready to be used.\n",
    "\n",
    "__ 2. Time to define our neural network__\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Copy the neural network from the __Neural Networks__ section above and modify it to take 3-channel images (instead of 1-channel images as it was defined).  \n",
    "Hint: You only have to change the first layer, change the number 1 to be 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Solution:__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "net = nn.Sequential()\n",
    "net:add(nn.SpatialConvolution(3, 6, 5, 5)) -- 3 input image channels, 6 output channels, 5x5 convolution kernel\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.SpatialMaxPooling(2,2,2,2))     -- A max-pooling operation that looks at 2x2 windows and finds the max.\n",
    "net:add(nn.SpatialConvolution(6, 16, 5, 5))\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.SpatialMaxPooling(2,2,2,2))\n",
    "net:add(nn.View(16*5*5))                    -- reshapes from a 3D tensor of 16x5x5 into 1D tensor of 16*5*5\n",
    "net:add(nn.Linear(16*5*5, 120))             -- fully connected layer (matrix multiplication between input and weights)\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.Linear(120, 84))\n",
    "net:add(nn.ReLU())                       -- non-linearity \n",
    "net:add(nn.Linear(84, 10))                   -- 10 is the number of outputs of the network (in this case, 10 digits)\n",
    "net:add(nn.LogSoftMax())                     -- converts the output to a log-probability. Useful for classification problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__3. Let us define the Loss function__\n",
    "\n",
    "Let us use a Log-likelihood classification loss. It is well suited for most classification problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "criterion = nn.ClassNLLCriterion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__4. Train the neural network__\n",
    "\n",
    "This is when things start to get interesting.  \n",
    "Let us first define an __nn.StochasticGradient__ object. Then we will give our dataset to this object's ___:train___ function, and that will get the ball rolling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainer = nn.StochasticGradient(net, criterion)\n",
    "trainer.learningRate = 0.001\n",
    "trainer.maxIteration = 5 -- just do 5 epochs of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainer:train(trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__5. Test the network, print accuracy__\n",
    "\n",
    "We have trained the network for 2 passes over the training dataset.  \n",
    "But we need to check if the network has learnt anything at all.  \n",
    "We will check this by predicting the class label that the neural network outputs, and checking it against the ground-truth. If the prediction is correct, we add the sample to the list of correct predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, first step. Let us display an image from the test set to get familiar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(classes[testset.label[100]])\n",
    "itorch.image(testset.data[100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we are done with that, let us normalize the test data with the mean and standard-deviation from the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "testset.data = testset.data:double()   -- convert from Byte tensor to Double tensor\n",
    "for i=1,3 do -- over each image channel\n",
    "    testset.data[{ {}, {i}, {}, {}  }]:add(-mean[i]) -- mean subtraction    \n",
    "    testset.data[{ {}, {i}, {}, {}  }]:div(stdv[i]) -- std scaling\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- for fun, print the mean and standard-deviation of example-100\n",
    "horse = testset.data[100]\n",
    "print(horse:mean(), horse:std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now let us see what the neural network thinks these examples above are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(classes[testset.label[100]])\n",
    "itorch.image(testset.data[100])\n",
    "predicted = net:forward(testset.data[100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "-- the output of the network is Log-Probabilities. To convert them to probabilities, you have to take e^x \n",
    "print(predicted:exp())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the network predictions. The network assigned a probability to each classes, given the image.\n",
    "\n",
    "To make it clearer, let us tag each probability with its class-name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i=1,predicted:size(1) do\n",
    "    print(classes[i], predicted[i])\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, fine. One single example sucked, but how many in total seem to be correct over the test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "correct = 0\n",
    "for i=1,10000 do\n",
    "    local groundtruth = testset.label[i]\n",
    "    local prediction = net:forward(testset.data[i])\n",
    "    local confidences, indices = torch.sort(prediction, true)  -- true means sort in descending order\n",
    "    if groundtruth == indices[1] then\n",
    "        correct = correct + 1\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(correct, 100*correct/10000 .. ' % ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks waaay better than chance, which is 10% accuracy (randomly picking a class out of 10 classes). Seems like the network learnt something.\n",
    "\n",
    "Hmmm, what are the classes that performed well, and the classes that did not perform well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class_performance = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}\n",
    "for i=1,10000 do\n",
    "    local groundtruth = testset.label[i]\n",
    "    local prediction = net:forward(testset.data[i])\n",
    "    local confidences, indices = torch.sort(prediction, true)  -- true means sort in descending order\n",
    "    if groundtruth == indices[1] then\n",
    "        class_performance[groundtruth] = class_performance[groundtruth] + 1\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i=1,#classes do\n",
    "    print(classes[i], 100*class_performance[i]/1000 .. ' %')\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, so what next? How do we run this neural network on GPUs?\n",
    "\n",
    "#### cunn: neural networks on GPUs using CUDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "require 'cunn';"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea is pretty simple. Take a neural network, and transfer it over to GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "net = net:cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, transfer the criterion to GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "criterion = criterion:cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainset.data = trainset.data:cuda()\n",
    "trainset.label = trainset.label:cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, let's train on GPU :) #sosimple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainer = nn.StochasticGradient(net, criterion)\n",
    "trainer.learningRate = 0.001\n",
    "trainer.maxIteration = 5 -- just do 5 epochs of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainer:train(trainset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why dont I notice MASSIVE speedup compared to CPU?\n",
    "Because your network is realllly small. \n",
    "\n",
    "**Exercise:** Try increasing the size of your network (argument 1 and 2 of nn.SpatialConvolution(...), see what kind of speedup you get."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goals achieved:**\n",
    "* Understand torch and the neural networks package at a high-level.\n",
    "* Train a small neural network on CPU and GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where do I go next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Build crazy graphs of networks: https://github.com/torch/nngraph\n",
    "* Train on imagenet with multiple GPUs: https://github.com/soumith/imagenet-multiGPU.torch\n",
    "* Train recurrent networks with LSTM on text: https://github.com/wojzaremba/lstm\n",
    "\n",
    "* More demos and tutorials: https://github.com/torch/torch7/wiki/Cheatsheet\n",
    "\n",
    "* Chat with developers of Torch: http://gitter.im/torch/torch7\n",
    "* Ask for help: http://groups.google.com/forum/#!forum/torch7\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "iTorch",
   "language": "lua",
   "name": "itorch"
  },
  "language_info": {
   "name": "lua",
   "version": "5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}