PyTorch Basics/pytorch-model-basics-2 [linear model].ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"pytorch-model-basics-2.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Building Blocks of Models\n","- ```nn.Linear```\n","- Nonlinear Activations\n","- Loss functions\n","- Optimizers"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"6a1f2dd8-7ff5-4a3e-f78b-0f89eda954b3","executionInfo":{"status":"ok","timestamp":1538894131201,"user_tz":420,"elapsed":171284,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/49/0e/e382bcf1a6ae8225f50b99cc26effa2d4cc6d66975ccf3fa9590efcbedce/torch-0.4.1-cp36-cp36m-manylinux1_x86_64.whl (519.5MB)\n","\u001b[K    100% |████████████████████████████████| 519.5MB 33kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x590bc000 @  0x7fd7ce2351c4 0x46d6a4 0x5fcbcc 0x4c494d 0x54f3c4 0x553aaf 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54efc1 0x54f24d 0x551ee0 0x54efc1 0x54f24d 0x551ee0 0x54e4c8 0x54f4f6 0x553aaf 0x54e4c8\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K    100% |████████████████████████████████| 61kB 20.8MB/s \n","\u001b[?25hCollecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K    100% |████████████████████████████████| 2.0MB 4.3MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n","  Found existing installation: Pillow 4.0.0\n","    Uninstalling Pillow-4.0.0:\n","      Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-0.4.1 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"7031d993-5a68-47dc-bfae-68c70ebb15d3","executionInfo":{"status":"ok","timestamp":1538894145751,"user_tz":420,"elapsed":7742,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","torch.__version__"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'0.4.1'"]},"metadata":{"tags":[]},"execution_count":3}]},{"metadata":{"id":"gyv2Sy5WO8lK","colab_type":"code","colab":{}},"cell_type":"code","source":["import torch.nn as nn"],"execution_count":0,"outputs":[]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. nn.Linear\n","```nn.Linear()``` is one of the basic building blocks of any neural network (NN) model\n","  - Performs linear (or affine) transformation in the form of ```Wx (+ b)```. In NN terminology, generates a fully connected, or dense, layer.\n","  - Two parameters, ```in_features``` and ```out_features``` should be specified\n","  - Documentation: [linear_layers](https://pytorch.org/docs/stable/nn.html#linear-layers)\n","  \n","```python\n","torch.nn.Linear(in_features,       # size of each input sample\n","                out_features,     # size of each output sample\n","                bias = True)         # whether bias (b) will be added or not\n","                         \n","```"]},{"metadata":{"id":"MZ30Xe3qFEgG","colab_type":"code","outputId":"1ba5f2b4-0789-426c-df9d-ea0ea7116869","executionInfo":{"status":"ok","timestamp":1538894184746,"user_tz":420,"elapsed":1829,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":87}},"cell_type":"code","source":["linear = nn.Linear(5, 1)             # input dim = 5, output dim = 1\n","x = torch.FloatTensor([1, 2, 3, 4, 5])    # 1d tensor\n","print(linear(x))      \n","y = torch.ones(3, 5)                      # 2d tensor\n","print(linear(y))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([3.5314], grad_fn=<ThAddBackward>)\n","tensor([[1.1357],\n","        [1.1357],\n","        [1.1357]], grad_fn=<ThAddmmBackward>)\n"],"name":"stdout"}]},{"metadata":{"id":"TIH5QLu0Pm6h","colab_type":"text"},"cell_type":"markdown","source":["## 2. Nonlinear activations\n","PyTorch provides a number of nonlinear activation functions. Most commonly used ones are:\n","```python\n","torch.nn.ReLU()                # relu\n","torch.nn.Sigmoid()         # sigmoid\n","torch.nn.Tanh()        # tangent hyperbolic\n","torch.nn.Softmax()        # softmax\n","```\n","  - Documentation: [nonlinear_activations](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)"]},{"metadata":{"id":"fCyPxp6QFe59","colab_type":"code","outputId":"b9ab9745-ac50-4b26-d642-0038f70febf7","executionInfo":{"status":"ok","timestamp":1538894578108,"user_tz":420,"elapsed":929,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":105}},"cell_type":"code","source":["relu = torch.nn.ReLU()\n","sigmoid = torch.nn.Sigmoid()\n","tanh = torch.nn.Tanh()\n","softmax = torch.nn.Softmax(dim = 0)   # when using softmax, explicitly designate dimension\n","\n","x = torch.randn(5)     # five random numbers\n","print(x)\n","print(relu(x))       \n","print(sigmoid(x))\n","print(tanh(x))\n","print(softmax(x))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([ 0.9238, -0.9667, -0.4237, -1.7181,  2.3956])\n","tensor([0.9238, 0.0000, 0.0000, 0.0000, 2.3956])\n","tensor([0.7158, 0.2755, 0.3956, 0.1521, 0.9165])\n","tensor([ 0.7277, -0.7472, -0.4000, -0.9376,  0.9835])\n","tensor([0.1713, 0.0259, 0.0445, 0.0122, 0.7462])\n"],"name":"stdout"}]},{"metadata":{"id":"pxfoFJoqUWio","colab_type":"text"},"cell_type":"markdown","source":["## 3. Loss Functions\n","There are a number of loss functions that are already implemented in PyTorch. Common ones include:\n","- ```nn.MSELoss```: Mean squared error. Commonly used in regression tasks.\n","- ```nn.CrossEntropyLoss```: Cross entropy loss. Commonly used in classification tasks"]},{"metadata":{"id":"Bq8fI8SgVmLX","colab_type":"code","outputId":"8be47b62-7f57-4708-c1b7-1a594a915136","executionInfo":{"status":"ok","timestamp":1538895848482,"user_tz":420,"elapsed":792,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["a = torch.FloatTensor([2, 4, 5])\n","b = torch.FloatTensor([1, 3, 2])\n","\n","mse = nn.MSELoss()\n","print(mse(a, b))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor(3.6667)\n"],"name":"stdout"}]},{"metadata":{"id":"OmEnzUaOVyDD","colab_type":"code","outputId":"d91b197e-07d0-48a9-a275-8cd1d9942843","executionInfo":{"status":"ok","timestamp":1538896097420,"user_tz":420,"elapsed":912,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# note that when using CrossEntropyLoss, input has to have (N, C) shape, where\n","# N is the batch size\n","# C is the number of classes\n","a = torch.FloatTensor([[0.5, 0], [4.5, 0], [0, 0.4], [0, 0.1]])   # input\n","b = torch.LongTensor([1, 1, 1, 0])                                # target\n","\n","ce = nn.CrossEntropyLoss()\n","print(ce(a,b))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor(1.6856)\n"],"name":"stdout"}]},{"metadata":{"id":"Rj9Dv9CbXINA","colab_type":"text"},"cell_type":"markdown","source":["## 4. Optimizers\n","- ```torch.optim``` provides various optimization algorithms that are commonly used. Some of them are: \n","```python\n","optim.Adagrad   \n","optim.Adam\n","optim.RMSprop\n","optim.SGD\n","```\n","- As arguments, (model) parameters and (optionally) learning rate are passed\n","- Model training process\n","  - ```optimizer.zero_grad()```: sets all gradients to zero (for every training batches)\n","  - ```loss_fn.backward()```: back propagate with respect to the loss function\n","  - ```optimizer.step()```: update model parameters"]},{"metadata":{"id":"8HI6YPvDXuli","colab_type":"code","colab":{}},"cell_type":"code","source":["## how pytorch models are trained with loss function and optimizers\n","\n","# input and output data\n","x = torch.randn(5)\n","y = torch.ones(1)\n","\n","model = nn.Linear(5, 1)  # generate model\n","loss_fn = nn.MSELoss()   # define loss function\n","optimizer = torch.optim.RMSprop(model.parameters(), lr = 0.01)     # create optimizer \n","optimizer.zero_grad()                      # setting gradients to zero\n","loss_fn(model(x), y).backward()            # back propagation\n","optimizer.step()                           # update parameters based on gradients computed"],"execution_count":0,"outputs":[]}]}