Deep Learning with PyTorch/DL-with-pytorch - 9 [RNN].ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 9 [RNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Recurrent Neural Networks\n","- IMDB review sentiment classification with RNN\n","  - Last time, we have started sentence classification with CNN and have achieved accuracy over 0.80. \n","  - This time, we try training with RNN to model text data"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"2a3e5076-f593-41ba-ada7-35277c683a89","executionInfo":{"status":"ok","timestamp":1545950044665,"user_tz":420,"elapsed":62689,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K    100% |████████████████████████████████| 591.8MB 26kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x6140c000 @  0x7efc5d2cd2a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K    100% |████████████████████████████████| 61kB 20.0MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K    100% |████████████████████████████████| 2.0MB 3.3MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Installing collected packages: torch, pillow, torchvision\n","  Found existing installation: Pillow 4.0.0\n","    Uninstalling Pillow-4.0.0:\n","      Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"eaf4c480-5417-4c4a-92a9-ac51e6484baf","executionInfo":{"status":"ok","timestamp":1545962635136,"user_tz":420,"elapsed":884,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":19,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":19}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- IMDB review dataset for sentiment analysis\n","  - [source](http://ai.stanford.edu/~amaas/data/sentiment/)\n","  - Let's cheat a while and use dataset provided by Keras"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"adf8d22d-c35d-422f-9b82-477796e21232","executionInfo":{"status":"ok","timestamp":1545962643802,"user_tz":420,"elapsed":7679,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["from keras.datasets import imdb\n","from keras.preprocessing import sequence\n","\n","num_words = 10000\n","maxlen = 50\n","\n","(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_words)\n","\n","X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'pre')\n","X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'pre')\n","    \n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":20,"outputs":[{"output_type":"stream","text":["(25000, 50) (25000, 50) (25000,) (25000,)\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating RNN model and training\n","\n","- Create and train RNN model for sentence classification, with one GRU layer\n","\n","\n","![](http://upload.wikimedia.org/wikipedia/commons/thumb/3/37/Gated_Recurrent_Unit%2C_base_type.svg/440px-Gated_Recurrent_Unit%2C_base_type.svg.png)"]},{"metadata":{"id":"tdmsy2B3weeK","colab_type":"code","colab":{}},"cell_type":"code","source":["class imdbTrainDataset(torch.utils.data.Dataset):\n","  def __init__(self):\n","    self.X = X_train\n","    self.y = y_train\n","  \n","  def __getitem__(self, idx):\n","    return self.X[idx], self.y[idx]\n","  \n","  def __len__(self):\n","    return len(self.X)\n","  \n","class imdbTestDataset(torch.utils.data.Dataset):\n","  def __init__(self):\n","    self.X = X_test\n","    self.y = y_test\n","  \n","  def __getitem__(self, idx):\n","    return self.X[idx], self.y[idx]\n","  \n","  def __len__(self):\n","    return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"xFBmmDxKw1t2","colab_type":"code","colab":{}},"cell_type":"code","source":["# create dataset & dataloader instances\n","train_dataset = imdbTrainDataset()\n","test_dataset = imdbTestDataset()\n","\n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create RNN with one GRU layer\n","class net(nn.Module):\n","  def __init__(self, input_dim, num_words, embedding_dim, hidden_size, device):\n","    super(net, self).__init__()\n","    self.input_dim = input_dim\n","    self.embedding_dim = embedding_dim\n","    self.hidden_size = hidden_size\n","    self.device = device\n","           \n","    self.embedding = nn.Embedding(num_words, self.embedding_dim)\n","    \n","    # recurrent layer (GRU)\n","    self.rnn = nn.GRU(input_size = self.embedding_dim, hidden_size = hidden_size, num_layers = 2, bidirectional = True)\n","    self.dense = nn.Linear(hidden_size * 2, 2)     \n","    \n","  def forward(self, x):\n","\n","    x = self.embedding(x)                                   # project to word embedding space\n","    \n","    h0 = torch.from_numpy(np.zeros((4, x.size(1), self.hidden_size))).float().to(self.device)\n","    x, _ = self.rnn(x, h0)\n","    x = x[:, -1, :]\n","    x = self.dense(x)\n","    return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = maxlen\n","EMBEDDING_DIM = 50\n","HIDDEN_SIZE = 30\n","NUM_WORDS = num_words\n","LEARNING_RATE = 1e-2\n","NUM_EPOCHS = 50        "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_WORDS, EMBEDDING_DIM, HIDDEN_SIZE, DEVICE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss()   # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","outputId":"db578f95-e97d-45ba-cf80-42f2d3578ce0","executionInfo":{"status":"ok","timestamp":1545968636915,"user_tz":420,"elapsed":561170,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":897}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n","  temp_loss = []\n","  for (x, y) in train_loader:\n","    x, y = x.long().to(DEVICE), y.to(DEVICE)  # beware that input to embedding should be type 'long'\n","    outputs = model(x)\n","    loss = criterion(outputs, y)\n","    temp_loss.append(loss.item())\n","    \n","    optimizer.zero_grad()\n","    loss.backward()\n","    optimizer.step()\n","    \n","  print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":44,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 0.6862862052357926\n","Loss at 1th epoch: 0.6511446015567196\n","Loss at 2th epoch: 0.6249510837452752\n","Loss at 3th epoch: 0.6097113420160449\n","Loss at 4th epoch: 0.6029126274646545\n","Loss at 5th epoch: 0.592951243629261\n","Loss at 6th epoch: 0.5927904642054013\n","Loss at 7th epoch: 0.5877855566083169\n","Loss at 8th epoch: 0.5805982343700468\n","Loss at 9th epoch: 0.5823732127948683\n","Loss at 10th epoch: 0.5791789341946038\n","Loss at 11th epoch: 0.5721504489073947\n","Loss at 12th epoch: 0.567636356365924\n","Loss at 13th epoch: 0.5686291468386747\n","Loss at 14th epoch: 0.5680928979905284\n","Loss at 15th epoch: 0.5664665400981903\n","Loss at 16th epoch: 0.5596896381098397\n","Loss at 17th epoch: 0.5586986120562164\n","Loss at 18th epoch: 0.559884287416935\n","Loss at 19th epoch: 0.5565901298303993\n","Loss at 20th epoch: 0.5537556493464781\n","Loss at 21th epoch: 0.5537246392697704\n","Loss at 22th epoch: 0.5574867477526471\n","Loss at 23th epoch: 0.5543662172799207\n","Loss at 24th epoch: 0.5542050087634398\n","Loss at 25th epoch: 0.5516191876056243\n","Loss at 26th epoch: 0.551948319892494\n","Loss at 27th epoch: 0.5519226010964842\n","Loss at 28th epoch: 0.5533486927042202\n","Loss at 29th epoch: 0.5516620681298022\n","Loss at 30th epoch: 0.5526148445752203\n","Loss at 31th epoch: 0.5525032734992553\n","Loss at 32th epoch: 0.5505632563513152\n","Loss at 33th epoch: 0.5500600316694805\n","Loss at 34th epoch: 0.5512854948031659\n","Loss at 35th epoch: 0.5485446401396576\n","Loss at 36th epoch: 0.5485566307087334\n","Loss at 37th epoch: 0.54866475718362\n","Loss at 38th epoch: 0.5484578428523881\n","Loss at 39th epoch: 0.5509587942945714\n","Loss at 40th epoch: 0.5505209299070495\n","Loss at 41th epoch: 0.550063682910131\n","Loss at 42th epoch: 0.5487506185867348\n","Loss at 43th epoch: 0.5487663972438598\n","Loss at 44th epoch: 0.5497341061733207\n","Loss at 45th epoch: 0.5474839957088841\n","Loss at 46th epoch: 0.5488186151701577\n","Loss at 47th epoch: 0.5494940557650158\n","Loss at 48th epoch: 0.5462039275741091\n","Loss at 49th epoch: 0.5493884527561615\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained RNN model with accuracy score \n","  - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"1333224a-acb7-4f8c-df07-f7d0e10cf0f1","executionInfo":{"status":"ok","timestamp":1545968664991,"user_tz":420,"elapsed":6426,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n","  for x, y in test_loader:\n","    x, y = x.long().to(DEVICE), y.to(DEVICE)       # beware that input to embedding should be type 'long'\n","    outputs = F.softmax(model(x)).max(1)[-1]       # predicted label\n","    y_true += list(y.cpu().numpy())                # true label\n","    y_pred += list(outputs.cpu().numpy())   "],"execution_count":45,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n","  \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"f35fe987-a7bb-4b6c-cfbd-686c7cde745d","executionInfo":{"status":"ok","timestamp":1545968664992,"user_tz":420,"elapsed":4120,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":46,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.59312"]},"metadata":{"tags":[]},"execution_count":46}]}]}