text-classification-with-rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fight online abuse - Text Classification with RNNs \n",
    "====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning for text classification with Torchtext, PyTorch & FastAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introduction\n",
    "----\n",
    "\n",
    "This notebook shows how to: \n",
    "1. Pre-processes NLP datasets for training using the Torchtext library, \n",
    "2. Create a RNN model (GRU, LSTM, standard RNN) using the PyTorch library and\n",
    "3. Train the model using the FastAI library.\n",
    "\n",
    "The dataset used is from the [Kaggle Toxic Comment Classification Challenge competition](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge). \n",
    "\n",
    "This notebook uses GPU by default. If you would like to run it on CPU instead, disable it by changing `USE_GPU` to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_GPU = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 1. Loading Dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 1.1. Download Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "First we download and extract the dataset from the [Kaggle competition page](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "DATA_PATH = 'data/toxic-comments/'\n",
    "os.makedirs(DATA_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "ZIP_DATA_PATH = f'{DATA_PATH}zip_files/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/download/sample_submission.csv.zip\n",
      "\n",
      "sample_submission.csv.zip 100% |####################| Time: 0:00:00   3.8 MiB/s\n",
      "\n",
      "downloading https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/download/test.csv.zip\n",
      "\n",
      "test.csv.zip 100% |#################################| Time: 0:00:00  31.6 MiB/s\n",
      "\n",
      "downloading https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/download/train.csv.zip\n",
      "\n",
      "train.csv.zip 100% |################################| Time: 0:00:00  39.0 MiB/s\n",
      "\n",
      "downloading https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/download/test_labels.csv.zip\n",
      "\n",
      "test_labels.csv.zip 100% |##########################| Time: 0:00:00   5.0 MiB/s\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p {ZIP_DATA_PATH} && kaggle competitions download -c jigsaw-toxic-comment-classification-challenge -p {ZIP_DATA_PATH} \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample_submission.csv.zip test_labels.csv.zip\r\n",
      "test.csv.zip              train.csv.zip\r\n"
     ]
    }
   ],
   "source": [
    "!ls {ZIP_DATA_PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "RAW_DATA_PATH = f'{DATA_PATH}raw_data_files/'\n",
    "os.makedirs(RAW_DATA_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "import zipfile\n",
    "def unzip(path, targ_path):\n",
    "    \"\"\" Function to unzip files in data path\"\"\"\n",
    "    for file in glob.glob(path):\n",
    "        zip_ref = zipfile.ZipFile(file, 'r')\n",
    "        zip_ref.extractall(targ_path)\n",
    "        zip_ref.close()\n",
    "        print('%s unzipped' %file.split('/')[-1])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train.csv.zip unzipped\n",
      "test_labels.csv.zip unzipped\n",
      "sample_submission.csv.zip unzipped\n",
      "test.csv.zip unzipped\n"
     ]
    }
   ],
   "source": [
    "unzip(f'{ZIP_DATA_PATH}*.zip', f'{RAW_DATA_PATH}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 1.2. Import Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def get_file_names(path, file_format=''):\n",
    "    \"\"\"Function to get filenames in data path\"\"\"\n",
    "    file_list=[]\n",
    "    path = path+'*'+file_format\n",
    "    for file in glob.glob(path):\n",
    "        file_list.append(file.split('/')[-1].split('.')[0])\n",
    "    return file_list   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['test_labels', 'test', 'train', 'sample_submission']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_names = get_file_names(f'{RAW_DATA_PATH}', file_format='.csv')  \n",
    "file_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "tables = [pd.read_csv(f'{RAW_DATA_PATH}{fname}.csv', low_memory=False) \n",
    "          for fname in file_names]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 2. Preprocessing Dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.1. Looking at the data\n",
    "The training data contains a row per comment, with an id, the text of the comment, and 6 different labels that we'll try to predict.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "sample_submission_df, raw_test_labels_df, raw_train_df, raw_test_df   = tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0000997932d777bf</td>\n",
       "      <td>Explanation\\nWhy the edits made under my usern...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000103f0d9cfb60f</td>\n",
       "      <td>D'aww! He matches this background colour I'm s...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id                                       comment_text  toxic  \\\n",
       "0  0000997932d777bf  Explanation\\nWhy the edits made under my usern...      0   \n",
       "1  000103f0d9cfb60f  D'aww! He matches this background colour I'm s...      0   \n",
       "\n",
       "   severe_toxic  obscene  threat  insult  identity_hate  \n",
       "0             0        0       0       0              0  \n",
       "1             0        0       0       0              0  "
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_train_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['toxic', 'severe_toxic', 'obscene', 'threat', 'insult',\n",
       "       'identity_hate'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_column_list = raw_train_df.columns[2:]; label_column_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Here's a couple of examples of comments for each category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toxic\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Fuck you asshole! \\n\\nGo fuck yourself! Dirty fuckin' asshole! Fuckin' scum!\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "severe_toxic\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'jam your message up your arse wikipedia dickhead'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "obscene\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Read the above numbskull, would one of you vegetables please get a move on and block this I.P. I ain't leaving until this address is blocked so hop to it please assholes..\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "threat\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Die \\n\\nI HATE YOU PRICK YOU DINT DESERVE A PLACE HERE'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "insult\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'YOU HAVE NO RIGHT FOR Your RACIST block Syrthiss!!  Iguarantee that Jimbo will hear about this and you shalll be punished accordingly!  White Hoods like you DONT belong in WIKI!  Your CYBERLYNCHING DAYS R Over You Racist Jerk!'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "identity_hate\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The Israelis are committing massacres in Gaza, but nobody listens. There is even no photos which exposes those who permitted these massacres.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in label_column_list:\n",
    "    print(i)\n",
    "    display(raw_train_df[getattr(raw_train_df, i) == 1].sample(1).iloc[0]['comment_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.2. Cleaning Up Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Next we use `clean` function to replace remove newline characters and replace some punctuations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "def clean(comment):\n",
    "    # remove \\n \n",
    "    # torchtext cannot read the .csv files correctly if there are newline characters, so replace with \" \"\n",
    "    comment = re.sub(\"\\\\n\",\" \",comment)\n",
    "    \n",
    "    # remove leaky elements like ip,user\n",
    "    comment = re.sub(\"\\d{1,3}.\\d{1,3}.\\d{1,3}.\\d{1,3}\",\" \",comment)\n",
    "    \n",
    "    # removing usernames\n",
    "    comment = re.sub(\"\\[\\[.*\\]\",\"\",comment)\n",
    "    \n",
    "    comment = re.sub(r\"[\\*\\\"“”\\n\\\\…\\+\\-\\/\\=\\(\\)‘•:\\[\\]\\|’\\!;]\", \" \", str(comment))\n",
    "    \n",
    "    comment = re.sub(r\"[ ]+\", \" \", comment)\n",
    "    \n",
    "    comment = re.sub(r\"\\!+\", \"!\", comment)\n",
    "    \n",
    "    comment = re.sub(r\"\\,+\", \",\", comment)\n",
    "    \n",
    "    comment = re.sub(r\"\\?+\", \"?\", comment)\n",
    "\n",
    "    return(comment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train cleaned ...\n",
      "test cleaned ...\n"
     ]
    }
   ],
   "source": [
    "raw_train_df['comment_text'] = raw_train_df.comment_text.apply(lambda x: clean(x))\n",
    "print('train cleaned ...')\n",
    "\n",
    "raw_test_df['comment_text'] = raw_test_df.comment_text.apply(lambda x: clean(x))\n",
    "print('test cleaned ...')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(159571, 8)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_train_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.3. Train/Val/Test Split\n",
    "\n",
    "Next, we would separate the train dataset into a fixed train and validation set for Torchtext\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "PROCESSED_DATA_PATH = f'{DATA_PATH}processed_data/'\n",
    "os.makedirs(PROCESSED_DATA_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(127656, 31915)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# split the training data into a train and validation dataset\n",
    "train_ds, valid_ds = train_test_split(raw_train_df, test_size=0.2, random_state=9)\n",
    "len(train_ds), len(valid_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#save train, val, and test datasets \n",
    "train_ds.to_csv(f'{PROCESSED_DATA_PATH}/train_ds.csv', index=None)\n",
    "valid_ds.to_csv(f'{PROCESSED_DATA_PATH}/valid_ds.csv', index=None)\n",
    "\n",
    "#save full cleaned datasets (train+valid and test) as well\n",
    "raw_train_df.to_csv(f'{PROCESSED_DATA_PATH}/full_train_ds.csv', index=None)\n",
    "raw_test_df.to_csv(f'{PROCESSED_DATA_PATH}/test_ds.csv', index=None)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "full_train_ds.csv test_ds.csv       train_ds.csv      valid_ds.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls {PROCESSED_DATA_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.4. Tokenizing\n",
    "\n",
    "Next, we use import the `spacy` tokenizer to be used to break sentences into a list of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from torchtext import *\n",
    "from torchtext import data, vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "tokenizer = data.get_tokenizer('spacy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.5. Declaring Fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We would like the `comment_text` field to be converted to lowercase and tokenized with the `spacy` tokenizer. So we pass that to the `TEXT` Field.\n",
    "\n",
    "Since our target `LABEL` fields are already converted into a binary encoding, all we need to do is to tell the Field class that the labels are already processed. We do this by passing the `use_vocab=False` keyword to the constructor\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT = data.Field(sequential=True, tokenize=tokenizer, lower=True)\n",
    "LABEL = data.Field(sequential=False, use_vocab=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.6. Construct the Dataset\n",
    "The `TEXT` and `LABEL` Fields now know what to do when given raw data. Next, we need to tell the Fields what data it should work on. This is where we use `TabularDataset` class in TorchText.\n",
    "\n",
    "There are various built-in Datasets in torchtext that handle common data formats. For csv/tsv files, the `TabularDataset` class is convenient. Here’s how we would read data from a csv file:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "trn_data_fields = [(\"id\", None),\n",
    "                   (\"comment_text\", TEXT), ('toxic', LABEL),\n",
    "                  ('severe_toxic', LABEL), ('obscene', LABEL),\n",
    "                  ('threat', LABEL), ('insult', LABEL),\n",
    "                  ('identity_hate', LABEL),]\n",
    "\n",
    "trn, vld = data.TabularDataset.splits(path=f'{PROCESSED_DATA_PATH}',\n",
    "                                     train='train_ds.csv', validation='valid_ds.csv',\n",
    "                                     format='csv', skip_header=True, fields=trn_data_fields)\n",
    "\n",
    "full_trn = data.TabularDataset(path=f'{PROCESSED_DATA_PATH}full_train_ds.csv', \n",
    "                                     format='csv', skip_header=True, fields=trn_data_fields)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_data_fields = [(\"id\", None),\n",
    "                   (\"comment_text\", TEXT)]\n",
    "\n",
    "tst = data.TabularDataset(path=f'{PROCESSED_DATA_PATH}test_ds.csv',\n",
    "                                     format='csv', skip_header=True, fields=test_data_fields)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.7. Load pretrained vectors & Build vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glove.6B.100d.txt     glove.6B.200d.txt.pt  glove.6B.50d.txt.pt\r\n",
      "glove.6B.100d.txt.pt  glove.6B.300d.txt     wiki.en.vec\r\n",
      "glove.6B.200d.txt     glove.6B.50d.txt\r\n"
     ]
    }
   ],
   "source": [
    "VECTOR_PATH = '.vector_cache'\n",
    "!ls {VECTOR_PATH}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "pretrained_vectors = 'glove.6B.50d' \n",
    "MAX_CHARS = 20000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "For the TEXT field to convert words into integers, it needs to be told what the entire vocabulary is. To do this, we run `TEXT.build_vocab`, on the full training dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "TEXT.build_vocab(full_trn, vectors=pretrained_vectors, max_size = MAX_CHARS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's take a look at what the vocab looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('.', 514412),\n",
       " ('the', 495846),\n",
       " (',', 470871),\n",
       " ('\"', 379291),\n",
       " ('to', 297111),\n",
       " ('i', 239115),\n",
       " ('of', 224181),\n",
       " ('and', 222987),\n",
       " ('you', 219888),\n",
       " ('a', 214419)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TEXT.vocab.freqs.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 2.8. Creating the data iterator, batch wrapper & FasAI ModelData\n",
    "During training, we'll be using the `BucketIterator`.\n",
    "When we pass data into a neural network, we want the data to be padded to be the same length so that we can process them in batches:\n",
    "\n",
    "e.g. [ [3, 15, 2, 7], [4, 1], [5, 5, 6, 8, 1] ] -> [ [3, 15, 2, 7, 0], [4, 1, 0, 0, 0], [5, 5, 6, 8, 1] ]\n",
    "\n",
    "The `BucketIterator` groups sequences of similar lengths together for each batch to minimize padding. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "batch_size=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "train_iter, val_iter = data.BucketIterator.splits(\n",
    "                        (trn, vld), batch_size=batch_size,\n",
    "                        device=(0 if USE_GPU else -1), \n",
    "                        sort_key=lambda x: len(x.comment_text), # the BucketIterator needs to be told what function it should use to group the data.\n",
    "                        sort_within_batch=False, repeat=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "For the test set, we don't want the data to be shuffled. Hence we'll be using the standard `Iterator`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_iter = data.Iterator(tst, batch_size=batch_size, device=(0 if USE_GPU else -1),\n",
    "                          sort=False, sort_within_batch=False, \n",
    "                          repeat=False,train=False, shuffle=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Currently, the iterator returns a custom datatype called `torchtext.data.Batch`. This makes Torchtext hard to use with other deep learning libraries like FastAI.\n",
    "\n",
    "Hence, we'll use the [`BatchWrapper`] function to a return a tuple in the form (x, y) where x is the independent variable (the input to the model) and y is the dependent variable (the supervision data).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "class BatchWrapper():\n",
    "    def __init__(self, dataset, x_var, y_var):\n",
    "        self.dataset, self.x_var, self.y_var = dataset, x_var, y_var\n",
    "        \n",
    "    def __iter__(self):\n",
    "        for batch in self.dataset:\n",
    "            x = getattr(batch, self.x_var) # we assume only one input in this wrapper\n",
    "            \n",
    "            if self.y_var is not None: # we will concatenate y into a single tensor\n",
    "                y = torch.cat([getattr(batch, feat).unsqueeze(1) for feat in self.y_var],\n",
    "                             dim=1).float()\n",
    "            else:\n",
    "                y = V(torch.zeros((1)))\n",
    "                \n",
    "                \n",
    "            yield (x, y)\n",
    "            \n",
    "    def __len__(self):\n",
    "        return len(self.dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_column_list = ['toxic', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate']; label_column_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "train_dl = BatchWrapper(train_iter, \"comment_text\", label_column_list)\n",
    "\n",
    "val_dl = BatchWrapper(val_iter, \"comment_text\", label_column_list)\n",
    "\n",
    "test_dl = BatchWrapper(test_iter, \"comment_text\", None)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.nlp import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "model_data = ModelData(PROCESSED_DATA_PATH, trn_dl=train_dl, val_dl=val_dl, test_dl=test_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1995, 499, 2394, 20002)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(model_data.trn_dl), len(model_data.val_dl), len(model_data.test_dl), len(TEXT.vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "t, z =next(model_data.trn_dl.__iter__())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([967, 64]), torch.Size([64, 6]))"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.size(), z.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/envs/fastai-cpu/lib/python3.6/site-packages/torchtext/data/field.py:322: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.\n",
      "  return Variable(arr, volatile=not train)\n"
     ]
    }
   ],
   "source": [
    "t, z =next(model_data.test_dl.__iter__())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([327, 64]), torch.Size([1]))"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.size(), z.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training the Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import *\n",
    "import torch.nn.functional as F\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RNNModel(nn.Module):\n",
    "    \"\"\"\n",
    "    Neural Network Module with an embedding layer, a recurent module and an output linear layer\n",
    "    \n",
    "    Arguments:\n",
    "        rnn_type(str) -- type of rnn module to use options are ['LSTM', 'GRU', 'RNN_TANH', 'RNN_RELU']\n",
    "        input_size(int) -- size of the dictionary of embeddings\n",
    "        embz_size(int) -- the size of each embedding vector\n",
    "        hidden_size(int) -- the number of features in the hidden state \n",
    "        batch_size(int) -- the size of training batches \n",
    "        output_size(int) -- the number of output classes to be predicted\n",
    "        num_layers(int, optional) -- Number of recurrent layers. Default=1 \n",
    "        dropout(float, optional) -- dropout probabilty. Default=0\n",
    "        bidirectional(bool, optional) -- If True, becomes a bidirectional RNN. Default=False\n",
    "        tie_weights(bool, optional) -- if True, ties the weights of the embedding and output layer. Default=False\n",
    "    \n",
    "    Inputs: input\n",
    "        input of shape (seq_length, batch_size) -- tensor containing the features of the input sequence\n",
    "    \n",
    "    Returns: output\n",
    "        output of shape (batch_size, output_size) -- tensor containing the sigmoid activation on the \n",
    "                                                    output features h_t from the last layer of the rnn, \n",
    "                                                    for the last time-step t.\n",
    "    \"\"\"\n",
    "    \n",
    "    \n",
    "    def __init__(self, rnn_type, input_size, embz_size, hidden_size, batch_size, output_size,\n",
    "                num_layers=1, dropout=0.5, bidirectional=True, tie_weights=False):\n",
    "        super().__init__()\n",
    "        \n",
    "        if bidirectional: self.num_directions = 2\n",
    "        else: self.num_directions = 1\n",
    "\n",
    "        self.hidden_size, self.output_size, self.embz_size = hidden_size, output_size, embz_size\n",
    "        self.bidirectional, self.rnn_type, self.num_layers = bidirectional, rnn_type, num_layers\n",
    "        self.drop = nn.Dropout(dropout)\n",
    "        self.embedding_layer = nn.Embedding(input_size, embz_size)\n",
    "        self.output_layer = nn.Linear(hidden_size*self.num_directions, output_size)\n",
    "        self.init_hidden(batch_size)\n",
    "        \n",
    "        if rnn_type in ['LSTM', 'GRU']:\n",
    "            self.rnn = getattr(nn, rnn_type)(embz_size, hidden_size, num_layers=num_layers, \n",
    "                           dropout=dropout, bidirectional=bidirectional)\n",
    "        else:\n",
    "            try:\n",
    "                nonlinearity = {'RNN_TANH':'tanh', 'RNN_RELU':'relu'}[rnn_type]\n",
    "            except KeyError:\n",
    "                raise ValueError(\"\"\"An invalid option for '--rnn_type' was supplied,\n",
    "                                    options are ['LSTM', 'GRU', 'RNN_TANH', 'RNN_RELU']\"\"\")\n",
    "            self.rnn = nn.RNN(embz_size, hidden_size, num_layers=num_layers, \n",
    "                           dropout=dropout, bidirectional=bidirectional, nonlinearity=nonlinearity)    \n",
    "            \n",
    "        if tie_weights:\n",
    "            if hidden_size != embz_size:\n",
    "                raise ValueError(\"When using the tied flag, hidden size must be equal to embeddign size\")\n",
    "            elif bidirectional:\n",
    "                raise ValueError(\"When using the tied flag, set bidirectional=False\")\n",
    "            self.output_layer.weight = self.embedding_layer.weight    \n",
    "                    \n",
    "    def init_emb_weights(self, vector_weight_matrix):\n",
    "        self.embedding_layer.weight.data.copy_(vector_weight_matrix)\n",
    "        \n",
    "    def init_identity_weights(self):\n",
    "        if self.rnn_type == 'RNN_RELU':\n",
    "            self.rnn.weight_ih_l0.data.copy_(torch.eye(self.hidden_size, self.embz_size))\n",
    "            self.rnn.weight_hh_l0.data.copy_(torch.eye(self.hidden_size, self.hidden_size))\n",
    "\n",
    "            if self.bidirectional:\n",
    "                self.rnn.weight_ih_l0_reverse.data.copy_(torch.eye(self.hidden_size, self.embz_size))\n",
    "                self.rnn.weight_hh_l0_reverse.data.copy_(torch.eye(self.hidden_size, self.hidden_size))  \n",
    "        else:\n",
    "            pass\n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        if self.rnn_type == 'LSTM':\n",
    "            self.hidden = (V(torch.zeros(self.num_layers*self.num_directions, batch_size, self.hidden_size)),\n",
    "                           V(torch.zeros(self.num_layers*self.num_directions, batch_size, self.hidden_size)))\n",
    "        else:\n",
    "            self.hidden = V(torch.zeros(self.num_layers*self.num_directions, batch_size, self.hidden_size))\n",
    "                \n",
    "    def forward(self, seq):\n",
    "        batch_size = seq[0].size(0)\n",
    "        if self.hidden[0].size(1) != batch_size:\n",
    "            self.init_hidden(batch_size)\n",
    "        input_tensor = self.drop(self.embedding_layer(seq))\n",
    "        output, hidden = self.rnn(input_tensor, self.hidden)\n",
    "        self.hidden = repackage_var(hidden)\n",
    "        output = self.drop(self.output_layer(output))\n",
    "        return F.sigmoid(output[-1, :, :])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RNNModel(\n",
       "  (drop): Dropout(p=0.5)\n",
       "  (embedding_layer): Embedding(20002, 50)\n",
       "  (output_layer): Linear(in_features=100, out_features=6)\n",
       "  (rnn): GRU(50, 50, dropout=0.5, bidirectional=True)\n",
       ")"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector_weight_matrix = TEXT.vocab.vectors\n",
    "input_size = vector_weight_matrix.size(0)\n",
    "hidden_size = vector_weight_matrix.size(1)\n",
    "output_size = 6\n",
    "embz_size = vector_weight_matrix.size(1)\n",
    "batch_size = batch_size\n",
    "rnn_type = 'GRU'\n",
    "model = RNNModel(rnn_type, input_size, embz_size, hidden_size, batch_size, output_size); model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the pretrained vectors (glove.6B.50d) into the embedding layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       " 0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000\n",
       " 0.0000  0.0000  0.0000  ...   0.0000  0.0000  0.0000\n",
       " 0.1516  0.3018 -0.1676  ...  -0.3565  0.0164  0.1022\n",
       "          ...             ⋱             ...          \n",
       " 0.4058  0.2113  0.0083  ...   0.5022 -0.7418 -0.3712\n",
       " 1.0987 -0.2232  0.3466  ...  -0.3047  1.0554 -0.2385\n",
       "-0.1730 -0.2120  0.6912  ...   0.8054 -0.3160 -0.6211\n",
       "[torch.cuda.FloatTensor of size 20002x50 (GPU 0)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.init_emb_weights(vector_weight_matrix)\n",
    "model.embedding_layer.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if USE_GPU:\n",
    "    model.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### 3.2. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "MODEL_PATH = f'{DATA_PATH}models/{rnn_type}/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "os.makedirs(f'{MODEL_PATH}', exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lo = LayerOptimizer(optim.Adam, model, 1e-2, 1e-5)\n",
    "loss_func = nn.BCELoss() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b34b7daf1674a42a2507a3cd5f8245a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=16), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                               \r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: generator 'Iterator.__iter__' raised StopIteration\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   \n",
      "    0      0.371394   0.069593  \n",
      "    1      0.371476   0.062126                                 \n",
      "    2      0.368127   0.060966                                 \n",
      "    3      0.374017   0.07338                                  \n",
      "    4      0.373226   0.062801                                 \n",
      "    5      0.367177   0.059734                                 \n",
      "    6      0.364643   0.055366                                 \n",
      "    7      0.37301    0.069894                                 \n",
      "    8      0.372339   0.068181                                 \n",
      "    9      0.373894   0.059369                                 \n",
      "    10     0.374497   0.063353                                 \n",
      "    11     0.370822   0.063703                                 \n",
      "    12     0.37109    0.060205                                 \n",
      "    13     0.366658   0.056104                                 \n",
      "    14     0.367601   0.053094                                 \n",
      "    15     0.372098   0.061729                                 \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([ 0.06173])]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "on_end = lambda sched, cycle: save_model(model, f'{MODEL_PATH}cyc_{cycle}')\n",
    "cb = [CosAnneal(lo, len(model_data.trn_dl), cycle_mult=2, on_cycle_end=on_end)]\n",
    "fit(model, model_data, 2**4, lo.opt, loss_func, callbacks=cb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 4. Making Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### 4.1. Generate predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2394/2394 [01:21<00:00, 29.32it/s]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/paperspace/anaconda3/envs/fastai/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: generator 'Iterator.__iter__' raised StopIteration\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "test_pred = []\n",
    "\n",
    "for x, y in tqdm(model_data.test_dl):\n",
    "    preds = model(VV(x))\n",
    "    preds = preds.data.cpu().numpy()\n",
    "    test_pred.append(preds)\n",
    "\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "test_preds = np.concatenate(test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(153164, 6)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 4.2. Make submission on Kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(153164, 2)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_ds = pd.read_csv(f'{PROCESSED_DATA_PATH}test_ds.csv', low_memory=False); test_ds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "for i , col in enumerate(label_column_list):\n",
    "    test_ds[col] = test_preds[:, i]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00001cee341fdb12</td>\n",
       "      <td>Yo bitch Ja Rule is more succesful then you wi...</td>\n",
       "      <td>0.686830</td>\n",
       "      <td>0.123798</td>\n",
       "      <td>0.679709</td>\n",
       "      <td>0.154812</td>\n",
       "      <td>0.458549</td>\n",
       "      <td>0.362866</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0000247867823ef7</td>\n",
       "      <td>== From RfC == The title is fine as it is, IMO.</td>\n",
       "      <td>0.005412</td>\n",
       "      <td>0.000691</td>\n",
       "      <td>0.003433</td>\n",
       "      <td>0.002556</td>\n",
       "      <td>0.003729</td>\n",
       "      <td>0.006844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00013b17ad220c46</td>\n",
       "      <td>\" == Sources == * Zawe Ashton on Lapland — / \"</td>\n",
       "      <td>0.005122</td>\n",
       "      <td>0.000674</td>\n",
       "      <td>0.003221</td>\n",
       "      <td>0.002630</td>\n",
       "      <td>0.003585</td>\n",
       "      <td>0.007108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00017563c3f7919a</td>\n",
       "      <td>:If you have a look back at the source, the in...</td>\n",
       "      <td>0.003710</td>\n",
       "      <td>0.000580</td>\n",
       "      <td>0.002605</td>\n",
       "      <td>0.002121</td>\n",
       "      <td>0.002988</td>\n",
       "      <td>0.005025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00017695ad8997eb</td>\n",
       "      <td>I do not anonymously edit articles at all.</td>\n",
       "      <td>0.005053</td>\n",
       "      <td>0.000671</td>\n",
       "      <td>0.003246</td>\n",
       "      <td>0.002612</td>\n",
       "      <td>0.003537</td>\n",
       "      <td>0.006941</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id                                       comment_text  \\\n",
       "0  00001cee341fdb12  Yo bitch Ja Rule is more succesful then you wi...   \n",
       "1  0000247867823ef7    == From RfC == The title is fine as it is, IMO.   \n",
       "2  00013b17ad220c46     \" == Sources == * Zawe Ashton on Lapland — / \"   \n",
       "3  00017563c3f7919a  :If you have a look back at the source, the in...   \n",
       "4  00017695ad8997eb         I do not anonymously edit articles at all.   \n",
       "\n",
       "      toxic  severe_toxic   obscene    threat    insult  identity_hate  \n",
       "0  0.686830      0.123798  0.679709  0.154812  0.458549       0.362866  \n",
       "1  0.005412      0.000691  0.003433  0.002556  0.003729       0.006844  \n",
       "2  0.005122      0.000674  0.003221  0.002630  0.003585       0.007108  \n",
       "3  0.003710      0.000580  0.002605  0.002121  0.002988       0.005025  \n",
       "4  0.005053      0.000671  0.003246  0.002612  0.003537       0.006941  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_ds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "SUBMISSION_PATH = f'{DATA_PATH}submission/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "os.makedirs(SUBMISSION_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "#writes the submission file to disk\n",
    "test_ds.drop(\"comment_text\", axis=1).to_csv(f'{SUBMISSION_PATH}submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9517"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#submits predictions to kaggle and returns public score\n",
    "!kg submit {SUBMISSION_PATH}submission.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### 4.3. Looking at example predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def to_label(x, threshold=0.5):\n",
    "    if x > threshold:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for i in range(len(label_column_list)):\n",
    "    test_ds[label_column_list[i]] = test_ds[label_column_list[i]].apply(to_label, threshold=0.45)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00001cee341fdb12</td>\n",
       "      <td>Yo bitch Ja Rule is more succesful then you wi...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0000247867823ef7</td>\n",
       "      <td>== From RfC == The title is fine as it is, IMO.</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00013b17ad220c46</td>\n",
       "      <td>\" == Sources == * Zawe Ashton on Lapland — / \"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00017563c3f7919a</td>\n",
       "      <td>:If you have a look back at the source, the in...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00017695ad8997eb</td>\n",
       "      <td>I do not anonymously edit articles at all.</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id                                       comment_text  toxic  \\\n",
       "0  00001cee341fdb12  Yo bitch Ja Rule is more succesful then you wi...      1   \n",
       "1  0000247867823ef7    == From RfC == The title is fine as it is, IMO.      0   \n",
       "2  00013b17ad220c46     \" == Sources == * Zawe Ashton on Lapland — / \"      0   \n",
       "3  00017563c3f7919a  :If you have a look back at the source, the in...      0   \n",
       "4  00017695ad8997eb         I do not anonymously edit articles at all.      0   \n",
       "\n",
       "   severe_toxic  obscene  threat  insult  identity_hate  \n",
       "0             0        1       0       1              0  \n",
       "1             0        0       0       0              0  \n",
       "2             0        0       0       0              0  \n",
       "3             0        0       0       0              0  \n",
       "4             0        0       0       0              0  "
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_ds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TOXIC\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'schumaker is a dick!'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SEVERE_TOXIC\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==benon!! ==what benon does at night with fucking curps there both nigger shitty asholes who fuck eachother==[[image:Doggy Sex Position.png|benon'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OBSCENE\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'WHO THE FUCK CARES, SHIT LIKE THIS IS WHY NOBODY TAKES WIKIPEDIA SERIOUSLY'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "THREAT\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DIE OF FACE CANCER AND DIE POOR. WIKIPEDIA IS INHABITED BY FAT CUNTS WHO FUCKING WASTE THEIR LIVES BEING FAT LOSERS FUCK YOU ALL DI'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INSULT\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'fuck you'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IDENTITY_HATE\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Like Rape Words And Bad Gay Jokes Meen Really like'thats what she say'He Say Weird Ass Noise Is Normal And Penis So PewDiePie Please Why Use Words LIKE THAT MAN D\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in label_column_list:\n",
    "    print(i.upper())\n",
    "    display(test_ds[getattr(test_ds_, i) == 1].sample().iloc[0]['comment_text'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "':Roll Grandiose in. Ultimately this is for fun and to encourage quality editing, and that goal is achieved with either 64 or 65 participants!'"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_ds_[(test_ds.toxic==0) & (test_ds.severe_toxic==0) & \n",
    "        (test_ds.obscene==0) & (test_ds.threat==0) &\n",
    "        (test_ds.insult==0) & (test_ds.identity_hate==0)].sample().iloc[0]['comment_text']   \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}