classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checking the LAPD's crime classifications\n",
    "\n",
    "The Times analyzed Los Angeles Police Department violent crime data from 2005 to 2012. Our analysis found that the Los Angeles Police Department misclassified an estimated 14,000 serious assaults as minor offenses, artificially lowering the city’s crime levels.\n",
    "\n",
    "To conduct the analysis, The Times used an algorithm that combined two machine learning classifiers. Each classifier read in a brief description of the crime, which it used to determine if it was a minor or serious assault. You can see a [sample of the data here](https://github.com/datadesk/lapd-crime-classification-analysis/blob/master/training_data.csv). An example of a minor assault reads: \"VICTS AND SUSPS BECAME INV IN VERBA ARGUMENT SUSP THEN BEGAN HITTING VICTS IN THE FACE.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "import nltk\n",
    "from nltk.util import ngrams\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.pipeline import Pipeline\n",
    "from nltk.classify import MaxentClassifier\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.classify.scikitlearn import SklearnClassifier\n",
    "from sklearn.feature_extraction.text import TfidfTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stemming and stop words\n",
    "\n",
    "We're going to clean up the crime descriptions in two steps. First, we're going to [stem](https://en.wikipedia.org/wiki/Stemming) the words -- this reduces the words to their root in order to limit differences based on tense or whether they appear in the plural or possessive form. Then, we're going to strip out a custom list of [stop words](https://en.wikipedia.org/wiki/Stop_words)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define a standard snowball stemmer\n",
    "STEMMER = SnowballStemmer('english')\n",
    "# Make a list of stopwords, including the stemmed versions\n",
    "# These are words that have no impact on the classification, and\n",
    "# can even occasionally mess up the classifier.\n",
    "STOPWORDS = [\n",
    "    'susp',\n",
    "    'susps',\n",
    "    's',\n",
    "    'v',\n",
    "    'in',\n",
    "    'ppa',\n",
    "    'vict',\n",
    "    'the',\n",
    "    'and',\n",
    "    '&',\n",
    "    '-s',\n",
    "    'after',\n",
    "    'for',\n",
    "    'heard',\n",
    "    'second',\n",
    "    'avoid',\n",
    "    'hold',\n",
    "    'holding',\n",
    "    'retrieved',\n",
    "    'battery',\n",
    "    'fist',\n",
    "    'of',\n",
    "    'to',\n",
    "    'a',\n",
    "]\n",
    "STOPWORDS += [STEMMER.stem(i) for i in STOPWORDS]\n",
    "STOPWORDS = list(set(STOPWORDS))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize\n",
    "\n",
    "This is a function to take a description and break it up into the individual \"features\" we're going to use to classify it. We separate the description into individual words, then stem them and remove stop words. From there, we make a list of individual words and then combine them into [bigrams](https://en.wikipedia.org/wiki/Bigram)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenize(description):\n",
    "    \"\"\"\n",
    "    Takes LAPD description text, strips out unwanted words and text,\n",
    "    and prepares it for the trainer.\n",
    "    \"\"\"\n",
    "    # first lower case and strip leading/trailing whitespace\n",
    "    description = description.lower().strip()\n",
    "    # kill the 'do-'s and any stray punctuation\n",
    "    description = description.replace('do-', '').replace('.', '').replace(',', '')\n",
    "    # make a list of words by splitting on whitespace\n",
    "    words = description.split(' ')\n",
    "    # Make sure each \"word\" is a real string / account for odd whitespace\n",
    "    words = [STEMMER.stem(i) for i in words if i]\n",
    "    words = [i for i in words if i not in STOPWORDS]\n",
    "    # let's see if adding bigrams improves the accuracy\n",
    "    bigrams = ngrams(words, 2)\n",
    "    bigrams = [\"%s|%s\" % (i[0], i[1]) for i in bigrams]\n",
    "    # bigrams = [i for i in bigrams if STEMMED_BIGRAMS.get(i)]\n",
    "    # set up a dict\n",
    "    out_dict = dict([(i, True) for i in words + bigrams])\n",
    "    # The NLTK trainer expects data in a certain format\n",
    "    return out_dict\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grab the features\n",
    "\n",
    "Loop through our example CSV and grab the features we're going to use to train our classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# open our sample file and use the CSV module to parse it\n",
    "f = open('training_data.csv', 'rU')\n",
    "data = list(csv.DictReader(f))\n",
    "# Make an empty list for our processed data\n",
    "features = []\n",
    "# Loop through all the lines in the CSV\n",
    "for i in data:\n",
    "    description = i.get('NARRATIVE')\n",
    "    classification = i.get('classification')\n",
    "    feats = tokenize(description)\n",
    "    features.append((feats, classification))\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "({u'kick|polic': True, u'use': True, u'his': True, u'leg': True, u'polic': True, u'under|arrest': True, u'right|leg': True, u'place|under': True, u'back': True, u'sergeant|back': True, u'arrest': True, u'right': True, u'place': True, u'sergeant': True, u'use|his': True, u'under': True, u'his|right': True, u'arrest|use': True, u'leg|kick': True, u'kick': True, u'polic|sergeant': True}, 'minor')\n"
     ]
    }
   ],
   "source": [
    "# Here's what this looks like\n",
    "print features[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the classifiers\n",
    "\n",
    "For this analysis we used two machine learning classifiers. The first is a linear [support vector machine](http://nlp.stanford.edu/IR-book/html/htmledition/support-vector-machines-the-linearly-separable-case-1.html) from the stellar [scikit-learn Python library](http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html). The second is a [maximum entropy classifier](http://www.nltk.org/book/ch06.html#maximum-entropy-classifiers). For the official analysis I used the [MegaM](http://www.umiacs.umd.edu/~hal/megam/) optimization package to dramatically improve the training speed. Here, for simplicity, I'm using the NLTK built in trainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<SklearnClassifier(Pipeline(steps=[('tfidf', TfidfTransformer(norm=u'l2', smooth_idf=True, sublinear_tf=False,\n",
       "         use_idf=True)), ('linearsvc', LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
       "     intercept_scaling=1, loss='squared_hinge', max_iter=1000,\n",
       "     multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,\n",
       "     verbose=0))]))>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train our classifiers. Let's start with Linear SVC\n",
    "# Make a data prep pipeline\n",
    "pipeline = Pipeline([\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('linearsvc', LinearSVC()),\n",
    "])\n",
    "# make the classifier\n",
    "linear_svc = SklearnClassifier(pipeline)\n",
    "# Train it\n",
    "linear_svc.train(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ==> Training (100 iterations)\n",
      "\n",
      "      Iteration    Log Likelihood    Accuracy\n",
      "      ---------------------------------------\n",
      "             1          -0.69315        0.500\n",
      "             2          -0.43483        0.970\n",
      "             3          -0.32266        0.990\n",
      "             4          -0.25840        0.990\n",
      "             5          -0.21623        1.000\n",
      "             6          -0.18624        1.000\n",
      "             7          -0.16375        1.000\n",
      "             8          -0.14621        1.000\n",
      "             9          -0.13215        1.000\n",
      "            10          -0.12061        1.000\n",
      "            11          -0.11097        1.000\n",
      "            12          -0.10279        1.000\n",
      "            13          -0.09577        1.000\n",
      "            14          -0.08967        1.000\n",
      "            15          -0.08432        1.000\n",
      "            16          -0.07959        1.000\n",
      "            17          -0.07538        1.000\n",
      "            18          -0.07161        1.000\n",
      "            19          -0.06821        1.000\n",
      "            20          -0.06513        1.000\n",
      "            21          -0.06232        1.000\n",
      "            22          -0.05976        1.000\n",
      "            23          -0.05741        1.000\n",
      "            24          -0.05524        1.000\n",
      "            25          -0.05324        1.000\n",
      "            26          -0.05139        1.000\n",
      "            27          -0.04966        1.000\n",
      "            28          -0.04805        1.000\n",
      "            29          -0.04655        1.000\n",
      "            30          -0.04514        1.000\n",
      "            31          -0.04383        1.000\n",
      "            32          -0.04258        1.000\n",
      "            33          -0.04142        1.000\n",
      "            34          -0.04031        1.000\n",
      "            35          -0.03927        1.000\n",
      "            36          -0.03828        1.000\n",
      "            37          -0.03735        1.000\n",
      "            38          -0.03646        1.000\n",
      "            39          -0.03561        1.000\n",
      "            40          -0.03481        1.000\n",
      "            41          -0.03404        1.000\n",
      "            42          -0.03331        1.000\n",
      "            43          -0.03261        1.000\n",
      "            44          -0.03194        1.000\n",
      "            45          -0.03130        1.000\n",
      "            46          -0.03069        1.000\n",
      "            47          -0.03010        1.000\n",
      "            48          -0.02953        1.000\n",
      "            49          -0.02899        1.000\n",
      "            50          -0.02847        1.000\n",
      "            51          -0.02797        1.000\n",
      "            52          -0.02748        1.000\n",
      "            53          -0.02702        1.000\n",
      "            54          -0.02657        1.000\n",
      "            55          -0.02613        1.000\n",
      "            56          -0.02571        1.000\n",
      "            57          -0.02531        1.000\n",
      "            58          -0.02492        1.000\n",
      "            59          -0.02454        1.000\n",
      "            60          -0.02417        1.000\n",
      "            61          -0.02382        1.000\n",
      "            62          -0.02347        1.000\n",
      "            63          -0.02314        1.000\n",
      "            64          -0.02281        1.000\n",
      "            65          -0.02250        1.000\n",
      "            66          -0.02219        1.000\n",
      "            67          -0.02190        1.000\n",
      "            68          -0.02161        1.000\n",
      "            69          -0.02133        1.000\n",
      "            70          -0.02106        1.000\n",
      "            71          -0.02079        1.000\n",
      "            72          -0.02053        1.000\n",
      "            73          -0.02028        1.000\n",
      "            74          -0.02004        1.000\n",
      "            75          -0.01980        1.000\n",
      "            76          -0.01957        1.000\n",
      "            77          -0.01934        1.000\n",
      "            78          -0.01912        1.000\n",
      "            79          -0.01890        1.000\n",
      "            80          -0.01869        1.000\n",
      "            81          -0.01849        1.000\n",
      "            82          -0.01828        1.000\n",
      "            83          -0.01809        1.000\n",
      "            84          -0.01790        1.000\n",
      "            85          -0.01771        1.000\n",
      "            86          -0.01753        1.000\n",
      "            87          -0.01735        1.000\n",
      "            88          -0.01717        1.000\n",
      "            89          -0.01700        1.000\n",
      "            90          -0.01683        1.000\n",
      "            91          -0.01667        1.000\n",
      "            92          -0.01650        1.000\n",
      "            93          -0.01635        1.000\n",
      "            94          -0.01619        1.000\n",
      "            95          -0.01604        1.000\n",
      "            96          -0.01589        1.000\n",
      "            97          -0.01575        1.000\n",
      "            98          -0.01560        1.000\n",
      "            99          -0.01546        1.000\n",
      "         Final          -0.01532        1.000\n"
     ]
    }
   ],
   "source": [
    "# Next, let's do the Maximum Entropy\n",
    "maxent = MaxentClassifier.train(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the classifiers\n",
    "\n",
    "Now let's test these out! For this example we're only using a training sample of 100 crimes, which is not going to produce very accurate results. For our official analysis, we used a training sample of more than 20,000 crimes we reviewed as part of a previous story in 2014. We also chose to use two classifiers because, though they agreed on the vast majority of crimes, each classifier did a better job with some edge cases we didn't want to miss. You can check out the results below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "correct: minor | maxent: serious | linear svc: serious\n",
      "correct: minor | maxent: minor | linear svc: minor\n",
      "correct: minor | maxent: serious | linear svc: serious\n",
      "correct: serious | maxent: serious | linear svc: serious\n",
      "correct: serious | maxent: serious | linear svc: serious\n",
      "correct: minor | maxent: minor | linear svc: minor\n",
      "correct: minor | maxent: minor | linear svc: minor\n",
      "correct: minor | maxent: serious | linear svc: serious\n",
      "correct: serious | maxent: serious | linear svc: serious\n",
      "correct: minor | maxent: minor | linear svc: minor\n",
      "correct: minor | maxent: minor | linear svc: minor\n"
     ]
    }
   ],
   "source": [
    "# Now, let's try these out\n",
    "test_data = list(csv.DictReader(open('test_data.csv', 'rU')))\n",
    "for i in test_data:\n",
    "    description = i.get('NARRATIVE')\n",
    "    classification = i.get('classification')\n",
    "    toks = tokenize(description)\n",
    "    # now grab the results of our classifiers\n",
    "    maxent_class = maxent.classify(toks)\n",
    "    svc_class = linear_svc.classify(toks)\n",
    "    print('correct: %s | maxent: %s | linear svc: %s' % (classification, maxent_class, svc_class))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}