part-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d9174bd-1d24-495b-98a7-28d444f42574",
   "metadata": {},
   "source": [
    "# Part 1\n",
    "\n",
    "In this part we discuss the building blocks of spaCy. It helps to show these things before applying spaCy to some real data.\n",
    "\n",
    "## Prep\n",
    "\n",
    "First we'll need a video that shows how we boot up a new virtualenv with spaCy and JupyterLab. \n",
    "\n",
    "In particular, we'll need folks to run this command beforehand. \n",
    "\n",
    "```\n",
    "python -m spacy download en_core_web_md\n",
    "```\n",
    "\n",
    "Once that's taken care of, we can go to the next vid. \n",
    "\n",
    "## Tokens\n",
    "\n",
    "The goal here is to show that you can load a spaCy model into an `nlp` variable that can then turn text into a `Doc` object. This `Doc` object has tokens with properties that the `nlp`-model has predicted for you. Here's one example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b0d0da58-0668-4614-a2a5-65ccc154b302",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy \n",
    "\n",
    "nlp = spacy.load(\"en_core_web_md\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f6b37427-5154-4011-9851-683426c2d59c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hi INTJ\n",
      ", PUNCT\n",
      "my PRON\n",
      "name NOUN\n",
      "is AUX\n",
      "Vincent PROPN\n",
      ". PUNCT\n",
      "I PRON\n",
      "like VERB\n",
      "to PART\n",
      "write VERB\n",
      "Python PROPN\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is Vincent. I like to write Python\")\n",
    "for token in doc:\n",
    "    print(token, token.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056d1f4e-4176-4ead-bb65-e232fcd3b647",
   "metadata": {},
   "source": [
    "You might be tempted to think that a token is a word. But that's not exactly true. Notice how in the previous example the punctuation is also a token? Here's another interesting example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "abff4a4f-a866-48c6-a877-d0941d014e61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python\n",
      "is\n",
      "n't\n",
      "just\n",
      "a\n",
      "language\n",
      ",\n",
      "it\n",
      "'s\n",
      "a\n",
      "community\n",
      "!\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Python isn't just a language, it's a community!\")\n",
    "for token in doc:\n",
    "    print(token)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "537e3e0e-adfa-4300-8c96-0b6ad004e33e",
   "metadata": {},
   "source": [
    "Notice how `n't` and `'s` are tokens here? It makes sense when you consider that `n't` basically means `not` and `'s` implies `is`. spaCy constructs these tokens internally using rules that depend on a language. These rules are different in English than they are for Dutch, just to name one example.\n",
    "\n",
    "We won't focus on this too much, but it's good to keep in mind that a token doesn't always imply a word.\n",
    "\n",
    "## More Token Properties\n",
    "\n",
    "I've made a function that can highlight some more properties that spaCy provides in a neat table format. This is a subset of all the properties that spaCy can calculate (mention [these docs](https://spacy.io/usage/linguistic-features#pos-tagging)). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d684db97-de2b-4660-afcd-c44dfdc8ab79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "text        lemma       pos     ent      shape   punct   morph                         \n",
      "---------   ---------   -----   ------   -----   -----   ------------------------------\n",
      "Hello       hello       INTJ             Xxxxx   False                                 \n",
      "internet    internet    NOUN             xxxx    False   Number=Sing                   \n",
      ".           .           PUNCT            .       True    PunctType=Peri                \n",
      "My          my          PRON             Xx      False   Number=Sing|Person=1|Poss=Yes|PronType=Prs\n",
      "name        name        NOUN             xxxx    False   Number=Sing                   \n",
      "is          be          AUX              xx      False   Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin\n",
      "Vincent     Vincent     PROPN   PERSON   Xxxxx   False   Number=Sing                   \n",
      "Warmerdam   Warmerdam   PROPN   PERSON   Xxxxx   False   Number=Sing                   \n",
      ".           .           PUNCT            .       True    PunctType=Peri                \n",
      "I           I           PRON             X       False   Case=Nom|Number=Sing|Person=1|PronType=Prs\n",
      "like        like        VERB             xxxx    False   Tense=Pres|VerbForm=Fin       \n",
      "to          to          PART             xx      False                                 \n",
      "write       write       VERB             xxxx    False   VerbForm=Inf                  \n",
      "Python      Python      PROPN            Xxxxx   False   Number=Sing                   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from wasabi import table\n",
    "\n",
    "def text_to_doctable(txt):\n",
    "    doc = nlp(txt)\n",
    "    header = (\"text\", \"lemma\", \"pos\", \"ent\", \"shape\", \"punct\", \"morph\")\n",
    "    data = [(tok.text, tok.lemma_, tok.pos_, tok.ent_type_, tok.shape_, tok.is_punct, tok.morph) for tok in doc]\n",
    "    formatted = table(data, header=header, divider=True)\n",
    "    print(formatted)\n",
    "\n",
    "text_to_doctable(\"Hello internet. My name is Vincent Warmerdam. I like to write Python\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0525b60-f2d7-4d29-9cef-cc88876422dc",
   "metadata": {},
   "source": [
    "In particular, one of the most useful properties that spaCy provides here is the \"named entity\". Notice that `Vincent` is a PERSON? This property is attached on the token, but we can also get it from the `doc`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f3629c47-2cbd-43d6-b2b3-744152358e53",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vincent Warmerdam PERSON\n",
      "Amsterdam GPE\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is Vincent Warmerdam. I live near Amsterdam and I like to write Python\")\n",
    "for ent in doc.ents:\n",
    "    print(ent, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b735c7aa-e6a5-4aec-b18c-f4c3cb262cab",
   "metadata": {},
   "source": [
    "Detecting these entities is something that spaCy is pretty renown for. These models have been trained on large datasets and can generalize to a bunch of use-cases."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2c714a5-cccf-4828-9467-7ba340e24c75",
   "metadata": {},
   "source": [
    "## displaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "079f14b3-d01a-49de-89ec-66c2eea12368",
   "metadata": {},
   "source": [
    "There's loads more stuff that spaCy can detect for you, like the grammatical dependencies. Showing that is easiest with the internal `diplacy` tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "11cdf7a1-14c9-4a68-bbe3-5bf5f037439a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"6459616cb23d43959b2eaef1aaf204d8-0\" class=\"displacy\" width=\"1800\" height=\"312.0\" direction=\"ltr\" style=\"max-width: none; height: 312.0px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Hi,</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">INTJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">my</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">name</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">Vincent.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">I</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">PRON</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">like</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1275\">to</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1275\">PART</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1450\">write</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1450\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"222.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1625\">Python</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1625\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-0\" stroke-width=\"2px\" d=\"M70,177.0 C70,2.0 575.0,2.0 575.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">intj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,179.0 L62,167.0 78,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-1\" stroke-width=\"2px\" d=\"M245,177.0 C245,89.5 395.0,89.5 395.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">poss</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M245,179.0 L237,167.0 253,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-2\" stroke-width=\"2px\" d=\"M420,177.0 C420,89.5 570.0,89.5 570.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,179.0 L412,167.0 428,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-3\" stroke-width=\"2px\" d=\"M595,177.0 C595,89.5 745.0,89.5 745.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">attr</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M745.0,179.0 L753.0,167.0 737.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-4\" stroke-width=\"2px\" d=\"M945,177.0 C945,89.5 1095.0,89.5 1095.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M945,179.0 L937,167.0 953,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-5\" stroke-width=\"2px\" d=\"M1295,177.0 C1295,89.5 1445.0,89.5 1445.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">aux</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1295,179.0 L1287,167.0 1303,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-6\" stroke-width=\"2px\" d=\"M1120,177.0 C1120,2.0 1450.0,2.0 1450.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">xcomp</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1450.0,179.0 L1458.0,167.0 1442.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-6459616cb23d43959b2eaef1aaf204d8-0-7\" stroke-width=\"2px\" d=\"M1470,177.0 C1470,89.5 1620.0,89.5 1620.0,177.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-6459616cb23d43959b2eaef1aaf204d8-0-7\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1620.0,179.0 L1628.0,167.0 1612.0,167.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy \n",
    "\n",
    "doc = nlp(\"Hi, my name is Vincent. I like to write Python\")\n",
    "displacy.render(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27382df0-986a-47de-a931-16e49025aa60",
   "metadata": {},
   "source": [
    "This tool can also be used to display entities nicely. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "ecf91f36-8a45-402d-bd58-222b4c6e9c84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Hi, my name is \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Vincent\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ". I like to write Python</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is Vincent. I like to write Python\")\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73b9aa2c-3c47-482f-bbbe-861c343ffb89",
   "metadata": {},
   "source": [
    "One nice thing about this visual is that it's easy to show that a `PERSON` entity can contain more than one token. So let's take this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7486e88b-7928-4bd4-959a-9d06c324e5ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Hi, my name is \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Vincent Warmerdam\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ". I like to write Python</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is Vincent Warmerdam. I like to write Python\")\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae270512-14bd-4a85-b915-0b32bd4957dd",
   "metadata": {},
   "source": [
    "One thing to remember is that our `nlp` model is a statistical model. So it won't be perfect, especially when the input text is different from the training data that spaCy used. One particular concequence of this is that capitalisation can matter a lot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e9a76773-056d-46f7-8916-b65208d15aff",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/codespace/.python/current/lib/python3.10/site-packages/spacy/displacy/__init__.py:213: UserWarning: [W006] No entities to visualize found in Doc object. If this is surprising to you, make sure the Doc was processed using a model that supports named entity recognition, and check the `doc.ents` property manually if necessary.\n",
      "  warnings.warn(Warnings.W006)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Hi, my name is vincent. I like to write Python</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is vincent. I like to write Python\")\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a8800a2-ed4b-4fc7-a358-e83877e4b125",
   "metadata": {},
   "source": [
    "## Document properties."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8c8e6fb-b42e-4b91-a97e-df32c4a6b1d8",
   "metadata": {},
   "source": [
    "Sofar we've mainly shown properties on tokens. These are super useful, but there's also properties on the actual `Doc` itself that can be useful. The first is to show the sentences in the doc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ecda0f04-8cec-48b3-aa4e-bb2f2c00a60c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Hi, my name is Vincent., I like to write Python]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is Vincent. I like to write Python\")\n",
    "list(doc.sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "204c021d-993d-472f-bc95-c1186a79db7d",
   "metadata": {},
   "source": [
    "Similarily, you can also ask for all the noun chunks in a document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "41e1d146-ecdb-40ed-9a00-e2ae2fcdb391",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Star Wars, a very popular science fiction series]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(\"Star Wars is a very popular science fiction series.\")\n",
    "list(doc.noun_chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef8a6fa0-9dfb-4c9b-a4c9-951a0f0ed4e3",
   "metadata": {},
   "source": [
    "Finally, you can also turn a `Doc` into a JSON representation. This is super useful if you want to expose the output of a spaCy model as an API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "56d5dade-c588-44c5-b3bd-e172d261e1e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'Hi, my name is Vincent. I like to write Python',\n",
       " 'ents': [{'start': 15, 'end': 22, 'label': 'PERSON'}],\n",
       " 'sents': [{'start': 0, 'end': 23}, {'start': 24, 'end': 46}],\n",
       " 'tokens': [{'id': 0,\n",
       "   'start': 0,\n",
       "   'end': 2,\n",
       "   'tag': 'UH',\n",
       "   'pos': 'INTJ',\n",
       "   'morph': '',\n",
       "   'lemma': 'hi',\n",
       "   'dep': 'intj',\n",
       "   'head': 4},\n",
       "  {'id': 1,\n",
       "   'start': 2,\n",
       "   'end': 3,\n",
       "   'tag': ',',\n",
       "   'pos': 'PUNCT',\n",
       "   'morph': 'PunctType=Comm',\n",
       "   'lemma': ',',\n",
       "   'dep': 'punct',\n",
       "   'head': 4},\n",
       "  {'id': 2,\n",
       "   'start': 4,\n",
       "   'end': 6,\n",
       "   'tag': 'PRP$',\n",
       "   'pos': 'PRON',\n",
       "   'morph': 'Number=Sing|Person=1|Poss=Yes|PronType=Prs',\n",
       "   'lemma': 'my',\n",
       "   'dep': 'poss',\n",
       "   'head': 3},\n",
       "  {'id': 3,\n",
       "   'start': 7,\n",
       "   'end': 11,\n",
       "   'tag': 'NN',\n",
       "   'pos': 'NOUN',\n",
       "   'morph': 'Number=Sing',\n",
       "   'lemma': 'name',\n",
       "   'dep': 'nsubj',\n",
       "   'head': 4},\n",
       "  {'id': 4,\n",
       "   'start': 12,\n",
       "   'end': 14,\n",
       "   'tag': 'VBZ',\n",
       "   'pos': 'AUX',\n",
       "   'morph': 'Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin',\n",
       "   'lemma': 'be',\n",
       "   'dep': 'ROOT',\n",
       "   'head': 4},\n",
       "  {'id': 5,\n",
       "   'start': 15,\n",
       "   'end': 22,\n",
       "   'tag': 'NNP',\n",
       "   'pos': 'PROPN',\n",
       "   'morph': 'Number=Sing',\n",
       "   'lemma': 'Vincent',\n",
       "   'dep': 'attr',\n",
       "   'head': 4},\n",
       "  {'id': 6,\n",
       "   'start': 22,\n",
       "   'end': 23,\n",
       "   'tag': '.',\n",
       "   'pos': 'PUNCT',\n",
       "   'morph': 'PunctType=Peri',\n",
       "   'lemma': '.',\n",
       "   'dep': 'punct',\n",
       "   'head': 4},\n",
       "  {'id': 7,\n",
       "   'start': 24,\n",
       "   'end': 25,\n",
       "   'tag': 'PRP',\n",
       "   'pos': 'PRON',\n",
       "   'morph': 'Case=Nom|Number=Sing|Person=1|PronType=Prs',\n",
       "   'lemma': 'I',\n",
       "   'dep': 'nsubj',\n",
       "   'head': 8},\n",
       "  {'id': 8,\n",
       "   'start': 26,\n",
       "   'end': 30,\n",
       "   'tag': 'VBP',\n",
       "   'pos': 'VERB',\n",
       "   'morph': 'Tense=Pres|VerbForm=Fin',\n",
       "   'lemma': 'like',\n",
       "   'dep': 'ROOT',\n",
       "   'head': 8},\n",
       "  {'id': 9,\n",
       "   'start': 31,\n",
       "   'end': 33,\n",
       "   'tag': 'TO',\n",
       "   'pos': 'PART',\n",
       "   'morph': '',\n",
       "   'lemma': 'to',\n",
       "   'dep': 'aux',\n",
       "   'head': 10},\n",
       "  {'id': 10,\n",
       "   'start': 34,\n",
       "   'end': 39,\n",
       "   'tag': 'VB',\n",
       "   'pos': 'VERB',\n",
       "   'morph': 'VerbForm=Inf',\n",
       "   'lemma': 'write',\n",
       "   'dep': 'xcomp',\n",
       "   'head': 8},\n",
       "  {'id': 11,\n",
       "   'start': 40,\n",
       "   'end': 46,\n",
       "   'tag': 'NNP',\n",
       "   'pos': 'PROPN',\n",
       "   'morph': 'Number=Sing',\n",
       "   'lemma': 'Python',\n",
       "   'dep': 'dobj',\n",
       "   'head': 10}]}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(\"Hi, my name is Vincent. I like to write Python\")\n",
    "doc.to_json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3b8df99-2e90-4c6e-ae02-97f91288edc6",
   "metadata": {},
   "source": [
    "There are a bunch more too, but I wanted to mention these because they tend to be pretty useful in my day to day work."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "76dc8a81-fb38-4124-9048-de8c3925d5b0",
   "metadata": {},
   "source": [
    "## Data Structures \n",
    "\n",
    "Sofar we've seen three main data structures in spaCy, but it might be good to take the time to make that explicit. \n",
    "\n",
    "1. Document. Collection of tokens and estimated properties.\n",
    "2. Token. Seperate tokens of text that make up the document.\n",
    "3. Span. Sequence of tokens. Typically because they are part of a sentence, noun chunk or entity. Belongs to a document, is made up of tokens. \n",
    "\n",
    "This will be explained by making doodles, as well as this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a6a6c72e-d981-46fe-9d2f-681861d5b31c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Hi. My name is Vincent., spacy.tokens.doc.Doc)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(\"Hi. My name is Vincent.\")\n",
    "doc, type(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "05dbc411-dbae-4371-8821-c3a6926b3fe7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Hi, spacy.tokens.token.Token)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc[0], type(doc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "82b0c93e-95c7-46b4-8019-2eee55b25eaa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Hi., spacy.tokens.span.Span)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc[:2], type(doc[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4850262-1767-4bba-8ee2-e7e31fc900b6",
   "metadata": {},
   "source": [
    "Note that spans can also contain a single token. But they are still a span. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "25934529-59b9-416f-a810-c61365530c03",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vincent <class 'spacy.tokens.span.Span'>\n"
     ]
    }
   ],
   "source": [
    "for ent in doc.ents:\n",
    "    print(ent, type(ent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "685a2ad9-38c2-45e4-a6e0-6f7ea4a6621c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Hi, ., My, name, is, Vincent, .]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[tok for tok in doc]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9fba28bc-8deb-453f-a06d-a1dd3c54aa77",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 6, 15, 22)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ent.start, ent.end, ent.start_char, ent.end_char"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}