notebooks/3-setup-jupyterbook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Creating a Jupyter Book with The Turing Way*\n",
    "\n",
    "\n",
    "# Module 3: Creating a Jupyter Book with The Turing Way\n",
    "\n",
    "**Learning Objective:** \n",
    "  - Explain what files exist in the repository that we will use for the hands on session in this module (if you haven't already, please download the data required for this tutorial described in [module 1](./1-welcome.ipynb))\n",
    "  - Explore the important/minimal components for creating a Jupyter Book\n",
    "  - Build the first minimal version of the Jupyter Book locally using example files from _The Turing Way_\n",
    "  <!---Set up the repository, add a chapter, create a table of content, build the book--->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Jupyter Book\n",
    "\n",
    "Welcome! In this Jupyter Notebook we will introduce the basic commands to generate your first Jupyter Book. \n",
    "\n",
    "In the previous module, we briefly looked into the awesome and very detailed [documentation](https://jupyterbook.org/intro.html) of Jupyter Book, and its [GitHub repository](https://github.com/executablebooks/jupyter-book). \n",
    "\n",
    "Jupyter Book has a [command-line interface](https://jupyterbook.org/reference/cli.html), so in this tutorial we will show you how to build your book using iPython's special syntax that lets you execute shell commands from a Jupyter Notebook. In this example we will do so by prefixing `!` in each cell.\n",
    "\n",
    "*__TIP__: If you are unfamiliar with executing shell commands from Jupyter Notebooks, read this [tutorial by Jake VanderPlas](https://jakevdp.github.io/PythonDataScienceHandbook/01.05-ipython-and-shell-commands.html).*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the content of your book based on _The Turing Way_\n",
    "\n",
    "In order to build our Jupyter Book, we first need to create a folder where all the content of your book exists.\n",
    "This is also where Jupyter Book will create the `html` files in order to host the book online. \n",
    "\n",
    "As demonstrated for *The Turing Way*, we will store all the content of our book in a folder named `book` located in the main repository (the folder doesn't need to be named this way for Jupyter Book to work). Let's create it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The folder should be empty as you have just created it!\n",
    "\n",
    "__NOTE__: If this folder already exists, you are probably re-running this notebook without having cleared the `book` folder. Make sure you remove it using this command:\n",
    "\n",
    "```\n",
    "$ rm -R ../book/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example files for this tutorial\n",
    "\n",
    "We will now add some contents such as a book chapter and other important components to our `book` folder. \n",
    "\n",
    "In this tutorial, we will be using some of the chapters from _[The Turing Way](https://the-turing-way.netlify.app/welcome.html)_. \n",
    "We have provided some example files, which are placed under the folder `content` in the main repository. \n",
    "\n",
    "Let's inspect whats inside it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ../content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside `content` there are 2 Markdown files: `welcome.md` and `reproducible-research.md`.\n",
    "These are the [welcome page](https://the-turing-way.netlify.app/welcome) and the introduction to the [Guide for Reproducible Research](https://the-turing-way.netlify.app/reproducible-research/reproducible-research.html) in _The Turing Way_ respectively.\n",
    "\n",
    "The `references.bib` is a bibtex file, which contains all the references cited in the book (to read more about citing using sphinx and Jupyter Book go [here](https://jupyterbook.org/content/citations.html#)). \n",
    "\n",
    "The `figures` folder contains all the images and figures used in these files.\n",
    "\n",
    "There are also two folders, `overview` and `open`, that contain the content of the `Overview` and `Open Research` chapters respectively (you can see them online in [_The Turing Way_ book](https://the-turing-way.netlify.app/reproducible-research/overview.html)).\n",
    "\n",
    "The folder `content` also contains some Jupyter Notebooks (titled `demo` and `demo_2`), but we will cover the use of these in a later module. We will also cover the file `bibliography.md` later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's start!\n",
    "\n",
    "For our first Jupyter Book we don't want to include all these files. \n",
    "Let's import one by one to our `book` folder the files we will use.\n",
    "\n",
    "We first need an index page for our book, which corresponds to the file `welcome.md` in _The Turing Way_.\n",
    "Let's copy it into our `book` folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ../content/welcome.md ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now need a chapter. Let's use the second Markdown file `reproducible-research.md` as our first chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ../content/reproducible-research.md ../book/reproducible-research.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These files link to some citations and images placed in the `references.bib` file and `figures` folder, which we will also copy to our `book` folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp -R ../content/figures/ ../book/figures\n",
    "!cp ../content/references.bib ../book/references.bib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make sure our `book` folder now contains all the files we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see printed a `figures`, `references.bib`, `welcome.md` and `reproducible-research.md` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**We are almost there!**\n",
    "\n",
    "But we can not yet build our Jupyter Book.\n",
    "We have to specify the structure of our book.\n",
    "Let's do this in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the structure of your Jupyter Book (`_toc.yml`)\n",
    "\n",
    "A book always has a structure, that is, a __Table Of Contents (TOC)__  that defines the hierarchy and order of the content-files.\n",
    "\n",
    "Similarly, to create a Jupyter Book we need to have a file that specifies the TOC of your online book. Jupyter Book defines the TOC using a [yaml](https://en.wikipedia.org/wiki/YAML) file that needs to be named `_toc.yml`. This can be created manually, or automatically by running:\n",
    "\n",
    "```shell\n",
    "$ jupyter-book toc {path}\n",
    "```\n",
    "\n",
    "as long as the intended structure is also ordered alphanumerically.\n",
    "\n",
    "But in _The Turing Way_ we are always creating new content and restructuring our book. In these cases, is not practical to have our filenames ordered alphanumerically. We will thus show you how to manually create our `_toc.yml` using the python library [ruamel.yaml](https://yaml.readthedocs.io/en/latest/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ruamel.yaml import YAML\n",
    "\n",
    "yaml = YAML()\n",
    "\n",
    "# Define the contents of our _toc.yml\n",
    "toc_document = \"\"\"\n",
    "- file: welcome\n",
    "- file: reproducible-research\n",
    "\"\"\"\n",
    "\n",
    "# Save _toc.yml in the book directory\n",
    "toc_file = open('../book/_toc.yml', 'w')\n",
    "yaml.dump(yaml.load(toc_document), toc_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Important things to note of our example are:\n",
    "\n",
    "- The `file:` specifies the location of our markdown files relative to the main directory.\n",
    "- The first `file` defined in our `_toc.yml` will be the index (opening page) of our book.\n",
    "\n",
    "__Note:__ You don't need Python to create your `_toc.yml`, you can also do it with any text editor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building your book\n",
    "\n",
    "Let's check that our `book` directory now contains the `_toc.yml` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It does! That means we can now create our first Jupyter Book. This can be achieved by executing the command \n",
    "\n",
    "```shell\n",
    "$ jupyter-book build {path-to-book}\n",
    "``` \n",
    "\n",
    "or its shorter version:\n",
    "\n",
    "```shell\n",
    "$ jb build {path-to-book}\n",
    "``` \n",
    "\n",
    "Read more about this command and its options [here](https://jupyterbook.org/reference/cli.html#jupyter-book-build).\n",
    "\n",
    "Let's try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter-book build ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Congratulations! You have built your first Jupyter Book!*** \n",
    "\n",
    "We now have a `_build` folder in our book path that contains the `html` files of our book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means we can preview our book! \n",
    "\n",
    "If you are running this tutorial locally, paste the path specified at the very bottom of the `jupyter-book build` output into a new tab on your browser. \n",
    "The link should look like this: `file:///Users/PATH/book/_build/html/index.html` (please update the `Users/PATH`). \n",
    "\n",
    "If you are running this tutorial using a Binder instance in JupyterLab, head over to the `Launcher` tab, and under the `Notebook` section you should see an icon named `HTTP Server`. Click it, and a list containing all the folders and files of the repository should appear. Head over to the `book` folder, then `_build` and then `html`. Once inside `html`, you should be able to see a preview of your Jupyter Book.\n",
    "\n",
    "__IMPORTANT__: If you cannot see the `HTTP Server` button, you might be missing this extension. To install it, run `jupyter labextension install @jupyterlab/server-proxy` on the terminal of JupyterLab, or head over to the extensions tab and install the one named `server-proxy`. In both cases, you need to rebuild your JupyterLab for the changes to take effect. \n",
    "\n",
    "If you are running this tutorial using a Binder instance with classic Jupyter Notebook, launch the HTTP Server as instructed [here](https://github.com/choldgraf/jupyter-http-server) and head over to the `html` folder following the path detailed above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding sections and chapters to your Jupyter Book\n",
    "\n",
    "We will now add the `Overview` chapter to our Jupyter Book.\n",
    "\n",
    "To do so, let's first copy the `overview` folder from the `content` to the `book` folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp -R ../content/overview ../book/overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also inspect the files inside the overview folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ../book/overview/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 4 Markdown files. One is `overview.md`, which will be the landing page for the `Overview` chapter.\n",
    "\n",
    "We also have files corresponding to the 3 subsections of the `Overview` chapter: `overview-definitions.md`, `overview-benefit.md` and `overview-resources.md` (you can inspect them in *The Turing Way* [here](https://the-turing-way.netlify.app/reproducible-research/overview.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we re-build our book?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter-book build ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect your deployed Jupyter Book. It did not change at all! Why? Jupyter Book gives us a warning: the files that we have moved over have not been included in our `_toc.yml`.\n",
    "\n",
    "We will need to update the `_toc.yml` to include `Overview` as a chapter in our book, and define the rest of the files as  subsections of it by specifying them as `sections`, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ruamel.yaml import YAML\n",
    "\n",
    "yaml = YAML()\n",
    "\n",
    "# Define the contents of our _toc.yml\n",
    "toc_document = \"\"\"\n",
    "- file: welcome\n",
    "- file: reproducible-research\n",
    "- file: overview/overview\n",
    "  title: Overview\n",
    "  sections:\n",
    "  - file: overview/overview-definitions.md\n",
    "  - file: overview/overview-benefit.md\n",
    "  - file: overview/overview-resources.md\n",
    "\"\"\"\n",
    "\n",
    "# Save _toc.yml in the book directory\n",
    "toc_file = open('../book/_toc.yml', 'w')\n",
    "yaml.dump(yaml.load(toc_document), toc_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have used `title:` under the `file:` description of the overview chapter.\n",
    "This is to provide a title to this chapter that will appear in the index of the navigation bar (in the left hand side of the book).\n",
    "\n",
    "If `title` is not specified (as in our welcome file), the first title in our markdown file will be used. \n",
    "\n",
    "Let's build our Jupyter Book again to incorporate the new changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter-book build ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___EXERCISE___\n",
    "\n",
    "So far, we only have one chapter in our book (`Overview`), let's add the `Open Research` chapter by taking the following steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Copy the `open` folder from `content` to your `book folder`\n",
    "!cp -r ../content/open ../book"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. Inspect the files inside the `book` folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3. Update the _toc.yml file to include the chapter `open.md` and name it `Open Research`\n",
    "# 4. Add subchapters (other .md) to the `open` chapter, in your preferred order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Solution__: Uncomment the command below and run it to see the `_toc.yml` file for steps 3 and 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%load ../answers/module_3/exercise-module-3.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 5. Re-build the updated Jupyter Book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining nested sections\n",
    "\n",
    "We now have multiple chapters and subchapters in our Jupyter Book!\n",
    "\n",
    "What if the two chapters `Overview` and `Open Research` were displayed under `Guide for Reproducible Research` in our book - just how they appear in _[The Turing Way](https://the-turing-way.netlify.app/reproducible-research/reproducible-research.html)_? \n",
    "\n",
    "Turns out we can define this in our `_toc.yml` by nesting them as `sections` of `Guide for Reproducible Research`, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yaml = YAML()\n",
    "\n",
    "# Define the contents of our _toc.yml\n",
    "# Optional: add titles to all our chapters\n",
    "\n",
    "toc_document = \"\"\"\n",
    "- file: welcome\n",
    "- file: reproducible-research\n",
    "  sections:\n",
    "    - file: overview/overview\n",
    "      title: Overview\n",
    "      sections:\n",
    "      - file: overview/overview-definitions\n",
    "        title: Definitions\n",
    "      - file: overview/overview-benefit\n",
    "        title: Benefits\n",
    "      - file: overview/overview-resources\n",
    "        title: Resources\n",
    "    - file: open/open\n",
    "      title: Open Research\n",
    "      sections:\n",
    "      - file: open/open-data\n",
    "        title: Open Data\n",
    "      - file: open/open-source\n",
    "        title: Open Source\n",
    "      - file: open/open-hardware\n",
    "        title: Open Hardware\n",
    "      - file: open/open-access\n",
    "        title: Open Access\n",
    "      - file: open/open-notebooks\n",
    "        title: Open Notebooks\n",
    "      - file: open/open-scholarship\n",
    "        title: Open Scholarship\n",
    "\"\"\"\n",
    "\n",
    "# Save _toc.yml in the book directory\n",
    "toc_file = open('../book/_toc.yml', 'w')\n",
    "yaml.dump(yaml.load(toc_document), toc_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter-book build ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___Further reading___\n",
    "\n",
    "What if instead we would have wanted the section `Guide for Reproducible Research` to be a part of our book that has no landing page of its own, but has subsections underneath it?\n",
    "\n",
    "To complete this exercise: \n",
    "\n",
    "1. Read more about Parts/Chapters/Sections and how to specify them in the [official documentation](https://jupyterbook.org/customize/toc.html).\n",
    "2. Define the `Guide for Reproducible Research` as a part of your book, and `Overview` and `Open Research` as sections of it.\n",
    "\n",
    "__HINT__: You will no longer need the `reproducible-research.md` file.\n",
    "\n",
    "__Answer:__ Run the command below to check that your `_toc.yml` looks the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ../answers/module_3/ex2.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numbering your Jupyter Book sections\n",
    "\n",
    "We can also decide to number the sections in our Jupyter Book. This can be achieved by using the key `numbered` and set it to true.\n",
    "\n",
    "Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yaml = YAML()\n",
    "\n",
    "# Define the contents of our _toc.yml\n",
    "toc_document = \"\"\"\n",
    "- file: welcome\n",
    "  numbered: true\n",
    "- file: reproducible-research\n",
    "  sections:\n",
    "    - file: overview/overview\n",
    "      title: Overview\n",
    "      sections:\n",
    "      - file: overview/overview-definitions\n",
    "        title: Definitions\n",
    "      - file: overview/overview-benefit\n",
    "        title: Benefits\n",
    "      - file: overview/overview-resources\n",
    "        title: Resources\n",
    "    - file: open/open\n",
    "      title: Open Research\n",
    "      sections:\n",
    "      - file: open/open-data\n",
    "        title: Open Data\n",
    "      - file: open/open-source\n",
    "        title: Open Source\n",
    "      - file: open/open-hardware\n",
    "        title: Open Hardware\n",
    "      - file: open/open-access\n",
    "        title: Open Access\n",
    "      - file: open/open-notebooks\n",
    "        title: Open Notebooks\n",
    "      - file: open/open-scholarship\n",
    "        title: Open Scholarship\n",
    "\"\"\"\n",
    "\n",
    "# Save _toc.yml in the book directory\n",
    "toc_file = open('../book/_toc.yml', 'w')\n",
    "yaml.dump(yaml.load(toc_document), toc_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter-book build ../book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___Further reading___\n",
    "\n",
    "We might want to create separate numberings for each section of the book. Read how to do so [here](https://jupyterbook.org/customize/toc.html#number-your-book-s-chapters-and-sections) and number the sections under the `Overview` and `Open Research` chapters separately. \n",
    "\n",
    "__Answer:__ Run the command below to check that your `_toc.yml` looks the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ../answers/module_3/ex3.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Takeaway \n",
    "---\n",
    "\n",
    "- Jupyter Book uses markdown files to create chapters of an online book. In this tutorial, these files are placed in a directory called `book`.\n",
    "- All the images used in the book should also be stored in `book` (we saved them under the folder `figures` in this tutorial).\n",
    "- References used in the book should be defined in a `references.bib` file, which should also be stored in `book`.\n",
    "- A `_toc.yml` should be generated inside the folder `book` to structure the Jupyter Book (`toc` is the abbreviation for table of content).\n",
    "- With these files in place, you can create a minimal book using the command `$ jupyter-book build {path-to-book}`.\n",
    "\n",
    "### References\n",
    "- Read more about customization of the table of contents in the [official documentation](https://jupyterbook.org/customize/toc.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}