code_recsys/capstone_book1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "683d76e5-299a-49aa-bd5c-67f4b343daec",
   "metadata": {},
   "source": [
    "# Capstone Project: Wine Recommender\n",
    "---\n",
    "**Book 1: Problem Statement & Data Cleaning**<br>\n",
    "Book 2: Exploratory Data Analysis, Preprocessing & Feature Engineering<br>\n",
    "Book 3: Modelling, Conclusion & Recommendation<br>\n",
    "Author: Lee Wan Xian"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11125493-1b6c-42c1-95e8-a523682f45b8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Problem Statement"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a4230c0-0ed8-4997-b06e-ab547acea9ad",
   "metadata": {},
   "source": [
    "Our client is in the wine retail business. Customers have informed them of decision fatigue when trying to choose from their huge catalogue of wines offered. To improve customer experience and engagement, they have tasked us to develop a wine recommender system that suggests a few suitable wines for their customers. In turn, this would alleviate the decision-making toil."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22efa94-2eff-4ec3-89b1-7eb214f0afc1",
   "metadata": {},
   "source": [
    "## Contents\n",
    "- [Background](#Background)\n",
    "- [Dataset Information](#Dataset-Information)\n",
    "- [Data Cleaning](#Data-Cleaning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a02069c-2f7d-4321-a976-06cab78cf2f2",
   "metadata": {},
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "062b6201-3eb7-4ea4-a1a0-e9a99f4d7418",
   "metadata": {},
   "source": [
    "There are more than 10,000 wine grape varieties in the world ([source](https://www.masterclass.com/articles/what-are-the-different-types-of-wine-grapes-a-guide-to-the-various-types-of-red-and-white-wine-grapes-in-the-world)). This could easily translate to more than 10,000 varieties of wine out in the global market. With such a wide variety available, it is easy to understand how a typical consumer would struggle to find the wine that suits their palate.\n",
    "\n",
    "The client has received feedback from customers who felt time and energy were wasted during browsing through their wine catalogue. Eventually, customers felt unsure whether the wine they chose was the right one after purchase. It resulted in a negative shopping experience for them. Thus, the client has engaged us to build a wine recommender system that provides good suggestions for them to choose from. \n",
    "\n",
    "As a team of data professionals, we will leverage the wine catalogue provided by the client and the relevant reviews/ratings given by professional sommeliers. These datasets will form the database needed to build our recommender system. To add on, we will also use a list of wine trait descriptive terms from RoboSomm wine wheels. These wine wheels were created and compiled by researchers and wine enthusiasts alike ([source](https://towardsdatascience.com/robosomm-chapter-3-wine-embeddings-and-a-wine-recommender-9fc678f1041e)). These terms will help enable customers to choose specific wine traits from the recommender system. In turn, reducing the list of recommended wines for customers to choose from.\n",
    "\n",
    "The points/ratings stated in the wine reviews follow the 100-points system ([source](https://winefolly.com/tips/wine-ratings-explained/)).\n",
    "\n",
    "Points/Ratings|Wine Quality\n",
    "---|---\n",
    "95-100|Classics\n",
    "90-94|Superior\n",
    "85-90|Good\n",
    "80-84|Above Average\n",
    "70-79|Average"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ba08a7-dfc8-49a2-93e2-4cebe2bffd29",
   "metadata": {},
   "source": [
    "## Dataset Information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b6486c7-ec79-4a72-a147-0d866998acfb",
   "metadata": {},
   "source": [
    "### Datasets used\n",
    "\n",
    "There are 2 datasets included in the `data` folder for this project.\n",
    "\n",
    "* [`winemag-data-130k-v2`](../data/winemag-data-130k-v2.csv): Wine reviews that were scraped from [WineEnthusiast](https://www.winemag.com/?s=&drink_type=wine) webpage by Zack Thoutt ([Source](https://www.kaggle.com/datasets/zynicide/wine-reviews)).\n",
    "* [`descriptor_mapping`](../data/descriptor_mapping.csv): List of standardized descriptors that are derived from RoboSomm wine wheels. This list is available on Roald Schuring's Github ([Source](https://github.com/RoaldSchuring/wine_recommender/blob/master/descriptor_mapping.csv))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb62843f-5ca8-41ee-88b1-3ff846f2697e",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Data Dictionary\n",
    "\n",
    "#### Wine reviews dataframe\n",
    "\n",
    "Feature|Description\n",
    "---|---\n",
    "country|The country that the wine is from\n",
    "description|The review given to the wine\n",
    "designation|The vineyard within the winery where the grapes that made the wine are from\n",
    "points|The number of points/ratings WineEnthusiast rated the wine on a scale of 1-100\n",
    "price|The cost for a bottle of the wine\n",
    "province|The province or state that the wine is from\n",
    "region_1|The wine growing area in a province or state (i.e. Napa)\n",
    "region_2|The specific region specified within a wine growing area (i.e. Rutherford inside the Napa Valley)\n",
    "taster_name|Name of Wine taster\n",
    "taster_twitter_handle|Wine taster's twitter handle\n",
    "title|Name, Year & Vineyard of the wine. This feature is the key feature to differentiate wines from each other\n",
    "variety|Type of wine (i.e. Pinot Noir)\n",
    "winery|The place where the wine was made\n",
    "\n",
    "#### Wine traits descriptor mapping dataframe\n",
    "\n",
    "Feature|Description\n",
    "---|---\n",
    "raw_descriptor|The raw descriptive terms used to describe a wine's traits\n",
    "level_3|The 3rd level descriptive terms from RoboSomm wine wheels. This is the most detailed trait layer.\n",
    "level_2|The 2nd level descriptive terms from RoboSomm wine wheels\n",
    "level_1|The 1st level descriptive terms from RoboSomm wine wheels This is the most generic trait layer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c781d4-04ab-4894-adf3-763812f724ef",
   "metadata": {},
   "source": [
    "## Python Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0eea4f57-9b2c-4b58-a209-dc691018a0ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a587a104-433a-42ec-99d6-9f20de26e4b3",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4d670b26-4886-469d-81d0-7bf2e80ef1e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import datasets\n",
    "df_wine = pd.read_csv('../data/winemag-data-130k-v2.csv', index_col = 0)\n",
    "descriptor_map = pd.read_csv('../data/descriptor_mapping.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a85361ea-14bd-46c2-9fc4-99d8d1a110ca",
   "metadata": {},
   "source": [
    "### Dataframe: Wine Reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "783002b4-3ba7-40bc-a292-a846a3935540",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(129971, 13)\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 129971 entries, 0 to 129970\n",
      "Data columns (total 13 columns):\n",
      " #   Column                 Non-Null Count   Dtype  \n",
      "---  ------                 --------------   -----  \n",
      " 0   country                129908 non-null  object \n",
      " 1   description            129971 non-null  object \n",
      " 2   designation            92506 non-null   object \n",
      " 3   points                 129971 non-null  int64  \n",
      " 4   price                  120975 non-null  float64\n",
      " 5   province               129908 non-null  object \n",
      " 6   region_1               108724 non-null  object \n",
      " 7   region_2               50511 non-null   object \n",
      " 8   taster_name            103727 non-null  object \n",
      " 9   taster_twitter_handle  98758 non-null   object \n",
      " 10  title                  129971 non-null  object \n",
      " 11  variety                129970 non-null  object \n",
      " 12  winery                 129971 non-null  object \n",
      "dtypes: float64(1), int64(1), object(11)\n",
      "memory usage: 13.9+ MB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>description</th>\n",
       "      <th>designation</th>\n",
       "      <th>points</th>\n",
       "      <th>price</th>\n",
       "      <th>province</th>\n",
       "      <th>region_1</th>\n",
       "      <th>region_2</th>\n",
       "      <th>taster_name</th>\n",
       "      <th>taster_twitter_handle</th>\n",
       "      <th>title</th>\n",
       "      <th>variety</th>\n",
       "      <th>winery</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Italy</td>\n",
       "      <td>Aromas include tropical fruit, broom, brimston...</td>\n",
       "      <td>Vulkà Bianco</td>\n",
       "      <td>87</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sicily &amp; Sardinia</td>\n",
       "      <td>Etna</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kerin O’Keefe</td>\n",
       "      <td>@kerinokeefe</td>\n",
       "      <td>Nicosia 2013 Vulkà Bianco  (Etna)</td>\n",
       "      <td>White Blend</td>\n",
       "      <td>Nicosia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Portugal</td>\n",
       "      <td>This is ripe and fruity, a wine that is smooth...</td>\n",
       "      <td>Avidagos</td>\n",
       "      <td>87</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Douro</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Roger Voss</td>\n",
       "      <td>@vossroger</td>\n",
       "      <td>Quinta dos Avidagos 2011 Avidagos Red (Douro)</td>\n",
       "      <td>Portuguese Red</td>\n",
       "      <td>Quinta dos Avidagos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>US</td>\n",
       "      <td>Tart and snappy, the flavors of lime flesh and...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>87</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Paul Gregutt</td>\n",
       "      <td>@paulgwine</td>\n",
       "      <td>Rainstorm 2013 Pinot Gris (Willamette Valley)</td>\n",
       "      <td>Pinot Gris</td>\n",
       "      <td>Rainstorm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>US</td>\n",
       "      <td>Pineapple rind, lemon pith and orange blossom ...</td>\n",
       "      <td>Reserve Late Harvest</td>\n",
       "      <td>87</td>\n",
       "      <td>13.0</td>\n",
       "      <td>Michigan</td>\n",
       "      <td>Lake Michigan Shore</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Alexander Peartree</td>\n",
       "      <td>NaN</td>\n",
       "      <td>St. Julian 2013 Reserve Late Harvest Riesling ...</td>\n",
       "      <td>Riesling</td>\n",
       "      <td>St. Julian</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>US</td>\n",
       "      <td>Much like the regular bottling from 2012, this...</td>\n",
       "      <td>Vintner's Reserve Wild Child Block</td>\n",
       "      <td>87</td>\n",
       "      <td>65.0</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Willamette Valley</td>\n",
       "      <td>Paul Gregutt</td>\n",
       "      <td>@paulgwine</td>\n",
       "      <td>Sweet Cheeks 2012 Vintner's Reserve Wild Child...</td>\n",
       "      <td>Pinot Noir</td>\n",
       "      <td>Sweet Cheeks</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    country                                        description  \\\n",
       "0     Italy  Aromas include tropical fruit, broom, brimston...   \n",
       "1  Portugal  This is ripe and fruity, a wine that is smooth...   \n",
       "2        US  Tart and snappy, the flavors of lime flesh and...   \n",
       "3        US  Pineapple rind, lemon pith and orange blossom ...   \n",
       "4        US  Much like the regular bottling from 2012, this...   \n",
       "\n",
       "                          designation  points  price           province  \\\n",
       "0                        Vulkà Bianco      87    NaN  Sicily & Sardinia   \n",
       "1                            Avidagos      87   15.0              Douro   \n",
       "2                                 NaN      87   14.0             Oregon   \n",
       "3                Reserve Late Harvest      87   13.0           Michigan   \n",
       "4  Vintner's Reserve Wild Child Block      87   65.0             Oregon   \n",
       "\n",
       "              region_1           region_2         taster_name  \\\n",
       "0                 Etna                NaN       Kerin O’Keefe   \n",
       "1                  NaN                NaN          Roger Voss   \n",
       "2    Willamette Valley  Willamette Valley        Paul Gregutt   \n",
       "3  Lake Michigan Shore                NaN  Alexander Peartree   \n",
       "4    Willamette Valley  Willamette Valley        Paul Gregutt   \n",
       "\n",
       "  taster_twitter_handle                                              title  \\\n",
       "0          @kerinokeefe                  Nicosia 2013 Vulkà Bianco  (Etna)   \n",
       "1            @vossroger      Quinta dos Avidagos 2011 Avidagos Red (Douro)   \n",
       "2           @paulgwine       Rainstorm 2013 Pinot Gris (Willamette Valley)   \n",
       "3                   NaN  St. Julian 2013 Reserve Late Harvest Riesling ...   \n",
       "4           @paulgwine   Sweet Cheeks 2012 Vintner's Reserve Wild Child...   \n",
       "\n",
       "          variety               winery  \n",
       "0     White Blend              Nicosia  \n",
       "1  Portuguese Red  Quinta dos Avidagos  \n",
       "2      Pinot Gris            Rainstorm  \n",
       "3        Riesling           St. Julian  \n",
       "4      Pinot Noir         Sweet Cheeks  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df_wine.shape)\n",
    "df_wine.info()\n",
    "df_wine.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "97f8b233-2c1c-40e6-af4a-dd7b5f33091a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "region_2                 79460\n",
       "designation              37465\n",
       "taster_twitter_handle    31213\n",
       "taster_name              26244\n",
       "region_1                 21247\n",
       "price                     8996\n",
       "country                     63\n",
       "province                    63\n",
       "variety                      1\n",
       "description                  0\n",
       "points                       0\n",
       "title                        0\n",
       "winery                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for missing values\n",
    "df_wine.isna().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe387332-d973-403d-904d-31d0809d7a11",
   "metadata": {},
   "source": [
    "Reviews with missing values do not provide many insights into the similarity between wines or wine tasters. The recommender system will not be able to use such data for training. Thus, we will remove all reviews with missing data values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e883f455-efd7-4bf9-a4ba-ca7710a30c50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove reviews with missing data\n",
    "df_wine.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "313cb881-71ed-4ecf-a6d1-602e511562cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    20493\n",
       "True      1894\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for duplicated data records\n",
    "df_wine.duplicated().value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c34f44e6-d365-4733-bb00-108b0acad8b7",
   "metadata": {},
   "source": [
    "Given that this is a recommender use case, we should ideally retain the integrity of the database. Only 8% of the dataset consists of duplicated records. The recommender should be able to manage multiple identical records ([source](https://github.com/apple/turicreate/issues/1433)) and should not be affected by this. Thus, we will leave the duplicated records untouched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a99cd7d2-6904-428c-b392-4ce36706c198",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reorder the columns & remove 'taster_twitter_handle' as it's unnecessary\n",
    "col = ['taster_name', 'title', 'points', 'variety', 'designation', 'winery', 'country', 'province', 'region_1', 'region_2', 'price', 'description']\n",
    "df_wine_ordered = df_wine[col]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72641f76-26c6-40ad-b0e6-da0373891f52",
   "metadata": {},
   "source": [
    "It is ideal for the wine recommender to recommend good quality wines instead of inferior ones. Thus, we will only extract wine reviews with points equal to 88 or above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "775d76e7-17de-4463-800d-86fc4e456cf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_wine_clean = df_wine_ordered.loc[df_wine_ordered['points'] >= 88, :].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8994f107-14c1-41ff-bd99-625e66470c2b",
   "metadata": {},
   "source": [
    "### Dataframe: Wine Trait Descriptors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "79d97a79-896a-4e6a-b810-195f4c664a3f",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1015, 4)\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1015 entries, 0 to 1014\n",
      "Data columns (total 4 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   raw descriptor  1015 non-null   object\n",
      " 1   level_3         1015 non-null   object\n",
      " 2   level_2         1015 non-null   object\n",
      " 3   level_1         1015 non-null   object\n",
      "dtypes: object(4)\n",
      "memory usage: 31.8+ KB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raw descriptor</th>\n",
       "      <th>level_3</th>\n",
       "      <th>level_2</th>\n",
       "      <th>level_1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>abras</td>\n",
       "      <td>abrasive</td>\n",
       "      <td>high_tannin</td>\n",
       "      <td>tannin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>acacia</td>\n",
       "      <td>acacia</td>\n",
       "      <td>flowery</td>\n",
       "      <td>flower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>acacia_flower</td>\n",
       "      <td>acacia</td>\n",
       "      <td>flowery</td>\n",
       "      <td>flower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aciddriven</td>\n",
       "      <td>acid_driven</td>\n",
       "      <td>high_acid</td>\n",
       "      <td>acid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>aggress</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>high_acid</td>\n",
       "      <td>acid</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  raw descriptor      level_3      level_2 level_1\n",
       "0          abras     abrasive  high_tannin  tannin\n",
       "1         acacia       acacia      flowery  flower\n",
       "2  acacia_flower       acacia      flowery  flower\n",
       "3     aciddriven  acid_driven    high_acid    acid\n",
       "4        aggress   aggressive    high_acid    acid"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(descriptor_map.shape)\n",
    "descriptor_map.info()\n",
    "descriptor_map.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e06f0aa8-fa37-46d6-9ef1-4ea30a4e79cd",
   "metadata": {},
   "source": [
    "There are no missing values in this dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a495a018-a9e4-4ab9-bd00-894e62c28df5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    1015\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for duplicated reviews\n",
    "descriptor_map.duplicated().value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d83a005a-4456-49e8-b871-97c99bac00df",
   "metadata": {},
   "source": [
    "There are no duplicate records in this dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f462184-58ef-4038-828d-1ff72ead3730",
   "metadata": {},
   "source": [
    "**Export `df_wine` dataframe into pickle file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6749d8ba-8b4c-4a75-b0e6-486106a08cce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export the dataframe into pickle file  \n",
    "df_wine_clean.to_pickle('../data/df_wine_clean.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c49c12-9ef9-44e4-8415-e4d770c3f6a2",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "**Wine Reviews**: All reviews with missing values were removed. Any reviews with points lower than 88 have been removed. The order of the columns have been reorganized for better visualisation. The `taster_twitter_handle` column has been dropped as it serves the same purpose as `taster_name`.\n",
    "\n",
    "**Wine Trait Descriptors**: No data cleaning was required as there are no duplicated records or missing data values."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f58f36ee-d3f0-4a55-a207-a498436294af",
   "metadata": {},
   "source": [
    "*Please proceed to Book 2 for EDA, Preprocessing & Feature Engineering*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:capstone]",
   "language": "python",
   "name": "conda-env-capstone-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}