chapter-04/4.2-linear-regression-of-an-indicator-matrix.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.2 Linear Regression of an Indicator Matrix\n",
    "\n",
    "- if $\\mathcal{G}$ has K classes, there will be K indicators $Y_k, k = 1,...,K$ with Y_k = 1 if G = k else 0.\n",
    "\n",
    "- The N training instance will form an $N \\times K$ *indicator response matrix* **Y**.\n",
    "\n",
    "We fit a linear regression model to columns of **Y** simultaneously (4.3):\n",
    "$$\n",
    "\\hat{\\mathbf{Y}} = \\mathbf{X}(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\mathbf{Y}\n",
    "$$\n",
    "\n",
    "and the $(p+1)\\times K$ coefficient matrix: $\\mathbf{\\hat{B}}=(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\mathbf{Y}$\n",
    "\n",
    "A new input x is classified as follows:\n",
    "\n",
    "- Compute the fitted output $\\hat{f}(x)^T=(1, x^T)\\hat{\\mathbf{B}}$, a K vector;\n",
    "\n",
    "- Identify the largest component and classify accordingly (4.4):\n",
    "$$\n",
    "\\hat{G}(x) = argmax_{k \\in \\mathcal{G}}  \\hat{f}_k(x)\n",
    "$$\n",
    "\n",
    "Some properties: \n",
    "- $\\sum_{k \\in \\mathcal{G}} \\hat{f}_k(x) = 1$\n",
    "\n",
    "- However $\\hat{f}_k(x)$ can be negative or greater than 1. Thus violets probability distribution.\n",
    "\n",
    "- The violation doesn't say that it won't work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "A more simplistic viewpoint is to construct targets $t_k$ for each class, where $t_k$ is the kth column of the $\\mathbf{I}_k$, then fit the linear model by least squares (4.5):\n",
    "$$\n",
    "\\underset{\\mathbf{B}}{min} \\sum_{i=1}^N \\| y_i - [(1, x_i^T)\\mathbf{B}]^T\\|^2\n",
    "$$\n",
    "\n",
    "is a sum-of-squared Euclidean distances of the fitted vectors from their targets. A new observation is classified to the closest target (4.6):\n",
    "\n",
    "$$\n",
    "\\hat{G}(x) = \\underset{k}{argmin} \\| \\hat{f}(x) - t_k \\|^2\n",
    "$$\n",
    "\n",
    "- The sum-of-squared-norm criterian is exactly the criterian for multiple response linear regression, just view differently. The components of (4.5) can rearranged as a separate linear model for each element.\n",
    "\n",
    "- The closest target classification rule is exactly as the same as the maximum fitted component cirterion.\n",
    "\n",
    "There is a serious problem with the regression approach when K >= 3. Because of the rigid nature of the regression model, classes can be *masked* by others. A general rule to fix this issue for K >= 4 classes is using polynomial terms up to degrees K - 1.\n",
    "\n",
    "\n",
    "**TODO**: implement FIGURE 4.2\n",
    "\n",
    "\n",
    "**TODO**: implement FIGURE 4.3\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}