chapter-03/3.2.2-the-gauss-markov-theorem.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.2.2 The Gauss–Markov Theorem\n",
    "\n",
    "The Gauss-Markov theorem states that the least squares estimates of the $\\beta$ have the smallest variance among all linear unbiased estimates. We focus on estimation of any linear combination of the parameters $\\theta=\\alpha^T\\hat{\\beta}$, i.e predictions $f(x_0)={x_0}^T\\beta$ are of this form. \n",
    "The least squares estimate is (3.17):\n",
    "\n",
    "$$\n",
    "\\hat{\\theta}=\\alpha^T\\hat{\\beta} = \\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\mathbf{y}\n",
    "$$\n",
    "\n",
    "and $\\alpha^T\\hat{\\beta}$ is unbiased, since (3.18):\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "E(\\alpha^T\\hat{\\beta}) &= E( \\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\mathbf{y})\\\\\n",
    "&=  \\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T\\mathbf{X}\\beta\\\\\n",
    "&= \\alpha^T\\beta\n",
    "\\end{align}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *Gauss-Markov* states that if we have any other linear estimator $\\tilde{\\theta}=\\mathbf{c}^T\\mathbf{y}$, that is, $E(\\mathbf{c}^T\\mathbf{y})=\\alpha^T\\beta$, then (3.19):\n",
    "\n",
    "$$\n",
    "Var(\\alpha^T\\hat{\\beta})\\le Var(\\mathbf{c}^T\\mathbf{y})\n",
    "$$\n",
    "\n",
    "*Proof*: \n",
    "\n",
    "Let's assume that $\\mathbf{c}^T\\mathbf{y}=(\\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T + \\mathbf{d}^T)\\mathbf{y}$, then:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "E(\\mathbf{c}^T\\mathbf{y})&= E((\\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T + \\mathbf{d}^T)\\mathbf{y})\\\\\n",
    "&= E((\\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T + \\mathbf{d}^T)(\\mathbf{X}\\beta + \\varepsilon))\\\\\n",
    "&= \\alpha^T\\beta + d^T\\mathbf{X}\\beta\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Since $E(\\mathbf{c}^T\\mathbf{y})$ is unbiased $d^T\\mathbf{X}=0$.\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "Var(\\mathbf{c}^T\\mathbf{y}) &= \\mathbf{c}^{T}Var(\\mathbf{y})\\mathbf{c} = \\sigma^2\\mathbf{c}^T\\mathbf{c}\\\\\n",
    "&= \\sigma^2(\\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\mathbf{X}^T + \\mathbf{d}^T)(\\mathbf{X}(\\mathbf{X}^T\\mathbf{X})^{-1}\\alpha + \\mathbf{d})\\\\\n",
    "&= \\sigma^2(\\alpha^T(\\mathbf{X}^T\\mathbf{X})^{-1}\\alpha+\\mathbf{d}^T\\mathbf{d})\\\\\n",
    "&= Var(\\alpha^T\\hat{\\beta})+\\sigma^2\\mathbf{d}^T\\mathbf{d}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Since $\\mathbf{d}^T\\mathbf{d}\\ge 0$, the proof is completed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the MSE of an estimator $\\tilde{\\theta}$ is estimating $\\theta$:\n",
    "$$\n",
    "\\begin{align}\n",
    "MSE(\\tilde{\\theta}) &= E(\\tilde{\\theta} - \\theta)^2\\\\\n",
    "&= Var(\\tilde{\\theta})+[E(\\tilde{\\theta}) - \\theta]^2\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first term  is the variance, while the second term is the squared bias. The theorem implies that the least squares estimator has the smallest mean squared error of all linear estimators with no bias. However, there may exist a biased estimator with smaller MSE.\n",
    "\n",
    "Mean squared error is related to prediction accuracy. Consider the prediction of the $x_0$, (3.21):\n",
    "\n",
    "$$\n",
    "Y_0 = f(x_0) + \\varepsilon_0\n",
    "$$\n",
    "\n",
    "Then the EPE of an estimate $\\tilde{f}(x_0)={x_0}^T\\tilde{\\beta}$ is (3.22):\n",
    "$$\n",
    "\\begin{align}\n",
    "E(Y_0 - \\tilde{f}(x_0)) &= \\sigma^2 + E({x_0}^T\\tilde{\\beta}-f(x_0))^2\\\\\n",
    "&= \\sigma^2 + MSE(\\tilde{f}(x_0))\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Therefore, EPE and MSE differ only by the constant $\\sigma^2$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}