chapter-04/4.4.4-L-1-regularized-logistic-regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.4.4 L1 Regularized Logistic Regression\n",
    "\n",
    "The L1 penalty for logistic regression, we would maximize a penalized version of 4.20 (4.31):\n",
    "$$\n",
    "\\max_{\\beta_0, \\beta} \\left\\{ \n",
    "\\sum_{i=1}^N \\left[y_i(\\beta_0+\\beta^Tx_i) - \\log(1+e^{\\beta_0+\\beta^Tx_i}) \\right]\n",
    "-\\gamma\\sum_{j=1}^p |\\beta_j|\n",
    "\\right\\}\n",
    "$$\n",
    "\n",
    "As with the lasso, we do not penalize the intercept term, and standardize the predictors for the penalty. Criterion (4.31) is concave, and a solution can be found using nonlinear programming methods. Alternatively, using the same quadratic approximations that were used in the Newton algorithm, we can solve by repeated application of a weighted lasso algorithm (4.32):\n",
    "$$\n",
    "\\mathbf{x}_j^T(\\mathbf{y}-\\mathbf{p})=\\gamma \\cdot sign(\\beta_j)\n",
    "$$\n",
    "\n",
    "Path algorithms such as LAR for lasso are more difficult, because the coefficient profiles are piecewise smooth rather than linear. Nevertheless, progress can be made using quadratice approximations."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}