Logistic_Regression_From_Scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "## Logistic regression is a probabilistic classifier similar to the Naïve Bayes\n",
    "\n",
    "The goal of binary logistic regression is to train a classifier that can make a binary decision about the class of a new input observation. In the sigmoid classifier that will help us make this decision.\n",
    "\n",
    "Consider a single input observation x, which we will represent by a vector of features [x1, x2,..., xn] (we’ll show sample features in the next subsection).\n",
    "\n",
    "The classifier\n",
    "\n",
    "output y can be 1 (meaning the observation is a member of the class) or 0 (the observation is not a member of the class). We want to know the probability P(y = 1|x) that this observation is a member of the class. So perhaps the decision is “positive sentiment” versus “negative sentiment”, the features represent counts of words in a\n",
    "document, P(y = 1|x) is the probability that the document has positive sentiment,\n",
    "and P(y = 0|x) is the probability that the document has negative sentiment\n",
    "\n",
    "Logistic regression solves this task by learning, from a training set, a vector of weights and a bias term. Each weight wi is a real number, and is associated with one of the input features xi. The weight wi represents how important that input feature is to the classification decision, and can be positive (providing evidence that the instance being classified belongs in the positive class) or negative (providing evidence that the instance being classified belongs in the negative class). Thus we might expect in a sentiment task the word awesome to have a high positive weight, and bias term abysmal to have a very negative weight. The bias term, also called the intercept, is\n",
    "intercept another real number that’s added to the weighted inputs.\n",
    "\n",
    "To make a decision on a test instance— after we’ve learned the weights in training— the classifier first multiplies each xi by its weight wi, sums up the weighted features, and adds the bias term b. The resulting single number z expresses the weighted sum of the evidence for the class.\n",
    "\n",
    "\n",
    "![Imgur](https://imgur.com/sX3ek9T.png)\n",
    "\n",
    "Above equation can be represented using the dot product notation from linear algebra. The dot product of two vectors a and b, written as a · b is the sum of the products of the corresponding elements of each vector. Thus the following is an equivalent formation of the above equation:\n",
    "\n",
    "## $$z = w· x+b$$\n",
    "\n",
    "Because this is simply the below vector calculation\n",
    "\n",
    "![Imgur](https://imgur.com/r2Ayx2o.png)\n",
    "\n",
    "---\n",
    "\n",
    "But note that nothing in above Eq forces z to be a legal probability, that is, to lie between 0 and 1. In fact, since weights are real-valued, the output might even be\n",
    "negative; z ranges from −∞ to ∞.\n",
    "\n",
    "To create a probability, we’ll pass z through the sigmoid function, σ(z). The\n",
    "sigmoid function (named because it looks like an s) is also called the logistic function, and gives logistic regression its name. The sigmoid has the following equation\n",
    "\n",
    "![Imgur](https://imgur.com/sjkaJq1.png)\n",
    "\n",
    "The sigmoid has a number of advantages; it takes a real-valued number and maps it into the range [0,1], which is just what we want for a probability. Because it is nearly linear around 0 but flattens toward the ends, it tends to squash outlier values toward 0 or 1. And it’s differentiable\n",
    "\n",
    "![Imgur](https://imgur.com/hpMykvK.png)\n",
    "\n",
    "If we apply the sigmoid to the sum of the weighted features, we get a number between 0 and 1. To make it a probability, we just need to make\n",
    "sure that the two cases, p(y = 1) and p(y = 0), sum to 1. We can do this as follows\n",
    "\n",
    "![Imgur](https://imgur.com/vlRcxZt.png)\n",
    "\n",
    "Now we have an algorithm that given an instance x computes the probability P(y = 1|x). How do we make a decision? For a test instance x, we say yes if the\n",
    "probability P(y = 1|x) is more than .5, and no otherwise. We call .5 the decision boundary:\n",
    "\n",
    "![Imgur](https://imgur.com/7KCnMNt.png)\n",
    "\n",
    "#### Now a little bit more of math to represent the same above\n",
    "\n",
    "\n",
    "In logistic regression, the function input z becomes the weighted sum of features. Given a data sample x with n features, x1, x2,\n",
    "..., xn (x represents a feature vector and x = (x1, x2, ..., xn)), and weights (also called coefficients) of the model w (w represents a vector (w1, w2, ..., wn)), z is expressed as follows:\n",
    "\n",
    "![Imgur](https://imgur.com/nXLUyuf.png)\n",
    "\n",
    "Also, occasionally, the model comes with an intercept (also called bias), w0. In this instance, the preceding linear relationship becomes:\n",
    "\n",
    "![Imgur](https://imgur.com/wnqt8Ys.png)\n",
    "\n",
    "#### Note the basic representation of the predicted y-value of a Logistic Function\n",
    "\n",
    "![Imgur](https://imgur.com/kxOrYA9.png)\n",
    "\n",
    "![Imgur](https://imgur.com/PgL9Ikv.png)\n",
    "\n",
    "\n",
    "A logistic regression model or, more specifically, its weight vector w is learned from the training data, with the goal of predicting a positive sample as close to 1 as possible and predicting a negative sample as close to 0 as possible. In mathematical language, the weights are trained so as to minimize the cost defined as the mean squared error (MSE), which measures the average of squares of the difference between the truth and the prediction.\n",
    "\n",
    "![Imgur](https://imgur.com/OkwP5KC.png)\n",
    "\n",
    "However the above cost function is non-convex, i.e. when searching for the optimal w, many local (suboptimal) optimums are found and the\n",
    "function does not converge to a global optimum.\n",
    "\n",
    "Examples of the convex and non-convex functions are plotted respectively below:\n",
    "\n",
    "![Imgur](https://imgur.com/28ri9b3.png)\n",
    "\n",
    "To overcome this, the cost function in practice is defined as follows:\n",
    "\n",
    "![Imgur](https://imgur.com/A4DuZxI.png)\n",
    "\n",
    "When the ground truth y(i) = 1, if the model predicts correctly with full confidence (the positive class with 100% probability), the sample cost j is 0; the cost j increases. And cost increased when predicted probability (y_hat) decreases. If the model incorrectly predicts that there is no chance of the positive class, the cost is infinitely high.\n",
    "\n",
    "![Imgur](https://imgur.com/zPo2TDZ.png)\n",
    "\n",
    "On the contrary, when the ground truth y(i) = 0, if the model predicts correctly with full confidence (the positive class with 0 probability, or the negative class with 100% probability), the sample cost j is 0;\n",
    "\n",
    "The cost j increases when the predicted probability y_hat increases. When it incorrectly predicts that there is using no chance of negative class, cost becomes When infinitely it incorrectly high. predicts We can visualize\n",
    "\n",
    "![Imgur](https://imgur.com/W0FDQS4.png)\n",
    "\n",
    "Minimizing this alternative cost function is actually equivalent to minimizing the MSE-based cost function. The advantages of choosing it over the MSE one include\n",
    "the following:\n",
    "\n",
    "![Imgur](https://imgur.com/luKwJzZ.png)\n",
    "\n",
    "For understanding the above graph also checkout Andrew Ng\n",
    "\n",
    "https://www.youtube.com/watch?v=HIQlmHxI6-0&ab_channel=ArtificialIntelligence-AllinOne\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Cost-Function of Logistic-Regression\n",
    "\n",
    "First a little comarison between Linear Regression and Logistic Regression\n",
    "\n",
    "1. **Linear regression** uses the following hypothesis:\n",
    "\n",
    "$$ h_\\theta(x) = \\theta_0 + \\theta_1 x $$\n",
    "\n",
    "Accordingly, the cost function is defined as:\n",
    "\n",
    "$$J(\\theta) = \\dfrac {1}{2m} \\displaystyle \\sum_{i=1}^m \\left (h_\\theta (x^{(i)}) - y^{(i)} \\right)^2$$\n",
    "\n",
    "2. The **logistic regression** uses a sigmoid/logistic function which is\n",
    "\n",
    "$$ 0 \\leq h_\\theta (x) \\leq 1 $$\n",
    "\n",
    "Defined as :\n",
    "\n",
    "$$ h_\\theta (x) =  \\dfrac{1}{1 + e^{-(\\theta^T x)}} $$\n",
    "\n",
    "Accordingly, our cost function has also changed. However, instead of plugging-in the new h(x) equation directly, we used logarithm.\n",
    "\n",
    "#### The main reason of using Logarithms here is to make the function a Convex one so it has a global minima and so Gradient Descent can be applied on it.\n",
    "\n",
    "\n",
    "$$ J(\\theta) = \\dfrac{1}{m} \\sum_{i=1}^m Cost(h_\\theta(x^{(i)}),y^{(i)}) $$\n",
    "\n",
    "$$ Cost(h_\\theta(x),y) = -log(h_\\theta(x)) \\;  \\text{if y = 1} $$\n",
    "\n",
    "$$ Cost(h_\\theta(x),y) = -log(1-h_\\theta(x)) \\;  \\text{if y = 0} $$\n",
    "\n",
    "And the new cost function is defined as:\n",
    "\n",
    "$$ J(\\theta) = - \\frac{1}{m} \\sum_{i=1}^m [y^{(i)}\\log (h_\\theta (x^{(i)})) + (1 - y^{(i)})\\log (1 - h_\\theta(x^{(i)}))]$$\n",
    "\n",
    "[Slightly modified from here](https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression)\n",
    "\n",
    "where $h_{\\theta}(x)$ is defined as follows\n",
    "\n",
    "$$h_{\\theta}(x)=g(\\theta^{T}x)$$\n",
    "$$g(z)=\\frac{1}{1+e^{-z}}$$\n",
    "\n",
    "Substituting value of z\n",
    "\n",
    "$$\n",
    "h_{\\theta}(\\mathbf{x}_i) = \\dfrac{1}{1+e^{(- \\theta^T \\mathbf{x}_i)}}\n",
    "$$\n",
    "\n",
    "So in Probability Terms\n",
    "\n",
    "$$\n",
    "P( y_i =1 | \\mathbf{x}_i ; \\theta) = h_{\\theta}(\\mathbf{x}_i) = \\dfrac{1}{1+e^{(- \\theta^T \\mathbf{x}_i)}}\n",
    "$$\n",
    "\n",
    "so $y_i = 1$ with probability $h_{\\theta}(\\mathbf{x}_i)$ and $y_i=0$ with probability $1-h_{\\theta}(\\mathbf{x}_i)$.\n",
    "\n",
    "---\n",
    "\n",
    "Noting again our Logistic Regression Vectorized Cost function is\n",
    "\n",
    "## $$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{i}\\log(h_\\theta(x^{i}))+(1-y^{i})\\log(1-h_\\theta(x^{i}))$$\n",
    "\n",
    "The $h_\\theta(x^{i})$ is the Predicted Y AND $y^{i}$ are the $y^{true}$ so the above formulae will be\n",
    "\n",
    "## $$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{true}\\log(Y_{pred})+(1-y^{true})\\log(1-Y_{pred})$$\n",
    "\n",
    "\n",
    "\n",
    "calculating the partial derivative with respect to $\\theta$ of the cost function (the logs are natural logarithms):\n",
    "\n",
    "$$ \\frac{\\partial}{\\partial\\theta_{j}}J(\\theta) = \\frac{1}{m}\\sum_{i=1}^{m}(h_\\theta(x^{i})-y^i)x_j^i$$\n",
    "\n",
    "So the \".. the more hypothesis is off from y, the larger the cost function output. If our hypothesis is equal to y, then our cost is 0.\"\n",
    "\n",
    "It's also mentioned in the class notes that MLE (maximum-likelihood estimation) is used to derive the logs in the cost function. I can see how logs function and set penalty values until we find the right values.\n",
    "\n",
    "[1]: https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "# Gradient Descent Loop for Logistic Regression\n",
    "\n",
    "![Imgur](https://imgur.com/MWrz5iE.png)\n",
    "\n",
    "[source](https://www.internalpointers.com/post/cost-function-logistic-regression)\n",
    "\n",
    "---\n",
    "\n",
    "Refer below for calculating the Gradient for Logistic Regression\n",
    "\n",
    "![Imgur](https://imgur.com/XY8Pnfk.png)\n",
    "\n",
    "[Source](https://web.stanford.edu/~jurafsky/slp3/5.pdf)\n",
    "\n",
    "---\n",
    "\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "# Regularization for Gradient Descent for logistic regression\n",
    "\n",
    "The cost function of the logistic regression is updated to penalize high values of the parameters and is given by,\n",
    "\n",
    "![Imgur](https://imgur.com/kD1tGgl.png)\n",
    "\n",
    "\n",
    "So, without regularization, the gradient descent for logistic regressionwas would be given by,\n",
    "\n",
    "![Imgur](https://imgur.com/WQMw99k.png)\n",
    "\n",
    "\n",
    "But since the equation for cost function has changed in (1) to include the regularization term, there will be a change in the derivative of cost function that was plugged in the gradient descent algorithm,\n",
    "\n",
    "![Imgur](https://imgur.com/WYSP0ca.png)\n",
    "\n",
    "Because the first term of cost fuction remains the same, so does the first term of the derivative. So taking derivative of second term gives $\\fracλ{m}θ_j$ as seen above.\n",
    "\n",
    "![Imgur](https://imgur.com/i2l8Es4.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "Now below is for Vectorized logistic regression with regularization\n",
    "\n",
    "# Gradient of loss function w.r.t each weight in weight vector\n",
    "\n",
    "### $$dw^{(t)} = x_n(y_n − σ((w^{(t)})^{T} x_n+b^{t}))- \\frac{λ}{N}w^{(t)})$$\n",
    "\n",
    "---\n",
    "\n",
    "# Gradient of the intercept\n",
    "\n",
    "### $$db^{(t)} = y_n- σ((w^{(t)})^{T} x_n+b^{t}))$$\n",
    "\n",
    "After calculating Gradient of loss function w.r.t each weight in weight vector AND gradient of the intercept, these are the two equations by which our model will learn to get better at cats and dogs image classification.\n",
    "\n",
    "![Imgur](https://imgur.com/Va11TbN.png)\n",
    "\n",
    "The α represents the learning rate for our gradient descent algorithm i.e. the step size for going down the hill. The term(s) next to α represent the gradients of the loss function corresponding to the weights and the bias respectively."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Derive partial derivative of Logistic Regression Cost Function\n",
    "\n",
    "### First the most important Basic Derivative Formulaes that we need here\n",
    "\n",
    "![Imgur](https://imgur.com/2huUIby.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Derive partial derivative of Logistic Regression Cost Function\n",
    "\n",
    "(Slightly adpted from here)[https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression]\n",
    "\n",
    "#### The partial derivative of\n",
    "\n",
    "$$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{i}\\log(h_\\theta(x^{i}))+(1-y^{i})\\log(1-h_\\theta(x^{i}))$$\n",
    "\n",
    "where $h_{\\theta}(x)$ is given as below\n",
    "\n",
    "$$h_{\\theta}(x)=g(\\theta^{T}x)$$\n",
    "$$g(z)=\\frac{1}{1+e^{-z}}$$\n",
    "\n",
    "is derived to be as below\n",
    "\n",
    "$$ \\frac{\\partial}{\\partial\\theta_{j}}J(\\theta) = \\frac{1}{m}\\sum_{i=1}^{m}(h_\\theta(x^{i})-y^i)x_j^i$$\n",
    "\n",
    "Meaning, calculating the partial derivative with respect to $\\theta$ of the cost function (the logs are natural logarithms):\n",
    "\n",
    "$$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{i}\\log(h_\\theta(x^{i}))+(1-y^{i})\\log(1-h_\\theta(x^{i}))$$\n",
    "\n",
    "---\n",
    "\n",
    "# Actual Derivation of the above partial derivative formulae\n",
    "\n",
    "$$\\theta x^i:=\\theta_0+\\theta_1 x^i_1+\\dots+\\theta_p x^i_p.$$\n",
    "\n",
    "Then\n",
    "\n",
    "$$\\log h_\\theta(x^i)=\\log\\frac{1}{1+e^{-\\theta x^i} }=-\\log ( 1+e^{-\\theta x^i} ),$$\n",
    "\n",
    "$$\\log(1- h_\\theta(x^i))=\\log(1-\\frac{1}{1+e^{-\\theta x^i} })=\\log (e^{-\\theta x^i} )-\\log ( 1+e^{-\\theta x^i} )=-\\theta x^i-\\log ( 1+e^{-\\theta x^i} )$$\n",
    "\n",
    "Because,\n",
    "\n",
    "$\\log(x/y) = \\log(x) - \\log(y)$\n",
    "\n",
    "Since our original cost function is the form of:\n",
    "\n",
    "$$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{i}\\log(h_\\theta(x^{i}))+(1-y^{i})\\log(1-h_\\theta(x^{i}))$$\n",
    "\n",
    "Plugging in the two simplified expressions above, we obtain\n",
    "\n",
    "$$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^m \\left[-y^i(\\log ( 1+e^{-\\theta x^i})) + (1-y^i)(-\\theta x^i-\\log ( 1+e^{-\\theta x^i} ))\\right]$$,\n",
    "\n",
    "which can be simplified to:\n",
    "\n",
    "$$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^m \\left[y_i\\theta x^i-\\theta x^i-\\log(1+e^{-\\theta x^i})\\right]=-\\frac{1}{m}\\sum_{i=1}^m \\left[y_i\\theta x^i-\\log(1+e^{\\theta x^i})\\right]$$\n",
    "\n",
    "where the second equality follows from\n",
    "\n",
    "$$\n",
    "-\\left[ \\log e^{\\theta x^i}+\n",
    "\\log(1+e^{-\\theta x^i} )\n",
    "\\right]=-\\log(1+e^{\\theta x^i}). $$\n",
    "\n",
    "we used $\\log(x) + \\log(y) = log(x y)$\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "All you need now is to compute the partial derivatives of $(*)$ w.r.t. $\\theta_j$. As\n",
    "\n",
    "$$\\frac{\\partial}{\\partial \\theta_j}y_i\\theta x^i = y_ix^i_j $$\n",
    "\n",
    "$$\\frac{\\partial}{\\partial \\theta_j}\\log(1+e^{\\theta x^i}) = \\frac{x^i_je^{\\theta x^i}}{1+e^{\\theta x^i}} = x^i_jh_\\theta(x^i) $$\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "#### Full Calculation of the above line\n",
    "\n",
    "**Also note, Anything without θ is treated as constant:**\n",
    "\n",
    "![Imgur](https://imgur.com/u2GRqyY.png)\n",
    "\n",
    "So finally the thesis follows as.\n",
    "\n",
    "$$\\frac{\\partial}{\\partial\\theta_{j}}J(\\theta) =\n",
    "\n",
    "\\frac{\\partial}{\\partial\\theta_{j}} \\left[-\\frac{1}{m}\\sum_{i=1}^m \\left[y_i\\theta x^i-\\log(1+e^{\\theta x^i})\\right]\\right]\n",
    "\n",
    "= - \\frac{1}{m}\\sum_{i=1}^{m} y_ix^i_j - x^i_jh_\\theta(x^i)\n",
    "\n",
    "= \\frac{1}{m}\\sum_{i=1}^{m} \\left[ x^i_jh_\\theta(x^i) - y_ix^i_j  \\right]\n",
    "\n",
    " $$\n",
    "\n",
    "#### Hence Proved\n",
    "\n",
    "---\n",
    "\n",
    "#### Proof of\n",
    "\n",
    "## $$\\frac{\\partial}{\\partial \\theta_j}\\log(1+e^{\\theta x^i})=\\frac{x^i_je^{\\theta x^i}}{1+e^{\\theta x^i}}$$\n",
    "\n",
    "[By chain rule][1].\n",
    "\n",
    "$$(u(v))' = u(v)' * v'$$\n",
    "\n",
    "\n",
    "**Also note, Anything without θ is treated as constant:**\n",
    "\n",
    "For example:\n",
    "\n",
    "$$y = \\sin(3x - 5)$$\n",
    "$$u(v) = \\sin(3x - 5)$$\n",
    "$$v = (3x - 5)$$\n",
    "$$y' = \\sin(3x - 5)' = \\cos(3x - 5) * (3 - 0) = 3\\cos(3x-5)$$\n",
    "\n",
    "So, Regarding:\n",
    "\n",
    "$$\\frac{\\partial}{\\partial \\theta_j}\\log(1+e^{\\theta x^i})=\\frac{x^i_je^{\\theta x^i}}{1+e^{\\theta x^i}}$$\n",
    "\n",
    "$$u(v) = \\log(1+e^{\\theta x^i})$$\n",
    "\n",
    "$$v = 1+e^{\\theta x^i}$$\n",
    "\n",
    "$$\\frac{\\partial}{\\partial \\theta}\\log(1+e^{\\theta x^i}) = \\frac{\\partial}{\\partial \\theta}\\log(1+e^{\\theta x^i}) * \\frac{\\partial}{\\partial \\theta}(1+e^{\\theta x^i}) = \\frac{1}{1+e^{\\theta x^i}} * (0 + xe^{\\theta x^i}) = \\frac{xe^{\\theta x^i}}{1+e^{\\theta x^i}} $$\n",
    "\n",
    "Note that $$\\log(x)' = \\frac{1}{x}$$\n",
    "\n",
    "\n",
    "  [1]: https://en.wikipedia.org/wiki/Chain_rule"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Time and Space Complexity of Logistic Regression\n",
    "\n",
    "### During Train - Roughly order of $n * d$\n",
    "\n",
    "Where n is number of samples and d is dimensionality\n",
    "\n",
    "\n",
    "### During Run Time - Its $O(d)$\n",
    "\n",
    "Because, in run-time I already have the the W vector in store calculated during training phase. Now I just need to multiply the sample with that optimum W vector.\n",
    "\n",
    "$W = [w1 , w2...wd]$\n",
    "\n",
    "Meaning given a query point x_q I just need to do\n",
    "\n",
    "And if the multiplication result is > 0 the label is determined to be positive, else negative.\n",
    "\n",
    "# $$W^T * x_q$$\n",
    "\n",
    "![Imgur](https://imgur.com/NYrKtMQ.png)\n",
    "\n",
    "[source of image-Applied AI Course](https://www.appliedaicourse.com/course/11/Applied-Machine-learning-course)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "raw",
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn import linear_model\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import make_classification"
   ],
   "metadata": {}
  },
  {
   "cell_type": "raw",
   "source": [
    "# Generating some manual data which will be used in my custom Logistic Regression Training below\n",
    "X_train = np.array(\n",
    "    [[6, 7], [2, 4], [3, 6], [4, 7], [1, 6], [5, 2], [2, 0], [6, 3], [4, 1], [7, 2]]\n",
    ")\n",
    "\n",
    "y_train = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])\n",
    "\n",
    "\n",
    "X_test = np.array([[6, 1], [1, 3], [3, 1], [4, 5]])"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "After a substantial number of iterations, the learned w and b are used to classify a new sample $x'$ by means of the following equation:\n",
    "\n",
    "![Imgur](https://imgur.com/hnZMLMr.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "raw",
   "source": [
    "def sigmoid(input):\n",
    "    return 1.0/(1+ np.exp(-input))\n",
    "\n",
    "def compute_prediction(X, weights):\n",
    "    #Compute the prediction y_hat based on current weights\n",
    "    z = np.dot(X, weights)\n",
    "    predictions = sigmoid(z)\n",
    "    return predictions\n",
    "\n",
    "def predict(X, weights):\n",
    "    if X.shape[1] == weights.shape[0] - 1:\n",
    "        intercept = np.ones((X.shape[0], 1))\n",
    "        X = np.hstack((intercept, X))\n",
    "    return compute_prediction(X, weights)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The decision threshold is 0.5 by default, but it definitely can be other values. In a case\n",
    "where a false negative is, by all means, supposed to be avoided, for example, when\n",
    "predicting fire occurrence (the positive class) for alerts, the decision threshold can be\n",
    "lower than 0.5, such as 0.3, depending on how paranoid we are and how proactively\n",
    "we want to prevent the positive event from happening. On the other hand, when the\n",
    "false positive class is the one that should be evaded, for instance, when predicting\n",
    "the product success (the positive class) rate for quality assurance, the decision\n",
    "threshold can be greater than 0.5, such as 0.7, or lower than 0.5, depending on how\n",
    "high a standard you set.\n",
    "\n",
    "Noting again our Logistic Regression Vectorized Cost function is\n",
    "\n",
    "### $$ J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{i}\\log(h_\\theta(x^{i}))+(1-y^{i})\\log(1-h_\\theta(x^{i})) $$\n",
    "\n",
    "The $h_\\theta(x^{i})$ is the Predicted Y AND $y^{i}$ are the $y^{true}$ so the above formulae will be\n",
    "\n",
    "### $$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{true}\\log(Y_{pred})+(1-y^{true})\\log(1-Y_{pred})$$"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn import linear_model\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import make_classification\n",
    "\n",
    "# Creating some manual data on which the custom Logistice function that I will define below will train on\n",
    "X_train = np.array(\n",
    "    [[6, 7], [2, 4], [3, 6], [4, 7], [1, 6], [5, 2], [2, 0], [6, 3], [4, 1], [7, 2]]\n",
    ")\n",
    "\n",
    "y_train = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])\n",
    "\n",
    "\n",
    "X_test = np.array([[6, 1], [1, 3], [3, 1], [4, 5]])"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "raw",
   "source": [
    "# Compute the cost J(w)\n",
    "def compute_cost(X, y, weights):\n",
    "    predictions = compute_prediction(X,weights)\n",
    "    cost_logistic = np.mean(-y * np.log(predictions) - (1 - y) * np.log(1 - predictions) )\n",
    "    return cost_logistic    "
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "After a substantial number of iterations, the learned w and b are used to classify a new sample $x'$ by means of the following equation:\n",
    "\n",
    "![Imgur](https://imgur.com/hnZMLMr.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "def sigmoid(input):\n",
    "    return 1.0/(1+ np.exp(-input))\n",
    "\n",
    "def compute_prediction(X, weights):\n",
    "    #Compute the prediction y_hat based on current weights\n",
    "    z = np.dot(X, weights)\n",
    "    predictions = sigmoid(z)\n",
    "    return predictions\n",
    "\n",
    "def predict(X, weights):\n",
    "    if X.shape[1] == weights.shape[0] - 1:\n",
    "        intercept = np.ones((X.shape[0], 1))\n",
    "        X = np.hstack((intercept, X))\n",
    "    return compute_prediction(X, weights)\n"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "The decision threshold is 0.5 by default, but it definitely can be other values. In a case\n",
    "where a false negative is, by all means, supposed to be avoided, for example, when\n",
    "predicting fire occurrence (the positive class) for alerts, the decision threshold can be\n",
    "lower than 0.5, such as 0.3, depending on how paranoid we are and how proactively\n",
    "we want to prevent the positive event from happening. On the other hand, when the\n",
    "false positive class is the one that should be evaded, for instance, when predicting\n",
    "the product success (the positive class) rate for quality assurance, the decision\n",
    "threshold can be greater than 0.5, such as 0.7, or lower than 0.5, depending on how\n",
    "high a standard you set.\n",
    "\n",
    "Noting again our Logistic Regression Vectorized Cost function is\n",
    "\n",
    "## $$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{i}\\log(h_\\theta(x^{i}))+(1-y^{i})\\log(1-h_\\theta(x^{i}))$$\n",
    "\n",
    "The $h_\\theta(x^{i})$ is the Predicted Y AND $y^{i}$ are the $y^{true}$ so the above formulae will be\n",
    "\n",
    "## $$J(\\theta)=-\\frac{1}{m}\\sum_{i=1}^{m}y^{true}\\log(Y_{pred})+(1-y^{true})\\log(1-Y_{pred})$$"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "# Compute the cost J(w)\n",
    "def compute_cost(X, y, weights):\n",
    "    predictions = compute_prediction(X,weights)\n",
    "    cost_logistic = np.mean(-y * np.log(predictions) - (1 - y) * np.log(1 - predictions) )\n",
    "    return cost_logistic"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Training a logistic regression model using stochastic gradient descent\n",
    "\n",
    "In gradient descent-based logistic regression models, all training samples are used\n",
    "to update the weights in every single iteration. Hence, if the number of training\n",
    "samples is large, the whole training process will become very time-consuming and\n",
    "computationally expensive, as you just witnessed in our last example.\n",
    "Fortunately, a small tweak will make logistic regression suitable for large-sized\n",
    "datasets. For each weight update, only one training sample is consumed, instead\n",
    "of the complete training set. The model moves a step based on the error calculated\n",
    "by a single training sample. Once all samples are used, one iteration finishes. This\n",
    "advanced version of gradient descent is called stochastic gradient descent (SGD).\n",
    "Expressed in a formula, for each iteration, we do the following:\n",
    "\n",
    "![Imgur](https://imgur.com/X7TkU1q.png)\n",
    "\n",
    "![Imgur](https://imgur.com/eweOXsV.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "def sgd_update_weights(X_train, y_train, weights, learning_rate):\n",
    "    \"\"\"One weight update iteration: moving weights by one step based on each individual sample\n",
    "    Args:\n",
    "        X_train, y_train (numpy.ndarray, training data set)\n",
    "        weights (numpy.ndarray)\n",
    "        learning_rate (float)\n",
    "    Returns:\n",
    "        numpy.ndarray, updated weights\n",
    "    \"\"\"\n",
    "    for X_each, y_each in zip(X_train, y_train):\n",
    "        predictions = compute_prediction(X_each, weights)\n",
    "        delta_weights = X_each.T * (y_each - predictions)\n",
    "        weights += learning_rate * delta_weights\n",
    "    return weights"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "def train_logistic_regression_sgd(\n",
    "    X_train, y_train, max_iter, learning_rate, fit_intercept=False\n",
    "):\n",
    "    \"\"\"Train a logistic regression model via SGD\n",
    "    Args:\n",
    "        X_train, y_train (numpy.ndarray, training data set)\n",
    "        max_iter (int, number of iterations)\n",
    "        learning_rate (float)\n",
    "        fit_intercept (bool, with an intercept w0 or not)\n",
    "    Returns:\n",
    "        numpy.ndarray, learned weights\n",
    "    \"\"\"\n",
    "    # If fit_intercept, then initialize a intercept value \n",
    "    if fit_intercept:\n",
    "        intercept = np.ones((X_train.shape[0], 1))\n",
    "        X_train = np.hstack((intercept, X_train))\n",
    "    \n",
    "    # initialize weights with zeros\n",
    "    weights = np.zeros(X_train.shape[1])\n",
    "\n",
    "    for iteration in range(max_iter):\n",
    "        weights = sgd_update_weights(X_train, y_train, weights, learning_rate)\n",
    "        # Check the cost for every 2 (for example) iterations\n",
    "        if iteration % 2 == 0:\n",
    "            print(compute_cost(X_train, y_train, weights))\n",
    "    \n",
    "    return weights"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "# Train the SGD model based on the manual data defined at the top of this file\n",
    "learned_weights_after_sgd = train_logistic_regression_sgd(\n",
    "    X_train, y_train, max_iter=50, learning_rate=0.01, fit_intercept=True\n",
    ")\n",
    "print(\"weights from train_logistic_regression_sgd \", learned_weights_after_sgd)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "0.5925561752624662\n",
      "0.45124164011276724\n",
      "0.3605989658416807\n",
      "0.2992023417251532\n",
      "0.25540548197896334\n",
      "0.22277248630624552\n",
      "0.1975824037055633\n",
      "0.17757297044312131\n",
      "0.16130291507522063\n",
      "0.1478155104202422\n",
      "0.13645276699626\n",
      "0.126748150185035\n",
      "0.11836206176219918\n",
      "0.11104160058314942\n",
      "0.1045946418751617\n",
      "0.09887265505222564\n",
      "0.0937590206989488\n",
      "0.08916090457696889\n",
      "0.08500348983349584\n",
      "0.08122580769790162\n",
      "0.07777767362590254\n",
      "0.07461740193360676\n",
      "0.07171007779924374\n",
      "0.06902623438155288\n",
      "0.06654082848892706\n",
      "weights from train_logistic_regression_sgd  [ 0.1520753   0.89428318 -1.08484352]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "def sgd_update_weights(X_train, y_train, weights, learning_rate):\n",
    "    \"\"\"One weight update iteration: moving weights by one step based on each individual sample\n",
    "    Args:\n",
    "        X_train, y_train (numpy.ndarray, training data set)\n",
    "        weights (numpy.ndarray)\n",
    "        learning_rate (float)\n",
    "    Returns:\n",
    "        numpy.ndarray, updated weights\n",
    "    \"\"\"\n",
    "    for X_each, y_each in zip(X_train, y_train):\n",
    "        predictions = compute_prediction(X_each, weights)\n",
    "        delta_weights = X_each.T * (y_each - predictions)\n",
    "        weights += learning_rate * delta_weights\n",
    "    return weights"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "pred_y = predict(X_test, learned_weights_after_sgd)\n",
    "print(pred_y)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[0.98825985 0.09901903 0.85196306 0.15511817]\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "source": [
    "def train_logistic_regression_sgd(\n",
    "    X_train, y_train, max_iter, learning_rate, fit_intercept=False\n",
    "):\n",
    "    \"\"\"Train a logistic regression model via SGD\n",
    "    Args:\n",
    "        X_train, y_train (numpy.ndarray, training data set)\n",
    "        max_iter (int, number of iterations)\n",
    "        learning_rate (float)\n",
    "        fit_intercept (bool, with an intercept w0 or not)\n",
    "    Returns:\n",
    "        numpy.ndarray, learned weights\n",
    "    \"\"\"\n",
    "    # If fit_intercept, then initialize a intercept value \n",
    "    if fit_intercept:\n",
    "        intercept = np.ones((X_train.shape[0], 1))\n",
    "        X_train = np.hstack((intercept, X_train))\n",
    "    \n",
    "    # initialize weights with zeros\n",
    "    weights = np.zeros(X_train.shape[1])\n",
    "\n",
    "    for iteration in range(max_iter):\n",
    "        weights = sgd_update_weights(X_train, y_train, weights, learning_rate)\n",
    "        # Check the cost for every 2 (for example) iterations\n",
    "        if iteration % 2 == 0:\n",
    "            print(compute_cost(X_train, y_train, weights))\n",
    "    \n",
    "    return weights"
   ],
   "outputs": [],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "# Train the SGD model based on the manual data defined at the top of this file\n",
    "learned_weights_after_sgd = train_logistic_regression_sgd(\n",
    "    X_train, y_train, max_iter=50, learning_rate=0.01, fit_intercept=True\n",
    ")\n",
    "print(\"weights from train_logistic_regression_sgd \", learned_weights_after_sgd)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "0.5925561752624662\n",
      "0.45124164011276724\n",
      "0.3605989658416807\n",
      "0.2992023417251532\n",
      "0.25540548197896334\n",
      "0.22277248630624552\n",
      "0.1975824037055633\n",
      "0.17757297044312131\n",
      "0.16130291507522063\n",
      "0.1478155104202422\n",
      "0.13645276699626\n",
      "0.126748150185035\n",
      "0.11836206176219918\n",
      "0.11104160058314942\n",
      "0.1045946418751617\n",
      "0.09887265505222564\n",
      "0.0937590206989488\n",
      "0.08916090457696889\n",
      "0.08500348983349584\n",
      "0.08122580769790162\n",
      "0.07777767362590254\n",
      "0.07461740193360676\n",
      "0.07171007779924374\n",
      "0.06902623438155288\n",
      "0.06654082848892706\n",
      "weights from train_logistic_regression_sgd  [ 0.1520753   0.89428318 -1.08484352]\n"
     ]
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "pred_y = predict(X_test, learned_weights_after_sgd)\n",
    "print(pred_y)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[0.98825985 0.09901903 0.85196306 0.15511817]\n"
     ]
    }
   ],
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "nbconvert_exporter": "python",
   "version": "3.8.6-final"
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}