Weight_Decay_Learning_Rate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weight decay in PyTorch and its relation with Learning Rate | L2 Regularization\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weight decay is a regularization technique by adding a small penalty, usually the L2 norm of the weights (all the weights of the model), to the loss function.\n",
    "\n",
    "```\n",
    "### loss = loss + weight decay parameter * L2 norm of the weights\n",
    "```\n",
    "\n",
    "![Imgur](https://imgur.com/262wI66.png)\n",
    "\n",
    "And the weights should themselves be updated as follows\n",
    "\n",
    "```\n",
    "w[t+1] = w[t] - learning_rate * dw - weight_decay * w\n",
    "\n",
    "```\n",
    "We have our loss function, now we add the sum of the squared norms from our weight matrices and multiply this by a constant denoted by lambda. This lambda here is called the regularization parameter and this is another hyperparameter that we’ll have to choose .\n",
    "\n",
    "If we set lambda to be a relatively large number then it would incentivize the model to set the weight close to 0 because the objective of SGD is to minimize the loss function and remember our original loss function is now being summed with the sum of the squared matrix norms.\n",
    "\n",
    "Some people prefer to only apply weight decay to the weights and not the bias. PyTorch applies weight decay to both weights and bias.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "### How do we use weight decay?\n",
    "\n",
    "To use weight decay, we can simply define the weight decay parameter in the `torch.optim.SGD` optimizer or the `torch.optim.Adam` optimizer. Here we use 1e-4 as a default for weight_decay\n",
    "\n",
    "```py\n",
    "\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, weight_decay=1e-4)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)\n",
    "\n",
    "```\n",
    "\n",
    "--------------------\n",
    "\n",
    "### Why do we use weight decay?\n",
    "\n",
    "1. To prevent overfitting.\n",
    "\n",
    "2. To keep the weights small and avoid exploding gradient. Because the L2 norm of the weights are added to the loss, each iteration of your network will try to optimize/minimize the model weights in addition to the loss. This will help keep the weights as small as possible, preventing the weights to grow out of control, and thus avoid exploding gradient.\n",
    "\n",
    "=================================================================\n",
    "\n",
    "### Disable weight-decay in few Layers OR to set different values for different layers\n",
    "\n",
    "PyTorch applies weight decay to both weights and bias.\n",
    "\n",
    "But some think that decay should not be applied to Bias, since those parameters are less likely to overfit. Furthermore, the decay should also not be applied to parameters with a shape of one, meaning the parameter is a vector and no matrix which is quite often for normalization modules, like batch-norm, layer-norm or weight-norm.\n",
    "\n",
    "With the introduction of the function named_parameters(), we also get a name along with the parameter value. For standard layers, biases are named as “bias” and combined with the shape, we can create two parameter lists, one with weight_decay and the other without it. Furthermore, we can easily use a skip_list to manually disable weight_decay for some layers, like embedding layers.\n",
    "\n",
    "```py\n",
    "def custom_weight_decay(net, l2_value, skip_list=()):\n",
    "    decay, no_decay = [], []\n",
    "    for name, param in net.named_parameters():\n",
    "        if not param.requires_grad: continue # frozen weights\n",
    "    if len(param.shape) == 1 or name.endswith(\".bias\") or name in skip_list:\n",
    "        no_decay.append(param)\n",
    "    else: decay.append(param)\n",
    "\n",
    "    return [{'params': no_decay, 'weight_decay': 0.}, {'params': decay, 'weight_decay': l2_value}]\n",
    "\n",
    "\n",
    "# and the returned list is passed to the optimizer:\n",
    "\n",
    "params = custom_weight_decay(pytorch_neural_net, 2e-5)\n",
    "sgd = torch.optim.SGD(params, lr=0.05)\n",
    "\n",
    "```\n",
    "\n",
    "Check - https://pytorch.org/docs/stable/optim.html#per-parameter-options\n",
    "\n",
    "Per-parameter options\n",
    "Optimizer s also support specifying per-parameter options. To do this, instead of passing an iterable of Variable s, pass in an iterable of dict s. Each of them will define a separate parameter group, and should contain a params key, containing a list of parameters belonging to it. Other keys should match the keyword arguments accepted by the optimizers, and will be used as optimization options for this group.\n",
    "\n",
    "---\n",
    "\n",
    "### Custom weight decay operation, not effecting grad values.\n",
    "\n",
    "```py\n",
    "\n",
    "import torch\n",
    "\n",
    "\n",
    "def weight_decay(optimizer, wd):\n",
    "    \"\"\"\n",
    "    Custom weight decay operation, not effecting grad values.\n",
    "    https://www.fast.ai/2018/07/02/adam-weight-decay/\n",
    "    \"\"\"\n",
    "    for group in optimizer.param_groups:\n",
    "        for param in group['params']:\n",
    "            current_lr = group['lr']\n",
    "            param.data = param.data.add(-wd * group['lr'], param.data)\n",
    "    return optimizer, current_lr\n",
    "\n",
    "```\n",
    "\n",
    "--------------------\n",
    "\n",
    "## BONUS\n",
    "\n",
    "What does the `model.parameters()` include?\n",
    "\n",
    "`model.parameters()` stores the weight and bias values of the model. It is given as an argument to an optimizer to update the weight and bias values of the model with one line of code optimizer.step()\n",
    "\n",
    "---\n",
    "\n",
    "## Get the [names of parameters](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.named_parameters)\n",
    "\n",
    "```py\n",
    "\n",
    "import torch\n",
    "import torchvision\n",
    "from torch import nn\n",
    "from torchvision import models\n",
    "\n",
    "my_model = models.resnet50(pretrained=False)\n",
    "\n",
    "for name, parameter in my_model.named_parameters():\n",
    "    print(name)\n",
    "\n",
    "```\n",
    "\n",
    "Output\n",
    "\n",
    "```\n",
    "conv1.weight\n",
    "bn1.weight\n",
    "bn1.bias\n",
    "layer1.0.conv1.weight\n",
    "layer1.0.bn1.weight\n",
    "layer1.0.bn1.bias\n",
    "\n",
    "...\n",
    "...\n",
    "...\n",
    "\n",
    "```\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Weight Decay Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original weights tensor([[0.4386, 0.0597, 0.3980, 0.7380, 0.1825],\n",
      "        [0.1755, 0.5316, 0.5318, 0.6344, 0.8494],\n",
      "        [0.7245, 0.6110, 0.7224, 0.3230, 0.3618],\n",
      "        [0.2283, 0.2937, 0.6310, 0.0921, 0.4337]], dtype=torch.float64,\n",
      "       requires_grad=True)\n",
      "0 weight decay tensor([[ 0.2489, -0.1300,  0.2084,  0.5483, -0.0072],\n",
      "        [ 0.0653,  0.4214,  0.4217,  0.5243,  0.7393],\n",
      "        [ 0.5694,  0.4559,  0.5674,  0.1679,  0.2067],\n",
      "        [ 0.0317,  0.0972,  0.4345, -0.1044,  0.2372]], dtype=torch.float64,\n",
      "       requires_grad=True)\n",
      "Reset Original weights tensor([[0.4386, 0.0597, 0.3980, 0.7380, 0.1825],\n",
      "        [0.1755, 0.5316, 0.5318, 0.6344, 0.8494],\n",
      "        [0.7245, 0.6110, 0.7224, 0.3230, 0.3618],\n",
      "        [0.2283, 0.2937, 0.6310, 0.0921, 0.4337]], dtype=torch.float64,\n",
      "       requires_grad=True)\n",
      "1 weight decay tensor([[ 0.2050, -0.1360,  0.1686,  0.4745, -0.0254],\n",
      "        [ 0.0478,  0.3683,  0.3685,  0.4608,  0.6544],\n",
      "        [ 0.4969,  0.3948,  0.4951,  0.1356,  0.1705],\n",
      "        [ 0.0089,  0.0678,  0.3714, -0.1136,  0.1938]], dtype=torch.float64,\n",
      "       requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "np.random.seed(123)\n",
    "np.set_printoptions(8, suppress=True)\n",
    "\n",
    "x_numpy = np.random.random((3, 4)).astype(np.double)\n",
    "w_numpy = np.random.random((4, 5)).astype(np.double)\n",
    "\n",
    "x_torch = torch.tensor(x_numpy, requires_grad=True)\n",
    "w_torch = torch.tensor(w_numpy, requires_grad=True)\n",
    "\n",
    "#######################################################\n",
    "\n",
    "print('Original weights', w_torch)\n",
    "\n",
    "lr = 0.1\n",
    "sgd = torch.optim.SGD([w_torch], lr=lr, weight_decay=0)\n",
    "\n",
    "y_torch = torch.matmul(x_torch, w_torch)\n",
    "loss = y_torch.sum()\n",
    "\n",
    "sgd.zero_grad()\n",
    "loss.backward()\n",
    "sgd.step()\n",
    "\n",
    "w_grad = w_torch.grad.data.numpy()\n",
    "print('0 weight decay', w_torch)\n",
    "\n",
    "\n",
    "#######################################################\n",
    "\n",
    "w_torch = torch.tensor(w_numpy, requires_grad=True)\n",
    "\n",
    "print('Reset Original weights', w_torch)\n",
    "\n",
    "sgd = torch.optim.SGD([w_torch], lr=lr, weight_decay=1)\n",
    "\n",
    "y_torch = torch.matmul(x_torch, w_torch)\n",
    "loss = y_torch.sum()\n",
    "\n",
    "sgd.zero_grad()\n",
    "loss.backward()\n",
    "sgd.step()\n",
    "\n",
    "w_grad = w_torch.grad.data.numpy()\n",
    "print('1 weight decay', w_torch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### As you can see, the weights are smaller when I use weight_decay=1 compared to weight_decay=0\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "f9f85f796d01129d0dd105a088854619f454435301f6ffec2fea96ecbd9be4ac"
  },
  "kernelspec": {
   "display_name": "Python 3.9.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}