Tensorflow_Mixed_Precision_Training_CIFAR10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow Mixed Precision Training — CIFAR10 dataset Implementation\n",
    "\n",
    "From TensorFlow 2.1, it has allowed for mixed-precision training, making use of the Tensor Cores available in the most recent NVidia GPUs.\n",
    "\n",
    "## What is Mixed Precision Training\n",
    "\n",
    "Mixed precision training is a technique used in training a large neural network where the model’s parameter are stored in different datatype precision (FP16 vs FP32 vs FP64). It offers significant performance and computational boost by training large neural networks in lower precision formats. With release of 30X series of GPUs it becomes even more important to utilize these features.\n",
    "\n",
    "One way to describe mixed-precision training, in TensorFlow could go like this: **MPT (Mixed Precision Training)** lets you train models where the weights are of type float32 or float64, as usual (for reasons of numeric stability), but the data — the tensors pushed between operations — have lower precision, namely, 16bit (float16).\n",
    "\n",
    "\n",
    "For instance, In Pytorch, the single precision float mean float32 and by default the parameters takes float32 datatype. Now if we have a parameter (W) which could be stored in FP16 while ensuring that no task specific accuracy is affected by this movement between precision, then why should we use FP32 or FP64?\n",
    "\n",
    "**Notations**\n",
    "\n",
    "**F16** — Half-Precision, 16bit Floating Point-occupies 2 bytes of memory\n",
    "\n",
    "**FP32** — Single-Precision, 32bit Floating Point-occupies 4 bytes of memory\n",
    "\n",
    "**FP64**— Double-Precision, 64bit Floating Point-occupies 8 bytes of memory\n",
    "\n",
    "Since the introduction of Tensor Cores in the Volta and Turing architectures (NVIDIA), significant training speedups are experienced by switching to mixed precision — up to 3x overall speedup on the most arithmetically intense model architectures. The ability to train deep learning networks with lower precision was introduced in the Pascal architecture and first supported in CUDA® 8 in the NVIDIA Deep Learning SDK.\n",
    "\n",
    "Some of the benefits are, faster Model training with compatible GPU, and because it use 16bits it will allow to use larger batch-size- With mixed-precision training, I can run batches of size 256 in cases where without using mixed-precision, I get an out-of-memory error pretty fast.\n",
    "\n",
    "\n",
    "### Fused Multiply-Add (FMA)\n",
    "\n",
    "To understand Mixed Precision Training we also need to get this concept of Fused Multiply-Add.\n",
    "\n",
    "Fused Multiply-Add is a type of multiply-accumulate operation. In multiply-accumulate, operands are multiplied and then added to an accumulator keeping track of the running sum.\n",
    "\n",
    "If “fused”, the whole “multiply-then-add” operation is performed with a single rounding at the end (as opposed to rounding once after the multiplication, and then again after the addition). Usually, this results in higher accuracy.\n",
    "\n",
    "For CPUs, FMA was introduced concurrently with AVX2. FMA can be performed on scalars or on vectors, “packed” in the way described as above.\n",
    "\n",
    "Why did we say this was so interesting to data scientists? Well, a lot of operations — dot products, matrix multiplications, convolutions — involve multiplications followed by additions.\n",
    "\n",
    "So “Matrix multiplication” here actually has us leave the realm of CPUs and jump to GPUs instead because what MPT does is make use of the new-ish NVidia Tensor Cores that extend FMA from scalars/vectors to matrices.\n",
    "\n",
    "![](https://imgur.com/MYZoDM2.png)\n",
    "\n",
    "From the very nice [**paper**](https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf) from NVIDIA.\n",
    "\n",
    "![](https://imgur.com/lWdgDNG.png)\n",
    "\n",
    "Basically, The operation takes place on 4x4 matrices; multiplications happen on 16-bit operands while the final result could be 16-bit or 32-bit.\n",
    "\n",
    "Tensor Cores operate on FP16 input data with FP32 accumulation. The FP16 multiply results in a full precision product that is then accumulated using FP32 addition with the other intermediate products for a 4x4x4 matrix multiply (see Figure 9). In practice, Tensor Cores are used to perform much larger 2D or higher dimensional matrix operations, built up from these smaller elements.\n",
    "\n",
    "![](https://imgur.com/Hxu98m3.png)\n",
    "\n",
    "\n",
    "### Python Impementation with CIFAR 10 Dataset\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "\n",
    "def create_model():\n",
    "    model = keras.Sequential([\n",
    "        keras.layers.Flatten(input_shape=(32, 32, 3)),\n",
    "        keras.layers.Dense(3000, activation='relu'),\n",
    "        keras.layers.Dense(1000, activation='relu'),\n",
    "        # keras.layers.Dense(10, activation='sigmoid'),\n",
    "        keras.layers.Dense(10,),\n",
    "        keras.layers.Activation('sigmoid', dtype='float32'),\n",
    "    ])\n",
    "\n",
    "    model.compile(optimizer='SGD',\n",
    "                  loss='categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "tf.keras.mixed_precision.set_global_policy('mixed_float16')\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()\n",
    "\n",
    "# There are 10 image classes\n",
    "classes = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n",
    "\n",
    "X_train_scaled = X_train / 255\n",
    "X_test_scaled = X_test / 255\n",
    "\n",
    "y_train_categorical = keras.utils.to_categorical(y_train, num_classes= 10, dtype='float')\n",
    "y_test_categorical = keras.utils.to_categorical(y_test, num_classes= 10, dtype='float')\n",
    "\n",
    "with tf.device('/GPU:0'):\n",
    "    model = create_model()\n",
    "    model.fit(X_train_scaled, y_train_categorical, epochs=50)\n",
    "\n",
    "model.evaluate(X_test_scaled, y_test_categorical)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**For the above code — why I am using dtype=’float32’ in the last Activation layer**[](#why-i-am-using-dtypefloat32-in-the-last-activation-layer \"Permanent link\")\n",
    "\n",
    "According to the official guide from Tensorflow, To use mixed precision properly, your sigmoid activation at the end of the model should be float32. Because we set the policy mixed_float16, the activation’s compute_dtype is float16.\n",
    "\n",
    "Thus, we had to overwrite the policy for the last layer to float32.\n",
    "\n",
    "keras.layers.Activation\n",
    "\n",
    "**In above code while scaling Images why did I divide by 255**\n",
    "\n",
    "When using the image as it is and passing through a Deep Neural Network, the computation of high numeric values may become more complex.\n",
    "\n",
    "To reduce this we can normalize the values to range from 0 to 1.\n",
    "\n",
    "In this way, the numbers will be small and the computation becomes easier and faster. As the pixel values range from 0 to 256, apart from 0 the range is 255. So dividing all the values by 255 will convert it to range from 0 to 1.\n",
    "\n",
    "Thanks for reading !!"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"
  },
  "kernelspec": {
   "display_name": "Python 3.9.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}