[visualize-convnet] Add new example: visualize-convnet

[caisq]

This TensorFlow.js example demonstrates some techniques of visualizing
the internal workings of a convolutional neural network (convnet), including:

• Finding what convolutional layers' filters are sensitive to after
  training: calculating maximally-activating input image for
  convolutional filters through gradient ascent in the input space.
• Getting the internal activation of a convnet by uisng the
  functional model API of TensorFlow.js
• Finding which part of an input image is most relevant to the
  classification decision made by a convnet (VGG16 in this case),
  using the gradient-based class activation map (CAM) approach.

Example screenshots:

---

This change is 

[tafsiri]

Very very cool example! (i'm still running it, but thought i'd send this initial review on since i had one or two big suggestions for your consideration).

Reviewed 18 of 18 files at r1.
Reviewable status: 0 of 1 approvals obtained (waiting on @caisq and @tafsiri)

---

visualize-convnet/main.js, line 42 at r1 (raw file):

const imagenetClasses = require('./imagenet_classes');

/**

What do you think of splitting the 3 vis techniques into 3 different files that can be called from this (and that maybe have shared functions in a file). I think from an educational point it may help this file feel less overwhelming and allow easier inspection of each individual algorithm. (also while reviewing :).

I really like all the comments and docs in this file! 👍

---

visualize-convnet/main.js, line 165 at r1 (raw file):

 *           and width of the layer's filter outputs.
 */
async function writeInternalActivationAndGetOutput(

I'd maybe separate the writing of the activation to a file from its computation, again just to help people see where the meat of the technique is. It would also match the other functions a bit more closely imo

---

visualize-convnet/package.json, line 2 at r1 (raw file):

{
  "name": "tfjs-examples-mobilenet",

update

---

visualize-convnet/README.md, line 27 at r1 (raw file):

1. install the necessary Python dependencies
2. download and convert the VGG16 model to TensorFlow.js format

Could we skip this step by hosting the converted model and weights? Having it this way assumes a working python setup and modifies the user environment.

---

visualize-convnet/utils.js, line 97 at r1 (raw file):

// 64 x 3 RGB colormap.
const RGB_COLORMAP = [

I think one would wonder why these values for the applyColorMap function below. Is this based on something in particular, would be nice to describe at a perceptual level how this transforms 1 channel images to color images. Also why do we want to do this conversion.

[caisq]

Reviewable status: 1 change requests, 0 of 1 approvals obtained (waiting on @tafsiri)

---

visualize-convnet/main.js, line 42 at r1 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

What do you think of splitting the 3 vis techniques into 3 different files that can be called from this (and that maybe have shared functions in a file). I think from an educational point it may help this file feel less overwhelming and allow easier inspection of each individual algorithm. (also while reviewing :).

I really like all the comments and docs in this file! 👍

SG. Done creating two new files:

• filters.js for getting internal activations and calculating maximal activating input images
• cam.js for calculating maximu

I think splitting the two functions in filters.js would be too fine-grained

---

visualize-convnet/main.js, line 165 at r1 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

I'd maybe separate the writing of the activation to a file from its computation, again just to help people see where the meat of the technique is. It would also match the other functions a bit more closely imo

It's now seperated from inputGradientAscent(), which lives in filters.js. This function now continues to live in main.js.

---

visualize-convnet/package.json, line 2 at r1 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

update

Changed to "tfjs-examples-visualize-convnet"

---

visualize-convnet/README.md, line 27 at r1 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

Could we skip this step by hosting the converted model and weights? Having it this way assumes a working python setup and modifies the user environment.

The model is huge (500 MB) and I'm a little nervous about hosting it ourselves just for this one example.

I added a comment here that the user would typically want to do it in a virtualenv or pipenv to avoid modifying the Python environment. Also note that if the Python requirements (keras, tensorflow) are already satisfied, the pip install steps will be no-op.

---

visualize-convnet/utils.js, line 97 at r1 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

I think one would wonder why these values for the applyColorMap function below. Is this based on something in particular, would be nice to describe at a perceptual level how this transforms 1 channel images to color images. Also why do we want to do this conversion.

Added comments to explain that.

[caisq]

Gentle ping.

[tafsiri]

Thanks @caisq I left a few more small comments, but nothing that I need to re-review! Cool stuff!

Reviewed 9 of 9 files at r2.
Reviewable status: 1 change requests, 0 of 1 approvals obtained (waiting on @caisq and @tafsiri)

---

visualize-convnet/package.json, line 22 at r2 (raw file):

  },
  "devDependencies": {
    "@tensorflow/tfjs-node": "0.2.1",

I think you should move the three dependencies for node back to dependencies rather than devDependencies. They are full dependencies for the desktop portion.

---

visualize-convnet/utils.js, line 97 at r1 (raw file):

Previously, caisq (Shanqing Cai) wrote…

Added comments to explain that.

Thanks, I think you could add some more comments for this though, e.g. where does this method come from? How does it work? There are potentially lots of ways to convert 1 channel into 3 channel images but its not clear why this one is meaningful

[caisq]

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @tafsiri)

---

visualize-convnet/package.json, line 22 at r2 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

I think you should move the three dependencies for node back to dependencies rather than devDependencies. They are full dependencies for the desktop portion.

Good point. Done.

---

visualize-convnet/utils.js, line 97 at r1 (raw file):

Previously, tafsiri (Yannick Assogba) wrote…

Thanks, I think you could add some more comments for this though, e.g. where does this method come from? How does it work? There are potentially lots of ways to convert 1 channel into 3 channel images but its not clear why this one is meaningful

Done.