Some essential feature request for tensorboard

[vishal733]

Tensorboard is amazing. But it could be even more amazing. Here are some thoughts:

1. When I create large projects with tensorboard, the memory usage goes to something like 16 GB (for a deep learning project involving computer vision). And it also keeps growing with time, and eventually crashes my system. Tensorboard is probably loading a lot of things without they even queried in the UI, and that could be leading to such high memory usage. This needs to be optimized properly, to enable us to work with large projects in tensorboard. [I have seen bug reports since at least 2017, and it's yet to be resolved].

2. One of the reasons why we end up with large memory usage is because things like images are written inside the tensorboard logs, and loaded into memory when tensorboard is launched.
   Just consider this alternate approach:

• Tensorboard doesn't store the images inside its logs. Rather just stores some URLs to the images.
• When an image is requested, it actually displays those URLs, rather than loading it from memory.

Advantages:

• Much lesser memory requirement for tensorboard (dynamic loading by loading of URLs).
• The same image can be re-used multiple times in the interface, without the user having to write the image multiple times to tensorboard.
• Could also enable custom workflows like, say I want to dynamically view only the misclassifications from class A to class B. And then switch to viewing misclassifications from class A to class C. Writing all of them as images becomes very large in size when we are considering >15-20 classes.

3. Please start supporting HTML rendering in tensorboard at the least (atleast things like image tags). That way I could just host some images using Python Flask server, and then use tensorboard to display them via links.

The argument about this being a security issue seems unreasonable to me. First of all, who is so much bothered about security when it comes to tensorboard? What we care about is using it to access the progress of our deep learning project.
And secondly, I'm sure it would be possible to make the system quite secure even if you were to enable HTML rendering inside of tensorboard.

4. Support for displaying tables like confusion matrix (without having to convert them to images).
   Image are huge in size, just provide native support for tables (again, you could reduce your development support by enabling some sort of HTML rendering support inside tensorboard instead).

[GalOshri]

I'm glad to hear you are enjoying TensorBoard, and thank you for the feedback! Responding to your suggestions:

1. This is tracked by Large memory consumption 0.4 #766. There were a few updates there several months ago. The team identified and resolved one issue, but it does look like the problem was not fully solved. Any chance you can share the problematic logs?

2. Do you want this to enable viewing images that were not generated during the training process, or only because it might make it more efficient to view images from training? There are several ways of addressing the latter, but I'm curious if the former is useful.

Note that this won't always work (TensorBoard needs to work offline in some settings. Additionally, this would just add extra steps to visualizing images generated during model training).

3. Security is definitely important :). TensorBoard is used in various settings (e.g. local laptop, cloud, ...) and users don't want their data and experiment results to be exposed.

It is possible to create your own plugins for TensorBoard. You should be able to create a dashboard that displays custom HTML. Note that there are limitations due to security constraints.

4. Confusion matrices are definitely something we want to support more natively in TensorBoard! We are starting to think about a dashboard that might be the right place to display them, but it is still at a very exploratory stage.