[Debugger Plugin] Exit TensorBoard on SIGINT even with debugger enabled (#975)

* [Debugger Plugin] Exit TensorBoard on SIGINT even with debugger enabled

Author: caisq

Diff for: tensorboard/plugins/debugger/README.md

@@ -16,7 +16,7 @@ This dashboard is in its **alpha release**. Some features are not yet fully func
 
 To enable the debugger dashboard, pass the `debugger_port` flag to TensorBoard. TensorBoard will then both receive gRPC messages from and issue gRPC messages to model logic via this port.
 
-This command demonstrates how to set the debugger port to 6064. 
+This command demonstrates how to set the debugger port to 6064.
 
 ```
 tensorboard \
@@ -166,10 +166,21 @@ For example, suppose a tensor has a shape of (500, 100), applying a slicing of `
 
 For each tensor, the time axis (history of the tensor's execution) is treated as an 1D array. Numpy-style slicing can be applied to time. For example, the default slicing of `-1` selects the most recent value. However, if the user changes that slicing parameter to `:`, the full history of the tensor will be shown (and the rank of the tensor being visualized is increased by 1).
 
-# Limitations
+# Frequently Asked Questions (FAQ)
 
-* Hitting Ctrl+C (issuing a SIGINT signal) might fail to terminate execution for a model that is instrumented with
-  `TensorBoardDebugWrapperSession` or its corresponding hook. The same limitation may be present in the tensorboard
-  process as well. In those cases, the user must manually
-  [kill](https://www.linux.com/learn/intro-to-linux/2017/5/how-kill-process-command-line) the processes.
-* The debugger dashboard does not yet support multiple users debugging at once.
+## Q: How to exit debugging?
+
+Answer: Follow these steps to interrupt your TensorFlow program being debugged
+and the TensorBoard process running the debugger plugin (*in that order*):
+
+1. Send `SIGINT` to the TensorFlow program being debugged, e.g., by using
+   `Ctrl+C`.
+2. Send `SIGINT` to the TensorBoard process running the debugger plugin, e.g.,
+   by using `Ctrl+C`.
+
+# Limitations and Known Issues
+
+The debugger plugin has the following limitations and known issues. We plan to
+fix them in future releases.
+
+* The debugger plugin does not yet support multiple users debugging at once.

Diff for: tensorboard/plugins/debugger/interactive_debugger_plugin.py

@@ -20,9 +20,11 @@
 
 import json
 import platform
+import signal
 import sys
 import threading
 
+from six.moves import xrange  # pylint:disable=redefined-builtin
 import tensorflow as tf
 from werkzeug import wrappers
 
@@ -69,6 +71,7 @@ def __init__(self, context):
     """
     del context  # Unused.
     self._debugger_data_server = None
+    self._server_thread = None
     self._grpc_port = None
 
   def listen(self, grpc_port):
@@ -96,8 +99,31 @@ def listen(self, grpc_port):
         interactive_debugger_server_lib.InteractiveDebuggerDataServer(
             self._grpc_port))
 
-    threading.Thread(target=self._debugger_data_server.
-                     start_the_debugger_data_receiving_server).start()
+    self._server_thread = threading.Thread(
+        target=self._debugger_data_server.run_server)
+    self._server_thread.start()
+
+    signal.signal(signal.SIGINT, self.signal_handler)
+    # Note: this is required because of a wontfix issue in grpc/python 2.7:
+    #   https://github.com/grpc/grpc/issues/3820
+
+  def signal_handler(self, unused_signal, unused_frame):
+    if self._debugger_data_server and self._server_thread:
+      print('Stopping InteractiveDebuggerPlugin...')
+      # Enqueue a number of messages to the incoming message queue to try to
+      # let the debugged tensorflow runtime proceed past the current Session.run
+      # in the C++ layer and return to the Python layer, so the SIGINT handler
+      # registered there may be triggered.
+      for _ in xrange(len(self._debugger_data_server.breakpoints) + 1):
+        self._debugger_data_server.put_incoming_message(True)
+      try:
+        self._debugger_data_server.stop_server()
+      except ValueError:
+        # In case the server has already stopped running.
+        pass
+      self._server_thread.join()
+      print('InteractiveDebuggerPlugin stopped.')
+    sys.exit(0)
 
   def get_plugin_apps(self):
     """Obtains a mapping between routes and handlers.

Diff for: tensorboard/plugins/debugger/interactive_debugger_server_lib.py

@@ -515,15 +515,6 @@ def __init__(self, receive_port):
     grpc_debug_server.EventListenerBaseServicer.__init__(
         self, receive_port, curried_handler_constructor)
 
-  def start_the_debugger_data_receiving_server(self):
-    """Starts the HTTP server for receiving health pills at `receive_port`.
-
-    After this method is called, health pills issued to host:receive_port
-    will be stored by this object. Calling this method also creates a file
-    within the log directory for storing health pill summary events.
-    """
-    self.run_server()
-
   def SendTracebacks(self, request, context):
     self._source_manager.add_graph_traceback(request.graph_version,
                                              request.graph_traceback)