HParam table view sort not working #3041
Comments
|
@patzm, |
|
@rmothukuru, yes I am using the Direction drop down menu to select one of the two sorting direction choices. Selection of sorting column and direction: The very same column in the table view: As you can see, these values are clearly not sorted. Sadly, I can't share the whole screenshot. If this is sufficient, I would like to avoid the effort of writing a reproducible script. Hopefully, this already helps you to narrow down the issue. I refreshed the site as well with the browser refresh button, and the TensorBoard refresh trigger (upper right). Neither of which helped. Also, I tried various versions of TensorBoard: 1.15.0, 2.0.0, 2.0.2, 2.1.0. It is also noteworthy, that the |
|
Hi @patzm! Thanks for the report and detailed background information. I’m assuming from your screenshot that for each trial you have runs # Context: <https://github.com/tensorflow/tensorboard/issues/3041>
import os
import random
import tensorflow as tf
from tensorboard.plugins.hparams import api as hp
HP_LR = hp.HParam("learning_rate", hp.RealInterval(0.01, 0.1))
HP_OPTIMIZER = hp.HParam("optimizer", hp.Discrete(["adam", "sgd"]))
SESSIONS = ("train", "eval")
METRIC_LOSS = "loss"
METRIC_ACC1 = "accuracy/top_1"
METRIC_ACC5 = "accuracy/top_5"
NUM_RUNS = 50
NUM_STEPS = 10
BASE_LOGDIR = "logs"
def main():
rng = random.Random(0)
for i in range(NUM_RUNS):
session_dir = os.path.join(BASE_LOGDIR, "%03d" % i)
with tf.summary.create_file_writer(session_dir).as_default():
hp.hparams(
{h: h.domain.sample_uniform(rng) for h in (HP_LR, HP_OPTIMIZER)}
)
for session in SESSIONS:
logdir = os.path.join(session_dir, session)
with tf.summary.create_file_writer(logdir).as_default():
for step in range(NUM_STEPS):
tf.summary.scalar(METRIC_LOSS, rng.random(), step=step)
tf.summary.scalar(METRIC_ACC1, rng.random(), step=step)
tf.summary.scalar(METRIC_ACC5, rng.random(), step=step)
if __name__ == "__main__":
main()Changing the sort key and direction always seems to work fine for me. Can you check and see if the data generated by that script works in your FWIW, I notice from your diagnostics that you’re running TensorBoard 2.0 |
|
(NB: I just edited the script in the above comment.) |
wchargin
added
stat:awaiting response
type:bug
and removed
core:frontend
stat:awaiting tensorflower
type:support
labels
I deleted the whole TensorBoard setup after submitting the issue and started from scratch. I created 3 virtual environments:
This didn't solve the sorting problem. So your intuition was right 😉 . |
|
I ran your script with a minor change. I inserted the following and replaced all import tensorflow as tf
tf.compat.v1.enable_eager_execution()
tf_v2 = tf.compat.v2This allowed me to run your script both in TensorFlow 1.15 and TensorFlow 2.0. In both cases, TensorBoard (Hparams) with the respectively matching version worked as expected. I.e. I can reproduce your results. Then I was running the same script with the
and it also worked in both cases. |
|
I just created a debug repo in which I just pushed the slightly modified demo script in patzm/tensorboard-3041@14e5a74. I will write a minimal example for my use case there. I am still working with TensorFlow 1.15 and relying on the non-eager, graph-based stuff. I wrote a custom |
Yes, almost: the |
|
Ok @wchargin, can you try to run the two scripts in my repo? I added small
I would suggest that you do 4 runs. Run your script twice, once for each of the two I think I have tracked down the issue to two things:
If those two things are in place, I observe the following:
I am curious to see if you can reproduce this. Sadly, this isn't exactly what this issue reported initially, but I have a strong feeling that this is related. |
|
I just ran the same thing with the |
|
So I'm experiencing this behavior as well. However for me the sorting is applied to the field that is sequentially following the one I selected. This is using Tensorboard 2.1.0 & Tensorflow 2.1.0 |
|
I experienced that as well sometimes. My observation was that it seemed to select the previously selected column. Maybe it is a different issue though. |
|
I'm experiencing similar behavior. So far, sorting never changes no matter what I do. It always remains sorted by Trial ID, ascending. The issue remains even when I have only eg 5 trials, regardless of the sorting direction I choose, the sorting column I choose, and the browser (Firefox v71.0 or Chrome 68.0.3440). I am using DiagnosticsDiagnostics output |
|
@wchargin, how to you want to proceed on this? The issue still persists in all deployment cases I am dealing with. Would love to solve this / see it solved. |
|
Hi @patzm—if you’re still experiencing this issue, could you check Changing the sort order actually triggers a network request (because the Could you share a few details about your network topology? Where your |
|
I'm having the same issue, sorting mostly doesn't work at all, or else it is very random. I tested this on runs with 2 trials and 1 epoch (stored locally) as well as 8 trials and 5 epochs (on google cloud). I'm using Chrome on MacOS 10.15.4 with package versions: Running TensorBoard on localhost. |
Hmm, interesting. If you’re up for a bit of debugging, would you mind
My suspicion is that it’s this |
|
I am having a very similar problem, and it seems to only be a problem with some columns, namely the metrics columns. However, this depends on the total number of columns
The requests to the server are fine in terms of completion. I will note that this problem also occurs when trying to filter values based on min-max range for a column where the sorting doesn't work. Interestingly enough, if you filter values before trying to sort, it works. If you try after, it no longer works, likely due to the client-side bugs that are messing with the table. |
|
I observe this issue but only when I have at least one hparam that is a string. That string creates an off-by-one miscalculation in the session_groups Request Payload judging by the position of the "order:" key. |
|
I also observe a similar problem when the number of Param string parameters becomes greater than a certain number. When adding additional parameters, the sorting is shifted two columns to the right, when sorting the third column, the fifth is sorted. |
|
still has this problem... |
|
Hi @TangJiakai , TensorBoard 2.11.0 should contain #5971 which may address some of the issues here. Anybody who is still following along or comes here in the future, could you:
|
Environment information
Diagnostics
Diagnostics output
For browser-related issues, please additionally specify:
Issue description
In the Table View
sorting only works sporadically. Most of the time, it doesn't work at all, other times, sorting is applied to the previously selected column. This seems to be independent of whether you first select the sorting direction and then the sort by drop down, or otherwise round. This seems to be independent of the number of trials. Currently, I observe that for around ~50 runs.
On the server side that hosts the TensorBoard, I don't observe CPU core utilization or RAM saturation.
The text was updated successfully, but these errors were encountered: