v1.3.1

[rsepassi]

No description provided.

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for the commit author(s). If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

[martinpopel]

Is the deletion of t2t-bleu intentional?
(I can understand it, if yes.)

[rsepassi]

Thanks for noticing that. No, it was not intentional and likely due to a bad internal merge of the PR.

[lukaszkaiser]

Sorry guys, it was me and halfway intentional. There was a python 2 vs 3 problem and I'm not sure if we want to maintain another binary when there is SacreBLEU now. But we should certainly add the new BLEU as a metric to Tensorboard. In any case: can we get this in and redo the BLEU binary in another PR? Would that be ok Martin? Sorry for the issue!

[martinpopel]

Yes, we can redo t2t-bleu.py in another PR, I don't want to slow down your work.
If really needed, I can look at the Python2 compatibility (btw. SacréBLEU is Python3 only).
Note that t2t-bleu.py does not really overlap with the sacréBLEU functionality (actually, it could use SacréBLEU internally) - it focuses on evaluating all checkpoints and storing to Tensorboard event file (see #436 summary).

[lukaszkaiser]

Let's do that. The problem with python2 is that Google uses it internally, so if it's not compatible then all our internal tests and stuff start failing. So yes, let's get this in and then redo the BLEU. Could we have your BLEU score added to metrics, or maybe replace the current approx-BLEU entirely? It's much more useful as it's much closer to the real BLEU than the current approx. What do you think?

[martinpopel]

Could we have your BLEU score added to metrics, or maybe replace the current approx-BLEU entirely?

"my" BLEU actually uses "your" bleu_hook.py compute_bleu(), i.e. the same code as is used for computing approx_bleu (I just added smoothing, but this is already merged).
The important difference is that for approx_bleu a tensor with subword IDs is used as the input reference_corpus and translation_corpus, but when used from t2t-bleu a list of words is used instead (tokenized from plaintext using bleu_tokenize()). I am not sure whether it is wise to use this instead of approx_bleu - it may be too slow, and also I am not sure if the detokenization+tokenization can be converted to static graph ops. To get close to the real BLEU, we would also need eval_run_autoregressive and beam search. Currently, internal evaluation does not work for multi-gpu training, so we need an external script/job anyway. I like it this way because I can run it on CPU (or older GPU) and not slow down the training on GPUs.

[martinpopel]

What about the failing Travis for this PR?
I've noticed the tests sometime fail non-deterministically, e.g. here.

[rsepassi]

Yes, that Algorithmic Algebra test is flaky. You can see that master builds fine (no change from this PR). We'll likely end up removing those problems as nobody is using them.