scripts for proper BLEU evaluation, batch translation and averaging

[martinpopel]

t2t-bleu computes the "real" BLEU
(giving the same result as sacréBLEU with --tokenization intl
and as mteval-v14.pl with --international-tokenization).
It can be used in two ways:

• To evaluate one translated file: t2t-bleu --translation=my-wmt13.de --reference=wmt13_deen.de
• To evaluate all translations in a given directory (created e.g. by t2t-translate-all).

t2t-translate-all translates all checkpoints in a given directory (created by t2t-trainer or t2t-avg-all).
A custom command (e.g. SGE cluster wrapper) can be used instead of t2t-decoder for the translation.

t2t-avg-all for each checkpoint in a given directory
it averages it with the N preceding ones.

All three scripts wait a given number of minutes for new checkpoints
(produced by t2t-decoder, which can be run concurrently with these scripts).

This PR is an improved version of #436 (which was not fully merged).

[rsepassi]

Thank you!

[terrychenism]

it seems has some small problems.

Traceback (most recent call last):
File "/usr/local/bin/t2t-bleu", line 6, in
exec(compile(open(file).read(), file, 'exec'))
File "/workspace/tf_project/tensor2tensor/tensor2tensor/bin/t2t-bleu", line 137, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/workspace/tf_project/tensor2tensor/tensor2tensor/bin/t2t-bleu", line 80, in main
bleu = 100 * bleu_hook.bleu_wrapper(FLAGS.reference, FLAGS.translation, case_sensitive=False)
File "/workspace/tf_project/tensor2tensor/tensor2tensor/utils/bleu_hook.py", line 202, in bleu_wrapper
ref_tokens = [bleu_tokenize(x) for x in ref_lines]
File "/workspace/tf_project/tensor2tensor/tensor2tensor/utils/bleu_hook.py", line 187, in bleu_tokenize
string = UnicodeRegex.nondigit_punct_re.sub(r'\1 \2 ', string)
AttributeError: type object 'UnicodeRegex' has no attribute 'nondigit_punct_re'

[martinpopel]

I see, I am sorry for the problems.
In my original code (#436), there was

class UnicodeRegex:
  """Ad-hoc hack to recognize all punctuation and symbols.

  without dependening on https://pypi.python.org/pypi/regex/."""
  def _property_chars(prefix):
    return ''.join(six.unichr(x) for x in range(sys.maxunicode)
                   if unicodedata.category(six.unichr(x)).startswith(prefix))
  punctuation = _property_chars('P')
  nondigit_punct_re = re.compile(r'([^\d])([' + punctuation + r'])')
  punct_nondigit_re = re.compile(r'([' + punctuation + r'])([^\d])')
  symbol_re = re.compile('([' + _property_chars('S') + '])')

which someone changed into

class UnicodeRegex(object):
  """Ad-hoc hack to recognize all punctuation and symbols."""

  def __init__(self):
    def _property_chars(prefix):
      return ''.join(six.unichr(x) for x in range(sys.maxunicode)
                     if unicodedata.category(six.unichr(x)).startswith(prefix))
    punctuation = self._property_chars('P')
    self.nondigit_punct_re = re.compile(r'([^\d])([' + punctuation + r'])')
    self.punct_nondigit_re = re.compile(r'([' + punctuation + r'])([^\d])')
    self.symbol_re = re.compile('([' + _property_chars('S') + '])')

but did not adapt accordingly the code few lines below.
In the end it is my fault, I haven't included any tests and rebased my new scripts on top of the master without checking the changes done in the master.
I will send a fix PR tomorrow.

[terrychenism]

Great! Thanks

[martinpopel]

see #514