t2t-translate-all error

[rolloff]

I'm running tensor2tensor from source, with the following version of tensorflow

pip list | grep tensor
tensorflow (1.4.0rc0)
tensorflow-tensorboard (0.4.0rc1)

I'm trying to translate 20 checkpoints in my checkpoint directory using t2t-translate all, with the end goal of computing the final BLEU score of the models learned in order to compare to the paper. The command I am running is

SOURCE_DIR=/tmp/t2t_datagen/dev/newstest2014-deen-src.en.sgm
TRANSLATIONS_DIR=/tmp/t2t_datagen/dev/translations

mkdir -p $TRANSLATIONS_DIR

t2t-translate-all --model_dir $MODEL_DIR \
                  --source $SOURCE_DIR \
                   --data_dir=~/t2t_data \
                   --problems=translate_ende_wmt32k \
                   --model=transformer \
                   --hparams_set=transformer_base_single_gpu \
                   --output_dir=~/t2t_train/base \
                   --t2t_usr_dir=~/t2t_usr/

The error I get is

Traceback (most recent call last):
  File "/home/ubuntu/tensor2tensor/tensor2tensor/bin/t2t-translate-all", line 91, in <module>
    tf.app.run()
  File "/home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "/home/ubuntu/tensor2tensor/tensor2tensor/bin/t2t-translate-all", line 73, in main
    for model in bleu_hook.stepfiles_iterator(model_dir, FLAGS.wait_minutes, FLAGS.min_steps):
AttributeError: module 'tensor2tensor.utils.bleu_hook' has no attribute 'stepfiles_iterator'

What's wrong?
@martinpopel

[martinpopel]

My guess: you are using t2t-translate-all from the github master (it is in your $PATH), but tensor2tensor.utils.bleu_hook from some older T2T version, which is missing the stepfiles_iterator

tensor2tensor/tensor2tensor/utils/bleu_hook.py

Line 226 in f55462a

def stepfiles_iterator(path_prefix, wait_minutes=0, min_steps=0,

You should install the github master version with pip install -e . or make sure you have setup $PYTHONPATH so it includes the bleu_hook.py from the github master.

[rolloff]

Installed tensor2tensor from the github master:

ubuntu@ip-172-31-35-220:~/tensor2tensor$ python3.6 -m pip install -e .
Obtaining file:///home/ubuntu/tensor2tensor
Requirement already satisfied: bz2file in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: future in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: gym in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: numpy in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: requests in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: scipy in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: sympy in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: six in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from tensor2tensor==1.4.1)
Requirement already satisfied: pyglet>=1.2.0 in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from gym->tensor2tensor==1.4.1)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from requests->tensor2tensor==1.4.1)
Requirement already satisfied: idna<2.7,>=2.5 in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from requests->tensor2tensor==1.4.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from requests->tensor2tensor==1.4.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from requests->tensor2tensor==1.4.1)
Requirement already satisfied: mpmath>=0.19 in /home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages (from sympy->tensor2tensor==1.4.1)
Installing collected packages: tensor2tensor
  Found existing installation: tensor2tensor 1.4.1
    Can't uninstall 'tensor2tensor'. No files were found to uninstall.
  Running setup.py develop for tensor2tensor
Successfully installed tensor2tensor-1.4.1

Appended to path

echo $PYTHONPATH
/home/ubuntu/tensor2tensor/tensor2tensor/utils/bleu_hook.py:/home/ubuntu/tensor2tensor/tensor2tensor/utils:/home/ubuntu/src/caffe2/build:/home/gauta/src/caffe2/build:/home/ubuntu/src/cntk/bindings/python

Same error when running the above command:

Traceback (most recent call last):
  File "/home/ubuntu/tensor2tensor/tensor2tensor/bin/t2t-translate-all", line 91, in <module>
    tf.app.run()
  File "/home/ubuntu/anaconda/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "/home/ubuntu/tensor2tensor/tensor2tensor/bin/t2t-translate-all", line 73, in main
    for model in bleu_hook.stepfiles_iterator(model_dir, FLAGS.wait_minutes, FLAGS.min_steps):
AttributeError: module 'tensor2tensor.utils.bleu_hook' has no attribute 'stepfiles_iterator'

[martinpopel]

Sorry, I was inaccurate: PYTHONPATH cannot include individual *.py files, in your case it should include /home/ubuntu/tensor2tensor/ (but if the pip install was successful, there is no need for adding anything to PYTHONPATH, these are alternative solutions).
Please try this:

import tensor2tensor.utils.bleu_hook
print(tensor2tensor.utils.bleu_hook.__file__)

and make sure the given file is the new one which includes the stepfiles_iterator.
Once T2T 1.4.2 is released (should be soon), it should work out of the box.

Of course, there may be bugs in t2t-translate-all and t2t-bleu, thanks for reporting them.

Now, I have noticed you install with python3.6, but it seems this is not your default python. Note that t2t-translate-all starts with line #!/usr/bin/env python so it will be executed with your default python. One solution is to set python3.6 as your default (e.g. with venv), another solution is to execute the script with python3.6 /home/ubuntu/tensor2tensor/tensor2tensor/bin/t2t-translate-all.

[rolloff]

With release of 1.4.2, my issue is fixed. Thanks for the quick answers.

[martinpopel]

I am happy it works for you. Unfortunately, I found few more bugs in t2t-translate-all and t2t-bleu. Some are my fault, some were introduced during the Google release refactoring (e.g. t2t-bleu was about 1000 times slower than it should be), which is always obscured in the Github history, so I didn't notice sooner. All (I am aware of) should be fixed in #524.

[zx1301]

I'm just trying to figure out how to use t2t-translate-all. Is $MODEL_DIR supposed to be the directory with all the checkpoints?

[martinpopel]

Yes, --model_dir should be the directory with all the checkpoints. This parameter is passed to t2t-decoder (or your wrapper of it if specified with decoder_command) as --output_dir, but I felt that "model_dir" is a better name in this case.