ricsinaruto / seq2seqchatbots Goto Github PK

Running the generate command fails with a missing file.

$ python t2t_csaky/main.py --mode=generate_data
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
Program is running in data generation mode.
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from spec or package, falling back on name and path
return f(*args, **kwds)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
INFO:tensorflow:Importing user module t2t_csaky from path /home/hannes/Seq2seqChatbots
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/utils/registry.py:180: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
args, varargs, keywords, _ = inspect.getargspec(rhp_fn)
INFO:tensorflow:Generating problems:
daily:
* daily_dialog_chatbot
INFO:tensorflow:Generating data for daily_dialog_chatbot.
:::MLPv0.5.0 transformer 1542920333.001461029 (/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/data_generators/text_problems.py:306) preproc_tokenize_training
Traceback (most recent call last):
File "/home/hannes/anaconda3/envs/tf_gpu/bin/t2t-datagen", line 28, in
tf.app.run()
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/hannes/anaconda3/envs/tf_gpu/bin/t2t-datagen", line 23, in main
t2t_datagen.main(argv)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/bin/t2t_datagen.py", line 198, in main
generate_data_for_registered_problem(problem)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/bin/t2t_datagen.py", line 260, in generate_data_for_registered_problem
problem.generate_data(data_dir, tmp_dir, task_id)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/data_generators/text_problems.py", line 306, in generate_data
self.generate_encoded_samples(data_dir, tmp_dir, split)), paths)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/data_generators/text_problems.py", line 263, in generate_encoded_samples
encoder = self.get_or_create_vocab(data_dir, tmp_dir)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/data_generators/text_problems.py", line 239, in get_or_create_vocab
replace_oov=self.oov_token)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/data_generators/text_encoder.py", line 311, in init
self._init_vocab_from_file(vocab_filename)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensor2tensor/data_generators/text_encoder.py", line 347, in _init_vocab_from_file
tokens = [token.strip() for token in f.readlines()]
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 188, in readlines
self._preread_check()
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 85, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "/home/hannes/anaconda3/envs/tf_gpu/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: data_dir/DailyDialog/no_stop_words/vocab.chatbot.16384; No such file or directory

The config file should be a class which works the same way as in: https://github.com/ricsinaruto/dialog-eval

Each parameter should be settable through the command line as well.

flake8 testing of https://github.com/ricsinaruto/Seq2seqChatbots on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./t2t_csaky/data_filtering/sentence_embedding.py:193:11: E999 IndentationError: expected an indented block
  processes=[]
          ^
./t2t_csaky/models/roulette_transformer.py:46:26: F821 undefined name 'eu'
      data_parallelism = eu.Parallelism([""])
                         ^
./t2t_csaky/models/roulette_transformer.py:193:24: F821 undefined name 'features_to_nonpadding'
            nonpadding=features_to_nonpadding(features, "targets"))
                       ^
./t2t_csaky/models/roulette_transformer.py:310:8: F821 undefined name 'self'
    if self._hparams.roulette=="Normal" or self._hparams.roulette=="Inverse":
       ^
./t2t_csaky/models/roulette_transformer.py:310:44: F821 undefined name 'self'
    if self._hparams.roulette=="Normal" or self._hparams.roulette=="Inverse":
                                           ^
./t2t_csaky/models/roulette_transformer.py:311:10: F821 undefined name 'self'
      if self._hparams.roulette=="Normal":
         ^
./t2t_csaky/models/roulette_transformer.py:317:25: F821 undefined name 'self'
        start=beam_size-self._hparams.roulette_beam_size
                        ^
./t2t_csaky/models/roulette_transformer.py:327:29: F821 undefined name 'self'
      for i in range(start ,self._hparams.roulette_beam_size):
                            ^
./t2t_csaky/scripts/filtering_visualization.py:116:23: F821 undefined name 'dataset'
    "../../data_dir/"+dataset+"/base_with_numbers/filtered_data/hash_jaccard/"
                      ^
./t2t_csaky/scripts/filtering_visualization.py:117:10: F821 undefined name 'source_clusters'
    +str(source_clusters)+"_clusters/fullSource_cluster_elements.txt")
         ^
1     E999 IndentationError: expected an indented block
9     F821 undefined name 'eu'
10

Hi,
Can anyone tell me where can I find the pre-trained models or processed and checkpoints folder for this chatbot or help me by sending it?

The code for this model needs to be refactored and updated to work with newest t2t version.

The gradient checkpointed optimizer might not be needed for the seq2seq. Instead the default tensor2tensor optimizer class should be used.

Currently the character-level chatbot is not working. There should be a parameter with which we can control for each dataset whether word-level or character-level data generation should be used (maybe also subwords?).

Bigram matrix creation is not related to filtering. A separate class should be created, and maybe within that an instance of the filter_problem class.

Currently to decode a specific checkpoint of a trained model you have to modify the checkpoint file, to point to the checkpoint. There should be a parameter in the config file, where we can set this instead.

Hello, thanks for sharing.

I download the data and find the problem when I try to run:
python t2t_csaky/main.py --mode=generate_data

How can I resolve this issue?

Thanks & Regards!

Code currently is not compatible with tensor2tensor versions higher than 1.5.1. However, this is not causing any problems, and it is a very low prio issue.

Did someone succeed in setting up the environment? I get various compatibility issues when I tried to set up. Does any one know the version of tensor2tensor and Tensorflow to be used to set it up?

File "/home/Network/Seq2seqChatbots/t2t_csaky/problems/word_chatbot.py", line 137, in hparams "inputs": (registry.Modalities.SYMBOL, source_vocab_size)} AttributeError: module 'tensor2tensor.utils.registry' has no attribute 'Modalities'
The class 'Modalities' has been removed from 'tensor2tensor.utils.registry' since t2t v1.11.0.

I have pre-trained models and I want to use other problems like word_chatbots, persona_chat_chatbots etc?