strubell / lisa Goto Github PK

Hello Ms.Strubell :-) I am trying to train and evaluate your LISA model on CoNLL05 dataset. I followed the recipe in this post https://github.com/strubell/preprocess-conll05 for preprocesing ConLL2005 dataset and I have adapted the data path in configuration file correspondingly. When I run the training, the initialization steps of tensorflow model seem to work normally. But after "filling up the shuffle buffer" , I got following error information immediately.. Do you have any ideas about the reason of this error? And could you have any pretrained models on CoNLL05 dataset ?

2018-10-18 23:39:20.446629: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:135] Shuffle buffer filled.
Traceback (most recent call last):
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[862] = 5199 is not in [0, 1968)
[[Node: LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@LISA/Nadam/update_LISA/word_type_embeddings/embeddings/ScatterAdd"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](LISA/Nadam/update_LISA/word_type_embeddings/embeddings/add_1, LISA/Nadam/update_LISA/word_type_embeddings/embeddings/Unique, LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3/axis)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "src/train.py", line 143, in
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
return executor.run()
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 590, in run
return self.run_local()
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 691, in run_local
saving_listeners=saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 376, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1145, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1173, in _train_model_default
saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1451, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 583, in run
run_metadata=run_metadata)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1059, in run
run_metadata=run_metadata)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1150, in run
raise six.reraise(*original_exc_info)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1135, in run
return self._sess.run(*args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1207, in run
run_metadata=run_metadata)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 987, in run
return self._sess.run(*args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[862] = 5199 is not in [0, 1968)
[[Node: LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@LISA/Nadam/update_LISA/word_type_embeddings/embeddings/ScatterAdd"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](LISA/Nadam/update_LISA/word_type_embeddings/embeddings/add_1, LISA/Nadam/update_LISA/word_type_embeddings/embeddings/Unique, LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3/axis)]]

Caused by op 'LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3', defined at:
File "src/train.py", line 143, in
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
return executor.run()
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 590, in run
return self.run_local()
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 691, in run_local
saving_listeners=saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 376, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1145, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1170, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1133, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/Users/xiaotang/Documents/SRL/LISA/src/model.py", line 294, in model_fn
train_op = optimizer.apply_gradients(zip(gradients, variables), global_step=tf.train.get_global_step())
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/contrib/optimizer_v2/optimizer_v2.py", line 866, in apply_gradients
self._distributed_apply, filtered, global_step=global_step, name=name)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/distribute.py", line 1053, in merge_call
return self._merge_call(merge_fn, *args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/distribute.py", line 1060, in _merge_call
return merge_fn(self._distribution_strategy, *args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/contrib/optimizer_v2/optimizer_v2.py", line 964, in _distributed_apply
var, update, grad)))
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/distribute.py", line 868, in update
return self._update(var, fn, *args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/training/distribute.py", line 1144, in _update
return fn(var, *args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/contrib/optimizer_v2/optimizer_v2.py", line 958, in update
return processor.update_op(self, g, state)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/contrib/optimizer_v2/optimizer_v2.py", line 81, in update_op
return optimizer._apply_sparse_duplicate_indices(g, self._v, *args)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/contrib/optimizer_v2/optimizer_v2.py", line 1204, in _apply_sparse_duplicate_indices
return self._apply_sparse(gradient_no_duplicate_indices, var, state)
File "/Users/xiaotang/Documents/SRL/LISA/src/lazy_adam_v2.py", line 228, in _apply_sparse
state)
File "/Users/xiaotang/Documents/SRL/LISA/src/lazy_adam_v2.py", line 212, in _apply_sparse_shared
m_bar_slice = array_ops.gather(m_bar, indices)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 2659, in gather
return gen_array_ops.gather_v2(params, indices, axis, name=name)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3142, in gather_v2
"GatherV2", params=params, indices=indices, axis=axis, name=name)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/Users/xiaotang/Documents/soft/miniconda3/envs/deep_nlp/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): indices[862] = 5199 is not in [0, 1968)
[[Node: LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3 = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@LISA/Nadam/update_LISA/word_type_embeddings/embeddings/ScatterAdd"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](LISA/Nadam/update_LISA/word_type_embeddings/embeddings/add_1, LISA/Nadam/update_LISA/word_type_embeddings/embeddings/Unique, LISA/Nadam/update_LISA/word_type_embeddings/embeddings/GatherV2_3/axis)]]

Hi, I have a question about the evaluation in the end-to-end setting.
When I study the code, I found your evaluation of CoNLL-2005 is different from He et al. (2018) [https://github.com/luheng/lsgn/blob/master/srl_eval_utils.py] in line 190-207.
Then, I conduct some evaluation experiments with the following two files (gold file and system file)

gold file:

  -                (AM-LOC*               *
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *)              *
  -                       *               *
  -                    (A0*)              *
  call                  (V*)              *
  -                    (A1*               *
  -                       *            (A1*
  -                       *)              *)
  -                    (A2*               *
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *               *
  discipline              *             (V*)
  -                       *               *
  -                       *        (AM-MNR*
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *)              *)
  -                       *               *

system file:

  -                (AM-LOC*               *
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *)              *
  -                       *               *
  -                    (A0*)              *
  call                  (V*)              *
  -                    (A1*               *
  -                       *            (A1*
  -                       *)              *)
  -                    (A2*               *
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *              *
  ,                       *               (V*)
  -                       *        (AM-MNR*
  -                       *               *
  -                       *               *
  -                       *               *
  -                       *)              *)
  -                       *               *

After running the evaluation script, we get the results as follows:

WARNING : sentence 0 : verb discipline at position 16 : missing predicted prop! Counting all arguments as missed!
Number of Sentences    :           1
Number of Propositions :           2
Percentage of perfect props :  50.00

              corr.  excess  missed    prec.    rec.      F1
------------------------------------------------------------
   Overall        4       0       2   100.00   66.67   80.00
----------
        A0        1       0       0   100.00  100.00  100.00
        A1        1       0       1   100.00   50.00   66.67
        A2        1       0       0   100.00  100.00  100.00
    AM-LOC        1       0       0   100.00  100.00  100.00
    AM-MNR        0       0       1     0.00    0.00    0.00
------------------------------------------------------------
         V        1       0       1   100.00   50.00   66.67
------------------------------------------------------------

The evaluation script only gives the correctly predicted number of 4, excess number of 0, and missed number of 2.
And the script warns that the second predicate ``discipline'' is missed and counting all arguments as missed.
In my mind, I think the precision should be 4 / (4 + 2), where 2 is the number of wrongly predicted predicates' arguments.
But in your code, you missed the number of wrongly predicted predicates' arguments, which would result in higher precision.

Did I miss something or understand something wrongly?

Thank you very much for your reply!

Hi Ms strubell, thanks for your great work ! I would like to try LISA with other state-of-the-art word embedding approach (e.g. elmo) to improve the accuracy on a private dataset :-) Does current LISA framework support other word embedding ? How should I modify the framework to use other word embedding ?

I am wondering if there is a place to config the eval with the golden parsing?

Hi!
Really nice and interesting work!
I was having a look at the code to better understand the bilinear classification step for srl (srl_bilinear method of output_fns (line 169)).
Why do you use a single MLP layer to project both the predicate vectors and the word (role) vectors (line 195), of which you take two slices afterward (line 196), to do all the computation, instead of using two separate MLPs for predicates and roles? Is it because in such a way the projection of roles affects also the predicates one, and the other way around (or at least, this is what should happen in my mind - it should be a fully connected layer).

many thanks!

Hello,
I'm here to ask a question.
It's not about code but about your paper.

To my understanding, the last term of the training objective should be
logP(yt_dep | Vg, X)
instead of logP(yt_dep | Pg, X).
Or Pg should be removed.

Because predicting predicates with POS tagging precedes parsing dependency.
And Pg is the answer for yt_dep.

I think it doesn't make sense to use Gold Parse(Pg) to predict yt_dep.

I'll look forward to your gold explanation. :)
TY

Hello,
I am trying to train and evaluate your LISA model on CoNLL dataset.
While trying to train the model on a GPU, I use the cmd as CUDA_VISIBLE_DEVICES=0 bin/evaluate.sh config/conll05-lisa.conf --save_dir model. However, it seems that nothing works on the GPU. The nvidia-smi shows volatile GPU-util is zero.
How to make best use of GPU for TensorFlow Estimators?
Do you have any ideas about the reason of this problem?