ilivans / tf-rnn-attention Goto Github PK

Thanks for your code about training a attention model~
But I didn't find a visualizing code which can product the picture in your README.md

Is it possible to share the code by which we can visualize the attention weight?

I was mentioning this paper

https://github.com/pemywei/attention-nmt/blob/master/seq2seq.py#L121
The link project use prev_state as one of the input parameter.
@ilivans Thank you!!!

Hi :
Which version of tensorflow is used?

I can only see the word level encoder. Am I missing something? There should be two Bi-GRU layers right?

Thank you very much for sharing your code.

When calculating attention scores, you take softmax over the whole sequence. In case the sequence is of variable length, the softmax takes into account the scores (which are zeros) over the padded tokens also. Although eventually multiplying their attention scores with zeros from hidden states masks their contribution, the softmax implementation is still not correct as it can diminish the contribution (alphas) of non-padded time-steps.

Regarding this something like sparse softmax (??) could help.

Thank You,

I am trying out the code with imdb reuters example, which has 46 classes. I am getting highly negative loss on each iteration.

In attention part, the attention_size is a Hyperparameter, when we calculate the alpha, the shape of alpha is not about of attention_size. So, the attention_size is to do what?

Hi ilivans,

Great implementation! I was just wondering if the padded sequences don't have to be masked out somehow for the attention layer. I imagine that the logits in rnn_outputs have values up to max_len, so they include the padded positions (which the Bilstm has ignored) and then this is used for the calculations in attention. Do we need to set their attention scores to zero or something like that, or is the network able to learn that the padded positions should not be taken into account?

Thanks in advance.

Hi Ilivans,

In your implementation, you padded all training set to a fixed length: 250

• X_train = zero_pad(X_train, SEQUENCE_LENGTH)
• batch_ph = tf.placeholder(tf.int32, [None, SEQUENCE_LENGTH])
  But I want to calculate this factor for each mini-batch, that's why I have to declare batch_ph of shape [None, None]. So in the attention.py file: sequence_length = inputs_shape[1].value will be "None" then the error will be appear in the command: exps = tf.reshape(tf.exp(vu), [-1, sequence_length]):
  "Failed to convert object of type <class 'list'> to Tensor. Contents: [-1, None]. Consider casting elements to a supported type."
  Do you have any idea for this problem?
  Thank you in advance!

I found the codes of in toy_example.py likes these:
accuracy = 1. - tf.reduce_mean(tf.cast(tf.equal(tf.round(y_hat), target_ph), tf.float32))
I am confused. Why don't you use the codes "accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.round(y_hat), target_ph), tf.float32))" ? Is it a bug ?

Why is there a +1 at line 114: seq_len = np.array([list(x).index(0) + 1 for x in x_batch]) # actual lengths of sequences and line 129 in train.py? Won't list(x).index(0) without the +1 be enough?

Love this repo, helps me a lot! is there any paper of yours I can cite?