about embedding matrix structure about qanet HOT 3 CLOSED

Hi @tongjinle123 , you're right about fine tuning the unknown_word vectors. The paper suggests learning the unknown word vectors and we should do it.
I am just not too sure what the most elegant way to do this is in tensorflow. I am open to suggestions so if anyone has a brilliant idea to do this please let me know. Or even better, pull request!

Problem:
We want to fix word embeddings but fine-tune 1 word vector for token "Unk" (token for all unknown words). We only look up with 1 word embeddings vector which makes partial training difficult.

from qanet.

To be frankly, i am wondering if keeping the pre-trained matrix freezed is reasonable.I think that no one could ensure the matrix is well trained for our task. Why not taking a second train?

Could you please share a paper,artical or experiment which is talking about this problem with me?
When it comes to the way to the implemention, i had a mind-storm-thought that define pre-trained matrix and unk matrix separately using tf.get_variable, run a concat op before trainop and run a assign op after trainop assigning the unk value to the unk matrix. I think that will not consume much time.

sorry for the bad comment format.... QAQ

from qanet.

Refer to #13 for this issue.

from qanet.