Comments (13)
Yeah, but I would like to use non-english Embeddings from ELMoForManyLangs and need to be able to retrain them with custom training data. tfhub is no option, I think.
from elmoformanylangs.
Thanks for answering! I think we can use the Lambda layer or create a custom one. I don' know how to handle the Tensors that keras gives and return a data structure that keras accepts.
e = Embedder('...')
sess = tf.Session()
def ElmoEmbedding(x):
with sess.as_default():
return tf.convert_to_tensor(e.sents2elmo(x.eval())[0]) # this does not work
...
embedding = Lambda(ElmoEmbedding, output_shape=(None, max_len, 1024))(input_text)
from elmoformanylangs.
So, I have a workaround but it is somewhat impractical. Obtain the vocabulary of your dataset, then create an embedding file similar to a word2vec or glove file (ie. word 1024-dim-weights per line). And then implement an custom weight embedding layer, it worked ok for Spanish.
from elmoformanylangs.
You can create a Keras Sequence, in which you can apply the embedder on the input sequences, and use model.fit_on_generator/predict_on_generator
from elmoformanylangs.
how about hub.Module("https://tfhub.dev/google/elmo/2",
trainable=True)`
from elmoformanylangs.
Hey juckeltour did you solve the problem?
I found a tutorial that comes close to a task that i try to solve. let me know if you found a solution
maybe this will help you out :
http://hunterheidenreich.com/blog/elmo-word-vectors-in-keras/
from elmoformanylangs.
Hi @juckeltour, have you found the way to use ELMoForManyLangs embeddings with Keras
from elmoformanylangs.
can anyone find any solution?
from elmoformanylangs.
can anyone find any solution?
Hi guys, anyone found a solution yet?
from elmoformanylangs.
No, I didn't solve this problem.
We switched to BERT (and pytorch)...
from elmoformanylangs.
@bazzmx do you have a code snippet?
from elmoformanylangs.
This is based on this blog post
First generate a list of unique words in your vocabulary and obtain their corresponding elmo embbedings using elmoformanylangs using sent2elmo and save to a file that contains one word and its 1024_dim_weights per line.
In this example this file is emb_table.txt and the words that compose my vocabulary are the set of lemmas and words that will be used during training and testing
vocab = list(set(list(words_train)+list(lemmas_train))) # unique words in your vocabulary
enumerated_vocab = enumerate(sorted(vocab), 1) # indexed words
index_labels = {} # index_no:word/label dict
for i in enumerated_vocab:
index_labels[i[0]]=i[1] # appends words and indexes to dict
labels_index = {v:k for k,v in index_labels.items()} # reversed dict = word/labels:index
With these dictionaries then you create the index to embedding matrix. Now that you are working with indexes you have to remember to convert all your words (string) to these indexes (int32).
embeddings_index = {}
f = open("./emb_table.txt", encoding="utf8") # opens generated elmo embeddings file
for line in f:
values = line.split() # splits each line
word = values[0] # first value is the word entry
coefs = np.asarray(values[1:], dtype='float32') # the rest is converted to an embedding array
embeddings_index[word] = coefs # appends data to dict
f.close()
Now we generate an embedding matrix putting all the pieces together:
EMBEDDING_DIM = 1024 # elmo's default size
embedding_matrix = np.zeros((len(vocab) + 1, EMBEDDING_DIM)) # creates a nx1024 matrix
for word, i in labels_index.items():
embedding_vector = embeddings_index.get(word)
if embedding_vector is not None:
embedding_matrix[i] = embedding_vector
With all this elements then you add an input layer and an embedding layer to your model, the input will be int32 because you have to convert each word to its index, then the embedding layer will assign the corresponding weights to each index:
input_words = Input(shape=(max_len,), dtype="int32", name="input_words")
EMB = Embedding(len(vocab) + 1,
EMBEDDING_DIM,
weights=[embedding_matrix],
input_length=(max_len,),
trainable=False, name="embedding")
emb_text = EMB(input_words)
Your input words should be an array of indexes that you can convert again to words using the dictionaries that were created.
This is an impractical workaround but it does the job for now, the key limitations are that you don't have acces to the embeddings in the same way that you would using tfhub.
I tried using the lambda layer approach but I ended up getting errors related to tensors and map_fn, etc,
from elmoformanylangs.
Anyone was able to make text classification in Keras successfully with sentence vectors?
from elmoformanylangs.
Related Issues (20)
- Error when importing elmofromanylans in my code HOT 2
- HELP! An error occur run: 'from elmoformanylangs import Embedder' HOT 2
- Convert csv to conllu format HOT 1
- anyone ever had this problem?
- Fine-tuning existing models
- token的时候create_one_batch方法中获取bow和eow的id的时候为什么互换了两个id的位置
- A suggestion about simplified-Chinese ELMo
- Programmatic way to get dimension of embedding
- Logging is very noisy HOT 1
- Default to sample...json as needed HOT 1
- 请问简体中文的训练语料可以从哪里获得呢?
- Document Embedding
- 请问如何进行elmo2sents
- --train_path data/en.raw \ 训练数据的格式是什么样的
- Dependency Parsing usage
- Running the model on GPU HOT 1
- from elmoformanylangs import Embedder HOT 9
- Pre-trained model download link invalid HOT 2
- The size of output model is very large when exporting to ONNX format.
- directory error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elmoformanylangs.