A tf.keras implementation of Facebook AI's MadGrad optimization algorithm
License: MIT License
🔭 I’m currently working on explainability in dialogue systems and causal language models.
🌱 I'm interested in argumentation, interpretability, and commonsense reasoning in large language models
🔎 I actively do research in the NLP subspace. Find my research on Google Scholar
📝 I write reports and articles on Weights & Biases | Medium
📫 Reach me at: [email protected]
⚡ Fun fact: I am a multicopter enthusiast
I am testing this optimizer but it seems kind of broken
15 epochs and nothing is changing
Curious since it is supposed to be one of the top
234/233 [==============================] - ETA: 0s - loss: 1.0557 - accuracy: 0.4815 - recall: 0.1011 - precision: 0.4922
Epoch 1: val_accuracy improved from -inf to 0.46253, saving model to efficient.model.hdf5
233/233 [==============================] - 220s 706ms/step - loss: 1.0557 - accuracy: 0.4815 - recall: 0.1011 - precision: 0.4922 - val_loss: 1.0635 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0100
Epoch 2/128
234/233 [==============================] - ETA: 0s - loss: 1.0537 - accuracy: 0.4831 - recall: 0.0837 - precision: 0.4549
Epoch 2: val_accuracy did not improve from 0.46253
233/233 [==============================] - 159s 679ms/step - loss: 1.0537 - accuracy: 0.4831 - recall: 0.0837 - precision: 0.4549 - val_loss: 1.0639 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0100
Epoch 3/128
234/233 [==============================] - ETA: 0s - loss: 1.0533 - accuracy: 0.4831 - recall: 0.0669 - precision: 0.4596
Epoch 3: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 676ms/step - loss: 1.0533 - accuracy: 0.4831 - recall: 0.0669 - precision: 0.4596 - val_loss: 1.0634 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0100
Epoch 4/128
234/233 [==============================] - ETA: 0s - loss: 1.0526 - accuracy: 0.4831 - recall: 0.1049 - precision: 0.4840
Epoch 4: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 678ms/step - loss: 1.0526 - accuracy: 0.4831 - recall: 0.1049 - precision: 0.4840 - val_loss: 1.0729 - val_accuracy: 0.4625 - val_recall: 0.4625 - val_precision: 0.4625 - lr: 0.0100
Epoch 5/128
234/233 [==============================] - ETA: 0s - loss: 1.0530 - accuracy: 0.4831 - recall: 0.1825 - precision: 0.4899
Epoch 5: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 677ms/step - loss: 1.0530 - accuracy: 0.4831 - recall: 0.1825 - precision: 0.4899 - val_loss: 1.0647 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0100
Epoch 6/128
234/233 [==============================] - ETA: 0s - loss: 1.0538 - accuracy: 0.4831 - recall: 0.0776 - precision: 0.4693
Epoch 6: val_accuracy did not improve from 0.46253
Epoch 6: ReduceLROnPlateau reducing learning rate to 0.0029999999329447745.
233/233 [==============================] - 158s 678ms/step - loss: 1.0538 - accuracy: 0.4831 - recall: 0.0776 - precision: 0.4693 - val_loss: 1.0641 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0100
Epoch 7/128
234/233 [==============================] - ETA: 0s - loss: 1.0518 - accuracy: 0.4831 - recall: 0.0000e+00 - precision: 0.0000e+00
Epoch 7: val_accuracy did not improve from 0.46253
233/233 [==============================] - 159s 680ms/step - loss: 1.0518 - accuracy: 0.4831 - recall: 0.0000e+00 - precision: 0.0000e+00 - val_loss: 1.0638 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0030
Epoch 8/128
234/233 [==============================] - ETA: 0s - loss: 1.0518 - accuracy: 0.4831 - recall: 0.0479 - precision: 0.4864
Epoch 8: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 676ms/step - loss: 1.0518 - accuracy: 0.4831 - recall: 0.0479 - precision: 0.4864 - val_loss: 1.0629 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0030
Epoch 9/128
234/233 [==============================] - ETA: 0s - loss: 1.0515 - accuracy: 0.4831 - recall: 0.0433 - precision: 0.4821
Epoch 9: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 675ms/step - loss: 1.0515 - accuracy: 0.4831 - recall: 0.0433 - precision: 0.4821 - val_loss: 1.0654 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0030
Epoch 10/128
234/233 [==============================] - ETA: 0s - loss: 1.0511 - accuracy: 0.4831 - recall: 0.0391 - precision: 0.5069
Epoch 10: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 676ms/step - loss: 1.0511 - accuracy: 0.4831 - recall: 0.0391 - precision: 0.5069 - val_loss: 1.0690 - val_accuracy: 0.4625 - val_recall: 0.4625 - val_precision: 0.4625 - lr: 0.0030
Epoch 11/128
234/233 [==============================] - ETA: 0s - loss: 1.0523 - accuracy: 0.4831 - recall: 0.0736 - precision: 0.4774
Epoch 11: val_accuracy did not improve from 0.46253
Epoch 11: ReduceLROnPlateau reducing learning rate to 0.0009000000078231095.
233/233 [==============================] - 158s 678ms/step - loss: 1.0523 - accuracy: 0.4831 - recall: 0.0736 - precision: 0.4774 - val_loss: 1.0638 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 0.0030
Epoch 12/128
234/233 [==============================] - ETA: 0s - loss: 1.0512 - accuracy: 0.4831 - recall: 0.0000e+00 - precision: 0.0000e+00
Epoch 12: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 678ms/step - loss: 1.0512 - accuracy: 0.4831 - recall: 0.0000e+00 - precision: 0.0000e+00 - val_loss: 1.0633 - val_accuracy: 0.4625 - val_recall: 0.0000e+00 - val_precision: 0.0000e+00 - lr: 9.0000e-04
Epoch 13/128
234/233 [==============================] - ETA: 0s - loss: 1.0512 - accuracy: 0.4831 - recall: 0.0000e+00 - precision: 0.0000e+00
Epoch 13: val_accuracy did not improve from 0.46253
233/233 [==============================] - 158s 677ms/step - loss: 1.0512 - accuracy: 0.4831 - recall: 0.0000e+00 - precision: 0.0000e+00 - val_loss: 1.0634
lr argument is deprecated, use learning_rate instead.lr argument is deprecated, use learning_rate instead.")ValueError Traceback (most recent call last)
in ()
2 epochs = 15
3
----> 4 model.compile(loss="categorical_crossentropy", optimizer=MadGrad(lr=1e-3), metrics=["accuracy"])
5 model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)
5 frames
/usr/local/lib/python3.7/dist-packages/keras/optimizers.py in get(identifier)
131 else:
132 raise ValueError(
--> 133 'Could not interpret optimizer identifier: {}'.format(identifier))
ValueError: Could not interpret optimizer identifier: <madgrad.madgrad.MadGrad object at 0x7f8690765110>
Hello.
Thank you for sharing great work.
When running the example, the following error is displayed.
AttributeError: module 'tensorflow.python.framework.ops' has no attribute 'convert_to_tensor v2 with dispatch'my TensorFlow version is 2.3.0.
I downgrade to 2.2.0, but the result is the same.
Is there any way to fix something?
Thank you in advance.
Best Regards,
Seungwoo
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.