Comments (8)
Hi @kayuksel I made a pull request (#22) that I think solves this problem - it should work with alpha < 1.0, and it's passing the tests. It would be great if you could try it and let us know if it worked.
from entmax.
It might be possible mathematically, but as far as I know our bisection algorithm only works for alpha>1.
from entmax.
Could you explain what you mean about sum-normalized sigmoids? I'm not sure what the connection is to entmax.
from entmax.
By sum-normalized sigmoids, I mean just taking the sigmoids of logits and then dividing them to their sum. Here (On Controllable Sparse Alternatives to Softmax) I believe that it is referred as sum normalization.
https://papers.nips.cc/paper/2018/file/6a4d5952d4c018a1c1af9fa590a10dda-Paper.pdf
I basically want to learn the sparsity and temperature parameter of the sparsemax at the same time. I thought entmax if it was possible to use alpha < 1.0 would be equivalent to that. I have tried but got NaNs
from entmax.
Hi @kayuksel , it is possible to use alpha instead of a temperature parameter to control the propensity for sparsity of entmax, and gradients with respect to alpha are supported, hence alpha can be learned (this was done here: https://arxiv.org/pdf/1909.00015.pdf). However I believe the current code only supports alpha >= 1 (I believe it should not be very hard to extend bisection for alpha < 1 though, but this won't be a sparse transformation). Is alpha < 1 crucial in your problem? I didn't quite get the connection with sum-normalized sigmoids, is the idea to consider sum-normalized "entmoids" with alpha < 1?
from entmax.
@andre-martins Since that I am able to use entmax to learn the alpha parameter, I would like to use it for learning both the optimal sparsity and temperature at the same time with a single alpha parameter. Yes, it would make a great addition to what I am working on (financial portfolio optimization). I would be more than glad if you can extend it for alpha < 1.0 in the future on your convenience.
from entmax.
It's merged on master now.
from entmax.
Hello @andre-martins & @bpopeters , sorry for late response due to extreme congestion, I will surely let you know about it.
from entmax.
Related Issues (20)
- Usage of alpha HOT 3
- Unexpected behaviour of sparsemax gradients for 3d tensors HOT 8
- entmax implementation for Tesnoflow 2 HOT 1
- Replicating the behaviour from the paper HOT 1
- entmax_bisect leads to loss becoming nan HOT 16
- Problem with sparse activations HOT 9
- Entmax fails when all inputs are -inf HOT 1
- Index -1 is out of bounds HOT 1
- A bug when alpha = 1 for entmax_bisect? HOT 3
- `entmax_bisect` is not stable around `alpha=1` HOT 3
- Release patch 1.0.1 with torch install_requires fix HOT 5
- Code to reproduce figure 3 HOT 1
- alpha-entmax
- Errors when using the loss function HOT 9
- entmax_bisect bugs with fp16/bf16 HOT 1
- Nans gradients for tensors with singleton dimensions for budget_bisect. HOT 5
- Instability of entmax bisect around 0? HOT 1
- EntmaxBisect forward requires unused argument HOT 2
- Sparse losses return nan when there is -inf in the input HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from entmax.