Comments (1)
As it turns out, we can use the same solution for both torch.float16
and torch.bfloat16
. This doesn't seem obvious because they offer different tradeoffs compared to full-precision floats: fp16 keeps the mantissa at the cost of reduced range, bf16 keeps the range at cost of reduced mantissa. But strangely enough we can, using the classic softmax stability trick of subtracting the largest logit from the vector. This makes intuitive sense for torch.float16
because of its reduced range, but what about for torch.bfloat16
?
It turns out that bfloat16
has weird problems far away from zero:
x = torch.tensor(0, dtype=torch.bfloat16, device="cuda")
x == (x - 1) # False, of course
y = torch.tensor(-500, dtype=torch.bfloat16, device="cuda")
y == (y - 1) # True?!?
So that's why the softmax stability trick works. Bringing it back to our earlier examples:
b = a.to(torch.float16)
entmax.entmax_bisect(b - b.max(), alpha=1.5) # one-hot!
c = a.to(torch.bfloat16)
entmax.entmax_bisect(c - c.max(), alpha=1.5) # one-hot!
Tests and PR upcoming.
from entmax.
Related Issues (20)
- Usage of alpha HOT 3
- Unexpected behaviour of sparsemax gradients for 3d tensors HOT 8
- entmax implementation for Tesnoflow 2 HOT 1
- Replicating the behaviour from the paper HOT 1
- entmax_bisect leads to loss becoming nan HOT 16
- Problem with sparse activations HOT 9
- Entmax fails when all inputs are -inf HOT 1
- Alpha value less than one? HOT 8
- Index -1 is out of bounds HOT 1
- A bug when alpha = 1 for entmax_bisect? HOT 3
- `entmax_bisect` is not stable around `alpha=1` HOT 3
- Release patch 1.0.1 with torch install_requires fix HOT 5
- Code to reproduce figure 3 HOT 1
- alpha-entmax
- Errors when using the loss function HOT 9
- Nans gradients for tensors with singleton dimensions for budget_bisect. HOT 5
- Instability of entmax bisect around 0? HOT 1
- EntmaxBisect forward requires unused argument HOT 2
- Sparse losses return nan when there is -inf in the input HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from entmax.