Comments (10)
Yes, I will try to figure out. I will check what test the test_layer
is doing to see why is not passing... The CPU version passes, but not the GPU version. But then when the test for equality in CPU and GPU passes, too, which is really odd.
Maybe rewriting the whole thing from scratch is easier than trying to figure out the typo, haha.
from dlib.
Yeah I noticed this a bit ago but didn't figure it out when I looked. Need to look again though and figure it out. Although maybe you will figure it out first :) Must just be some subtle typo somewhere.
from dlib.
Yeah hard to say, bugs can be anywhere and you never know until after the fact 🤷 :D
from dlib.
It's really weird. If I change this line:
Line 605 in 46e59a2
to the size that
test_layer
tests
resizable_tensor x(4, 2, 2, 4);
Then, this one fails: the values are entirely different. But with the previous size, the values are the same.
Line 650 in 46e59a2
from dlib.
I will tackle this again at some point. Sometimes, when things like this happen and I have absolutely no idea why, I question myself a lot...
from dlib.
Na everyone thinks that from time to time. Don't sweat it :)
from dlib.
I could be naive here, is there a reason why Layernorm isn't using CUDNN ?
Will cudnnNormalizationForwardInference
, cudnnNormalizationForwardTraining
and cudnnNormalizationBackward
work ?
It looks like those functions can be used for batchnorm, layernorm and groupnorm.
from dlib.
Oh, I wasn't aware it was possible to do Layer Normalization with cuDNN.
This link says that: https://docs.nvidia.com/deeplearning/cudnn/api/cudnn-ops-library.html#cudnnnormalizationforwardtraining is deprecated in v9.
Could you please show me how to use it for LayerNorm? I can't see any mention in the API.
It seems weird that this layer, now used in most Transformer-Based networks, has no cuDNN implementation…
from dlib.
Somewhere in the docs I read it could be used for multiple types of normalization...
I agree, it's hard to believe cudnn doesn't have first class support for it. Maybe libraries like Pytorch, GGML etc have moved away from cudnn and just use vanilla CUDA or https://github.com/NVIDIA/cccl to write their kernels from scratch. Don't know.
Also a lot of Transformers use different normalization functions now, like RMSNorm for example.
from dlib.
Ah, maybe you're referring to this? This is for FC or Conv mode in Batch Norm, which dlib already uses.
https://docs.nvidia.com/deeplearning/cudnn/api/cudnn-ops-library.html#cudnnnormmode-t
from dlib.
Related Issues (20)
- [Bug]: Fhog_example does not work with example data HOT 4
- [Bug]: HOT 1
- [Bug]: DLIB_IN_PROJECT_BUILD incorrectly puts -llapack into `Libs` of `dlib-1.pc` HOT 3
- [Bug]: ImportError: /usr/lib/libpango-1.0.so.0: undefined symbol: g_string_free_and_steal HOT 2
- [Bug]: having issues converting opencv mat to iplimage HOT 4
- [Bug]: Cannot build wheel on macOS since v19.24.2 HOT 4
- [Bug]: runtime error importing dlib HOT 7
- [Bug]: python setup.py install (_dlib_pybind11: not found error) / pip install dlib (CUDA disabled error) HOT 7
- [Bug]: dlib doesnt install. says cmake is required but have installed cmake. HOT 12
- [Bug]: RuntimeError in dlib.load_rgb_image: Failing expression was false with Python HOT 10
- [Bug]: dlib.DLIB_USE_CUDA: false HOT 6
- [Bug]: Importing fuction from dlib not working. HOT 3
- [Bug]: Installation failed in 'microcanda env' HOT 4
- [Bug]: dlib doesn't build with FFmpeg 7 HOT 8
- [Bug]: unable to access dlib.net from work because there is no valid signed certificate HOT 3
- [Bug]: Cmake regex error while trying to make install HOT 4
- [Bug]: Multiple errors using dlib on macOS (either dynamic linked or statically compiled) with test target HOT 3
- [Bug]: Segmentation fault (core dumped) on cuda 12.4 & cudnn 9.3.0
- dlib website update HOT 6
- [Bug]: Bus error: 10 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dlib.