Comments (3)
We have tests that are checking performance now (#63). These tests seem to pass consistently on my local machine as well as on Github Actions. A small number was failing on carmine though. Carmine is the only machine of the 3 I just mentioned that is using GPUs. We should indeed check out torch.cuda.seed().
from platalea.
With commit 1ffcb82, I made it possible to run the tox test suite on GPU, by setting PLATALEA_DEVICE="cuda:0"
(or another number) in the shell before running tox
(by default, environment variables are not forwarded into the tox environment, so you have to manually add the ones you want in tox.ini).
I tried running the testsuite on carmine and everything passes.
However, since we're using approximate value checks in the tests, this obviously does not tell us whether the code is now deterministic or whether we're still missing some random seed.
@cwmeijer, you mention in #78 (comment) that the values get rounded differently on different machines. Did you look into where this could have come from? For instance, if it's just about different library versions, we could pin those and maybe then use exact value equality asserts.
from platalea.
In branch https://github.com/spokenlanguage/platalea/tree/exact_equal_experiment_tests, I switched the assert to check results exactly, so I could look into determinism of the test results.
On carmine, using the tox test suite, I could not reproduce any non-determinism, not on GPU, nor on CPU. The tests fail and then print the diffs of the results. If I run tests multiple times, the diffs are exactly the same each time.
Note that I also tried installing the same dependency versions (at least the Python ones, can't control system dependencies). As @cwmeijer saw before, this still does not make the results consistent across machines. One reason may be that my laptop is a Mac, so there may be different basic underlying libraries that give inconsistent results with those on carmine's Linux setup. PyTorch indeed does not guarantee determinism across different platforms, see https://pytorch.org/docs/stable/notes/randomness.html.
So, it seems the different results we saw may just indeed have been platform or version differences. Given the fact that we cannot seem to get them equal to perform further tests and I could not reproduce non-determinism in the first place, I vote we close this issue.
from platalea.
Related Issues (20)
- Reproduce label_encoders.pkl HOT 3
- [ZRVG] CPC feature extraction
- Merge zerospeech21-vg branch? HOT 4
- add to RSD HOT 1
- Add issue template
- Create a conda package HOT 1
- Consider creating a conda channel HOT 1
- Use static analysis/code quality service HOT 2
- Description of the software HOT 1
- Supported platform HOT 1
- Add code of conduct
- Add code documentation HOT 1
- No help message when mandatory argument is missing. HOT 2
- CI fails HOT 3
- Test case with SpokenCOCO dataset
- Store dtype in metadata of memmap
- Versioning, releasing, DOI, Zenodo HOT 7
- Linting
- Fix warnings about code quality
- Add reference documentation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from platalea.