Comments (8)
Thanks for raising this - can you provide a bit more context about how you're using the library? (I.e., are you calling it directly, or through some other package?) Would it be easy to just add parallel=False
to your call site?
I don't think a warning is necessarily appropriate, but we can certainly consider reverting the default back to parallel=False. The thinking here was that multiprocessing applications like yours are probably not the default / most common setting, but we do want to make it easy to limit multi-core usage for exactly this kind of application.
It does sound like for your application, it would actually make a lot of sense to limit numba core usage globally (and not just for resampy) - that way it would apply to all numba-parallel usage and not just our package. This can be done through a global environment variable as noted in #106 .
from resampy.
Thanks for the quick reply!
So yes, if there's a way to auto-detect being in a subprocess, and disable it accordingly, that would also be great.
Looking at #106, I think what they are seeing is this exact same problem. I'm also using librosa, and librosa hasn't updated their call to resampy.resample
to specify parallel
. So I don't have the control of turning it off. Took me some time to finally get to the bottom of this...
I'll test numba.set_num_threads(1)
to see if it works as well.
from resampy.
So yes, if there's a way to auto-detect being in a subprocess, and disable it accordingly, that would also be great.
That seems pretty unlikely to me. I wouldn't trust such an auto-detection to be reliable, since there are many ways in which an upstream caller might be executing in parallel. I'd rather just have a good default setting and be upfront about documentation.
Looking at #106, I think what they are seeing is this exact same problem.
That seems likely, but their example seems very complicated and not something I could easily reproduce locally.
I'll test
numba.set_num_threads(1)
to see if it works as well.
Please report back if it works!
from resampy.
Ah it doesn't work unfortunately. That line effectively sets numba threads to 1, but I still got this error: Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
I'm guessing it's because numba will still try to spawn the processes even though num_threads is set to one? I don't know, this is really outside of my domain haha.
from resampy.
That's a bummer, but good to know! To the previous question: are you able to call resampy directly with parallel=False? That will force it to use the single-threaded implementation, and should also fix it for you. The catch here is that if you're using resampy through an intermediate library (eg librosa) the API isn't directly exposed yet.
from resampy.
Yeah you're right. I'll explore that option as well when I get time. For now the old version works well.
from resampy.
We had the same problem with librosa. It was finally traced to a problem with the dependency package resampy. I'm here to leave a comment. The easiest way to reproduce this is to use the multiprocessing library and run it in parallel.
from resampy.
Have same issue with huggingface huggingface/datasets#4820
from resampy.
Related Issues (20)
- v0.3.0 produces other results than v0.2.2 HOT 3
- Resampy 0.3.0 is slow for multi-dimensional input HOT 5
- Issues with Numba 0.55.2 HOT 4
- Resampy 0.3.1 slows down other libraries HOT 43
- Bringing import time back down
- Edge Case, rounding error in length calculation HOT 3
- Missing argument in guvectorize decorator ? HOT 2
- Quality issues? HOT 5
- resampling to the same hz HOT 1
- Incompatible with latest numba HOT 3
- Best quality resampling?
- PyPI status HOT 2
- Add linting CI
- Add pypi packaging workflow HOT 1
- Drop legacy python support
- Fix documentation builds HOT 1
- Update README badges
- Improve default filters and make the whole process reproducible HOT 14
- Cache filters on load
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from resampy.