sonos / pyflac Goto Github PK
View Code? Open in Web Editor NEWReal-time lossless audio compression in Python
Home Page: https://pyflac.readthedocs.io
License: Apache License 2.0
Real-time lossless audio compression in Python
Home Page: https://pyflac.readthedocs.io
License: Apache License 2.0
I am getting an occasional error while using python multiprocessing. I am unsure if the problem is related to multiprocessing, but I am processing millions of flac files. I have varified that the audios that are involved in the error seem to load when tried again in a normal single process and multi-process environment, or files that didn't have problem one run may throw this error the next run. So it doesn't seem to be correlated to the files themselves.
Any help would be appriciated, Thank you
@dataclass
class AudioBuffer:
audio: List[npt.NDArray[Any]] = field(default_factory=list)
sample_rate: int = 0
num_channels: int = 0
def _callback(
audio: npt.NDArray[Any],
sample_rate: int,
num_channels: int,
num_samples: int,
buffer: AudioBuffer,
) -> None:
buffer.audio.append(audio)
buffer.sample_rate = sample_rate
buffer.num_channels = num_channels
def decode(binary: bytes) -> Tuple[npt.NDArray[np.float_], int, int]:
flac_buffer = FlacDecoder.AudioBuffer()
flac_decoder = pyflac.StreamDecoder(partial(FlacDecoder._callback, buffer=flac_buffer))
flac_decoder.process(binary)
flac_decoder.finish() # wait until finished processing bytes
decoded_flac = np.concatenate(flac_buffer.audio).squeeze()
sample_rate = flac_buffer.sample_rate
num_channels = flac_buffer.num_channels
return (
normalize_if_int(decoded_flac), # Normlize -1 <= x <=1
sample_rate,
num_channels,
)
def from_file(file_path: str) -> Tuple[npt.NDArray[np.float_], int, int]:
with open(file_path, "rb") as fb:
audio_bytes = fb.read()
return decode(audio_bytes)
Traceback (most recent call last):
File "train_single_pipeline_dist.py", line 1319, in main
mp.spawn(worker, nprocs=ngpus_per_node, args=(world_size, ngpus_per_node, train, cfg))
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
exec_fn(rank, device_id, device, ngpus_per_node, cfg, training_callbacks)
File "/share/private/jake/dev/nansy/nansy++/train_single_pipeline_dist.py", line 833, in train
train_loop(
File "/share/private/jake/dev/nansy/nansy++/train_single_pipeline_dist.py", line 926, in train_loop
for step, batch in enumerate(train_loader):
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
data = self._next_data()
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1313, in _next_data
return self._process_data(data)
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
raise exception
pyflac.decoder.DecoderProcessException: Caught DecoderProcessException in DataLoader worker process 7.
Original Traceback (most recent call last):
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/share/private/jake/dev/nansy/nansy++/dataset/loader.py", line 179, in __getitem__
audio, spkr, path, voice_type, lang = self._load_sample(idx)
File "/share/private/jake/dev/nansy/nansy++/dataset/loader.py", line 399, in _load_sample
audio, _, sr = load_audio(path)
File "/share/private/jake/dev/nansy/nansy++/dataset/utils.py", line 645, in load_audio
audio_np, audio_len, sr, _ = load_audio_and_return_format(audio_file)
File "/share/private/jake/dev/nansy/nansy++/dataset/utils.py", line 595, in load_audio_and_return_format
audio_np, sr, _ = FlacDecoder()(bytes_data)
File "/share/private/jake/dev/nansy/nansy++/dataset/utils.py", line 556, in __call__
flac_decoder.finish() # wait until finished processing bytes
File "/home/conda_user/miniconda3/lib/python3.8/site-packages/pyflac/decoder.py", line 231, in finish
raise DecoderProcessException(self._error)
pyflac.decoder.DecoderProcessException: A fatal read, write, or memory allocation error occurred
Hello everyone,
I am using the code: pyflac example.wav -o pyflac.flac to get FLAC file.
However, the output file is always empty and there is no warning or error.
Can you give me some hints for it?
Also, I can successfully save FLAC file with soundfile, which partially validate my FLAC library is correct.
We have a server that processes flac streams. We've noticed that the worker thread that StreamingDecoder creates is not always joined when the decoder is finished. So the number of active threads increases without bound over time.
Looking at the code we see some race conditions. For instance, the while
loop in decoder.py: _read_callback
will never end if this loop is active with an empty _buffer
at the point in time when the main thread calls StreamingDecoder.finish()
to set the _done
flag:
Lines 333 to 337 in e17d036
sleep()
and timeouts is always a red-flag in multithreading code anyway.)
We are using a modified version of StreamingDecoder that has no sleeps or timeouts and it doesn't leak threads. If there's interest we can post a couple of PRs (including a regression test).
I have been experimenting with using the encoder and decoder on some data and I have found that the stream decoder is several orders of magnitude slower than the encoder.
I have made some very minor alterations to the passthrough.py example file, starting a timer with the perf_counter_ns()
command from the time
module, and printing the result for both encoding and decoding. For example, encoding a file takes 0.59 ms, while decoding it seems to take over 3 seconds.
This appears to be limited to the stream decoder, where the file decoder offers performance that is on parity with the encoder. This is including the overhead that the file decoder has with performing several reads and writes to files, while the stream encoder should be much faster.
Hi,
First of all thank you for the package and the examples!
I'm currently looking for methods to compress data from neural electrophysiology recorded from the brain. We use neural probes with many channels (e.g. 384) and I was wondering if there is a way to tweak the library to allow for more than 2 channels. Is it possible?
Thank you in advance!
Alessio
Currently the package only supports int16
data format.
An int24
type is not supported by numpy
which makes this non-trivial with the current design.
This currently blocks pyFLAC supporting Apple Silicon. See bastibe/python-soundfile#325
Hi,
I would like to use this library, however I'm having problems to get installed.
I'm using a virtual environment for Python (3.11.2)
The documenation states that for linux distributions we need to execute an additional step. However, the SounFile link in the documentation gets a 404 error.
Anyway, assuming that it was related to the SoundFile library, I tried to install the libsndfile library using
apt-get install -y libsndfile1
And it gets installed correctly. But when I try again the pip installation, it fails with the same error:
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pyflac
Using cached pyFLAC-3.0.0.tar.gz (1.8 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [40 lines of output]
Traceback (most recent call last):
File "/opt/webrtc/goog/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/opt/webrtc/goog/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/webrtc/goog/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 366, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 14, in <module>
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 158, in setup
dist.parse_config_files()
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 632, in parse_config_files
pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 68, in apply_configuration
config = read_configuration(filepath, True, ignore_option_errors, dist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 104, in read_configuration
asdict = load_file(filepath) or {}
^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py", line 33, in load_file
from ..compat.py310 import tomllib
File "/tmp/pip-build-env-4gcx_umb/overlay/lib/python3.11/site-packages/setuptools/compat/py310.py", line 8, in <module>
import tomllib
File "/usr/lib/python3.11/tomllib/__init__.py", line 7, in <module>
from ._parser import TOMLDecodeError, load, loads
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 936, in exec_module
File "<frozen importlib._bootstrap_external>", line 1069, in get_code
File "<frozen importlib._bootstrap_external>", line 729, in _compile_bytecode
ValueError: bad marshal data (unknown type code)
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Probably I'm doing something wrong. Do you have any clues or suggestions that I could use?
Thanks in advance,
Hello,
I am trying to use pyflac on raspberry pi zero, but I am getting Illegal instruction error.
Collecting pyflac
Downloading https://files.pythonhosted.org/packages/21/ec/621fd90f1b98477fa1c147edab0ea54cb8bd4b34773f698d5da665a197f3/pyFLAC-2.0.0.tar.gz (1.8MB)
|████████████████████████████████| 1.8MB 1.0MB/s
Requirement already satisfied: cffi>=1.4.0 in /home/pi/.local/lib/python3.8/site-packages (from pyflac) (1.15.0)
Collecting SoundFile>=0.8.0 (from pyflac)
Downloading https://files.pythonhosted.org/packages/eb/f2/3cbbbf3b96fb9fa91582c438b574cff3f45b29c772f94c400e2c99ef5db9/SoundFile-0.10.3.post1-py2.py3-none-any.whl
Requirement already satisfied: numpy in /home/pi/.local/lib/python3.8/site-packages (from pyflac) (1.22.4)
Requirement already satisfied: pycparser in /home/pi/.local/lib/python3.8/site-packages (from cffi>=1.4.0->pyflac) (2.21)
Building wheels for collected packages: pyflac
Building wheel for pyflac (setup.py) ... done
Created wheel for pyflac: filename=pyFLAC-2.0.0-cp38-cp38-linux_armv6l.whl size=1938080 sha256=30e340d066abeca1358fca4f26c1f82151e68ece9a1880f5345360b523eecd3b
Stored in directory: /home/pi/.cache/pip/wheels/5a/96/19/863b328487fa805e47b53d7da91885d99770ceae1af630e38e
Successfully built pyflac
Installing collected packages: SoundFile, pyflac
Successfully installed SoundFile-0.10.3.post1 pyflac-2.0.0
Python 3.8.1 (default, May 25 2022, 20:18:49)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile
>>> import pyflac
Illegal instruction
Lines 127 to 129 in 19b3322
The third line here has no effect. If encoder._init()
raises an exception (which we expect it to), all subsequent lines won't run at all.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.