sonos / pyflac Goto Github PK

View Code? Open in Web Editor NEW

125.0 4.0 7.0 7.14 MB

Real-time lossless audio compression in Python

Home Page: https://pyflac.readthedocs.io

License: Apache License 2.0

Python 31.72% C 66.72% Shell 1.57%

audio lossless compression python flac realtime

pyflac's People

Contributors

Stargazers

Watchers

Forkers

ali1234 stjordanis drr3d theintrepidthird rahedges hughsw brandyn

pyflac's Issues

Add support for 24bit

Currently the package only supports int16 data format.

An int24 type is not supported by numpy which makes this non-trivial with the current design.

decoder.StreamingDecoder leaks active threads

We have a server that processes flac streams. We've noticed that the worker thread that StreamingDecoder creates is not always joined when the decoder is finished. So the number of active threads increases without bound over time.

Looking at the code we see some race conditions. For instance, the while loop in decoder.py: _read_callback will never end if this loop is active with an empty _buffer at the point in time when the main thread calls StreamingDecoder.finish() to set the _done flag:

pyFLAC/pyflac/decoder.py

Lines 333 to 337 in e17d036

    
           while len(decoder._buffer) == 0: 
        
               # ---------------------------------------------------------- 
        
               # Wait until there is something in the buffer 
        
               # ---------------------------------------------------------- 
        
               time.sleep(0.01)

(Sigh, the use of sleep() and timeouts is always a red-flag in multithreading code anyway.)

We are using a modified version of StreamingDecoder that has no sleeps or timeouts and it doesn't leak threads. If there's interest we can post a couple of PRs (including a regression test).

only produce empty flac file

Hello everyone,
I am using the code: pyflac example.wav -o pyflac.flac to get FLAC file.
However, the output file is always empty and there is no warning or error.
Can you give me some hints for it?

Also, I can successfully save FLAC file with soundfile, which partially validate my FLAC library is correct.

Illegal Instruction

Hello,
I am trying to use pyflac on raspberry pi zero, but I am getting Illegal instruction error.

Collecting pyflac
  Downloading https://files.pythonhosted.org/packages/21/ec/621fd90f1b98477fa1c147edab0ea54cb8bd4b34773f698d5da665a197f3/pyFLAC-2.0.0.tar.gz (1.8MB)
     |████████████████████████████████| 1.8MB 1.0MB/s
Requirement already satisfied: cffi>=1.4.0 in /home/pi/.local/lib/python3.8/site-packages (from pyflac) (1.15.0)
Collecting SoundFile>=0.8.0 (from pyflac)
  Downloading https://files.pythonhosted.org/packages/eb/f2/3cbbbf3b96fb9fa91582c438b574cff3f45b29c772f94c400e2c99ef5db9/SoundFile-0.10.3.post1-py2.py3-none-any.whl
Requirement already satisfied: numpy in /home/pi/.local/lib/python3.8/site-packages (from pyflac) (1.22.4)
Requirement already satisfied: pycparser in /home/pi/.local/lib/python3.8/site-packages (from cffi>=1.4.0->pyflac) (2.21)
Building wheels for collected packages: pyflac
  Building wheel for pyflac (setup.py) ... done
  Created wheel for pyflac: filename=pyFLAC-2.0.0-cp38-cp38-linux_armv6l.whl size=1938080 sha256=30e340d066abeca1358fca4f26c1f82151e68ece9a1880f5345360b523eecd3b
  Stored in directory: /home/pi/.cache/pip/wheels/5a/96/19/863b328487fa805e47b53d7da91885d99770ceae1af630e38e
Successfully built pyflac
Installing collected packages: SoundFile, pyflac
Successfully installed SoundFile-0.10.3.post1 pyflac-2.0.0

Python 3.8.1 (default, May 25 2022, 20:18:49)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import soundfile
>>> import pyflac
Illegal instruction

Tests don't properly check the raised exception

pyFLAC/tests/test_encoder.py

Lines 127 to 129 in 19b3322

    
           with self.assertRaises(EncoderInitException) as err: 
        
               self.encoder._init() 
        
               self.assertEqual(str(err), 'FLAC__STREAM_ENCODER_INIT_STATUS_INVALID_BLOCK_SIZE')

The third line here has no effect. If encoder._init() raises an exception (which we expect it to), all subsequent lines won't run at all.

Compress more than 2 channels

Hi,

First of all thank you for the package and the examples!

I'm currently looking for methods to compress data from neural electrophysiology recorded from the brain. We use neural probes with many channels (e.g. 384) and I was wondering if there is a way to tweak the library to allow for more than 2 channels. Is it possible?

Thank you in advance!
Alessio

DecoderProcessException and multiprocessing?

I am getting an occasional error while using python multiprocessing. I am unsure if the problem is related to multiprocessing, but I am processing millions of flac files. I have varified that the audios that are involved in the error seem to load when tried again in a normal single process and multi-process environment, or files that didn't have problem one run may throw this error the next run. So it doesn't seem to be correlated to the files themselves.

Any help would be appriciated, Thank you

Code to read flac file from in memory buffer

I use in memory for reading files from web or parse the raw byte header for wav or flac tokens

@dataclass
    class AudioBuffer:
        audio: List[npt.NDArray[Any]] = field(default_factory=list)
        sample_rate: int = 0
        num_channels: int = 0

def _callback(
        audio: npt.NDArray[Any],
        sample_rate: int,
        num_channels: int,
        num_samples: int,
        buffer: AudioBuffer,
    ) -> None:
        buffer.audio.append(audio)
        buffer.sample_rate = sample_rate
        buffer.num_channels = num_channels

def decode(binary: bytes) -> Tuple[npt.NDArray[np.float_], int, int]:
        flac_buffer = FlacDecoder.AudioBuffer()
        flac_decoder = pyflac.StreamDecoder(partial(FlacDecoder._callback, buffer=flac_buffer))

        flac_decoder.process(binary)
        flac_decoder.finish()  # wait until finished processing bytes
        decoded_flac = np.concatenate(flac_buffer.audio).squeeze()
        sample_rate = flac_buffer.sample_rate
        num_channels = flac_buffer.num_channels

        return (
           normalize_if_int(decoded_flac),  # Normlize -1 <= x <=1
            sample_rate,
            num_channels,
        )

    def from_file(file_path: str) -> Tuple[npt.NDArray[np.float_], int, int]:
        with open(file_path, "rb") as fb:
            audio_bytes = fb.read()

        return decode(audio_bytes)

Stack Trace of error

Traceback (most recent call last):
  File "train_single_pipeline_dist.py", line 1319, in main
    mp.spawn(worker, nprocs=ngpus_per_node, args=(world_size, ngpus_per_node, train, cfg))
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)

    exec_fn(rank, device_id, device, ngpus_per_node, cfg, training_callbacks)
  File "/share/private/jake/dev/nansy/nansy++/train_single_pipeline_dist.py", line 833, in train
    train_loop(
  File "/share/private/jake/dev/nansy/nansy++/train_single_pipeline_dist.py", line 926, in train_loop
    for step, batch in enumerate(train_loader):
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1313, in _next_data
    return self._process_data(data)
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
pyflac.decoder.DecoderProcessException: Caught DecoderProcessException in DataLoader worker process 7.
Original Traceback (most recent call last):
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/share/private/jake/dev/nansy/nansy++/dataset/loader.py", line 179, in __getitem__
    audio, spkr, path, voice_type, lang = self._load_sample(idx)
  File "/share/private/jake/dev/nansy/nansy++/dataset/loader.py", line 399, in _load_sample
    audio, _, sr = load_audio(path)
  File "/share/private/jake/dev/nansy/nansy++/dataset/utils.py", line 645, in load_audio
    audio_np, audio_len, sr, _ = load_audio_and_return_format(audio_file)
  File "/share/private/jake/dev/nansy/nansy++/dataset/utils.py", line 595, in load_audio_and_return_format
    audio_np, sr, _ = FlacDecoder()(bytes_data)
  File "/share/private/jake/dev/nansy/nansy++/dataset/utils.py", line 556, in __call__
    flac_decoder.finish()  # wait until finished processing bytes
  File "/home/conda_user/miniconda3/lib/python3.8/site-packages/pyflac/decoder.py", line 231, in finish
    raise DecoderProcessException(self._error)
pyflac.decoder.DecoderProcessException: A fatal read, write, or memory allocation error occurred

Soundfile doesn't yet support Apple Silicon

This currently blocks pyFLAC supporting Apple Silicon. See bastibe/python-soundfile#325

Streaming decoder is very slow

I have been experimenting with using the encoder and decoder on some data and I have found that the stream decoder is several orders of magnitude slower than the encoder.

I have made some very minor alterations to the passthrough.py example file, starting a timer with the perf_counter_ns() command from the time module, and printing the result for both encoding and decoding. For example, encoding a file takes 0.59 ms, while decoding it seems to take over 3 seconds.

This appears to be limited to the stream decoder, where the file decoder offers performance that is on parity with the encoder. This is including the overhead that the file decoder has with performing several reads and writes to files, while the stream encoder should be much faster.

sonos / pyflac Goto Github PK

pyflac's People

Contributors

Stargazers

Watchers

Forkers

pyflac's Issues

Add support for 24bit

decoder.StreamingDecoder leaks active threads

only produce empty flac file

Illegal Instruction

Tests don't properly check the raised exception

Compress more than 2 channels

DecoderProcessException and multiprocessing?

Code to read flac file from in memory buffer

Stack Trace of error

Soundfile doesn't yet support Apple Silicon

Streaming decoder is very slow

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	while len(decoder._buffer) == 0:
	# ----------------------------------------------------------
	# Wait until there is something in the buffer
	# ----------------------------------------------------------
	time.sleep(0.01)

	with self.assertRaises(EncoderInitException) as err:
	self.encoder._init()
	self.assertEqual(str(err), 'FLAC__STREAM_ENCODER_INIT_STATUS_INVALID_BLOCK_SIZE')