Support audio format for tensorflow-io about io HOT 8 CLOSED

@yongtang thanks for the clarification.

WAVDataset will performs better with really huge WAV files (e.g., GBs) as decode_wav requires disk IO to read the entire file into memory before doing any decoding.

does this mean that WAVDataset supports seeking/memory mapping?
Maybe I can elaborate the use case: Imagine you have few but rather long audio recordings (like in dcase or music datasets, a WAVDataset could give you access to the full decoded file. However, in practice, you would only train with chunked data of a few seconds per sample. The chunking could be done on the fly buy supporting seeking in the actual data loading. If this is not what you meant, I would propose to open an issue, if you agree that this could be a useful addition.

WAVDataset also simply the interaction with tf.keras.

because it can directly be used with tf.keras.fit?

For FFmpeg operators, we already have support for ffmpeg video operators. The audio operators with ffmpeg should not be very difficult to add (planed to add but haven't had enough time yet with many changes of 1.x -> 2.0).

that would be great!

from io.

Added a PR #307 as the first step to have FFmpeg support for audio streams.

from io.

@yongtang @terrytangyuan great to see that audio is getting some focus here in tf/io with the recent additions in 0.6.0. Regarding the WAVDataset could you please comment on this from the user perspective?

• comparing WAVInput and tf2.0 decode_wav, which one should one use to read wavs?
• for a wav reading pipeline, is WAVDataset faster than manually mapping tf.audio.decode_wav?
• do you plan to add ffmpeg operators?

from io.

@faroit WAVDataset and decode_wav have different use cases but the biggest difference is that, WAVDataset is a data pipeline that directly support tf.keras for training and inference. WAVDataset will be similar in performance, except WAVDataset will performs better with really huge WAV files (e.g., GBs) as decode_wav requires disk IO to read the entire file into memory before doing any decoding.

In essence, WAVDataset is like decode_wav as both are part of the TF's graph. while decode_wav is a node (op) in the graph, WAVDataset optimize and is really a small subgraph in the graph. WAVDataset also simply the interaction with tf.keras.

For FFmpeg operators, we already have support for ffmpeg video operators. The audio operators with ffmpeg should not be very difficult to add (planed to add but haven't had enough time yet with many changes of 1.x -> 2.0).

from io.

@faroit

I would propose to open an issue, if you agree that this could be a useful addition.

Definitely! please open an issue.

because it can directly be used with tf.keras.fit?

Yes. You can use WAVDataset in a similar way as the beginner tutorial with tf.keras (model.fit).

from io.

@faroit I added a PR #406 for reading a chunk of WAV file. You can use read_wav and pass start=start, count=count to only read a chunk of the samples.

I also tried to add 24bit support, though I need a sample 24bit WAV file to see the actual memory layout to figure out how to fit 24bit into an int32.

from io.

@yongtang, here's an example audio file of a 24-bit WAV PCM file created from a MP3 with the SoX command:
sox example.mp3 -b 24 example.wav

example.zip

Input File     : 'example.wav'
Channels       : 2
Sample Rate    : 44100
Precision      : 24-bit
Duration       : 00:00:20.00 = 882000 samples = 1500 CDDA sectors
File Size      : 5.29M
Bit Rate       : 2.12M
Sample Encoding: 24-bit Signed Integer PCM

from io.

@carlthome Created a PR #409 for 24 bit support.

from io.