kyungyunlee / ismir2018-revisiting-svd Goto Github PK

Hi @kyungyunlee ,
thanks for your repo and your ideas.
I am new in the field of audio processing and I would like to know how features are treated in the preprocessing phase in the specific case of CNN training for SVD, and how prediction can be associated to input features in order to apply mask ( 0 if no_voice , 1 if voice) and recreate audio of same length where it can be possible to hear voice if prediction is VOICE, and no audio if prediction is NO_VOICE.

SR = 22050
FRAME_LEN = 1024
HOP_LENGTH = 315
CNN_INPUT_SIZE = 115 # 1.6 sec
CNN_OVERLAP = 5 # Hopsize of 5 for training, 1 for inference
N_MELS = 80
CUTOFF = 8000 # fmax = 8kHz

1. Step: Load an audio file as a floating point time series.

y, _ = librosa.load("audio.mp3", sr=22050)

From this I get 4280832 samples.

2. Step: Mel Spectrogram

x = log_melgram(y, SR=22050, FRAME_LEN=1024, HOP_LENGTH=315, N_MELS=80, 27.5, 8000)

Size of x results (80, 13590).

3. Step: Segmentation

for i in range(0, x.shape[1] - CNN_INPUT_SIZE, 1):
    x_segment = x[:, i: i + CNN_INPUT_SIZE]
    # pick the center frame label 
    total_x.append(x_segment)

# Normalization steps
X = (total_x - mean) / std
X = np.expand_dims(X, axis=3)

After this step total_x has shape (13475, 80, 115,1)

So these are the main steps in order to get X that is fed to the network.
What is not so clear to me is the transition between Step 2 and Step 3 , so why (80, 13590) dimension becomes (13475, 80, 115) and what is its meaning.
That is the keypoint I think also to understand how return back to audio where I can apply SVD prediction of the network and build an audio with SVD and with the same original shape.
VAD prediction has shape (13475, 1).

Thank you very much

I want to use this for my voice detection(RNN-VD).

But i don't know what y_pred means.

I know y_pred is (segment,RNN_INPUT_SIZE,1), but in one segment there's RNN_INPUT_SIZE probabilities and i don't know which to choose.

can you help?

I've been thinking a lot about this code fragment in
https://github.com/kyungyunlee/ismir2018-revisiting-svd/blob/master/leglaive_lstm/audio_processor.py
in function process_single_audio (Compute double stage HPSS for the given audio file) in lines 24-33:

    audio_src, _ = librosa.load(audio_file, sr=SR)
    # Normalize audio signal
    audio_src = librosa.util.normalize(audio_src)
    # first HPSS
    D_harmonic, D_percussive = ono_hpss(audio_src, N_FFT1, N_HOP1)
    # second HPSS
    D2_harmonic, D2_percussive = ono_hpss(D_percussive, N_FFT2, N_HOP2)

    assert D2_harmonic.shape == D2_percussive.shape
    print(D2_harmonic.shape, D2_percussive.shape)

The D2_harmonic and D2_percussive are calculated from the D_percussive component.

Is this right? I'm currently checking the original paper and i will keep you updated if i discover something.

This seems kinda odd, since my intuition says that the harmonic component has more importance to voice activity detection.

The link to Jamendo dataset redirects to a weird Chinese webpage