Voice Flickering during RT Use about dtln HOT 15 CLOSED

Hi,

I added some code to illustrate, how real time processing with overlap add works. I hope that removes the flickering. I will add the script to the repo.

Best,
Nils

import soundfile as sf
import numpy as np
import tensorflow as tf

block_len = 512
block_shift = 128

# load model
model = tf.saved_model.load('.pretrained_model/dtln_saved_model')
infer = model.signatures["serving_default"]

# load audio file (please change)
audio,fs = sf.read('audioset_realrec_airconditioner_2TE3LoA2OUQ.wav')
# preallocate output audio
out_file = np.zeros((len(audio)))
# calculate number of blocks
num_blocks = (audio.shape[0] - (block_len-block_shift)) // block_shift
# iterate over the number of blcoks        
for idx in range(num_blocks):
    # take the block
    in_block = audio[idx*block_shift:(idx*block_shift)+block_len]
    # create a batch dimension of one
    in_block = np.expand_dims(in_block, axis=0).astype('float32')
    # process one block
    out_block= infer(tf.constant(in_block))['conv1d_1']
    # write block to output file
    out_file[idx*block_shift:(idx*block_shift)+block_len] = out_file[idx*block_shift:(idx*block_shift)+block_len]  + out_block
    
sf.write('out.wav', out_file, fs) 
print('Test_finished.')

from dtln.

It could also maybe come from low level input. Try a model with normalization of the STFT features from the pretrained_model folder.

from dtln.

It is working now. Thank you! The overlap code did the trick.

However, I want to add, while processing live microphone feed, flicker is not completely gone. If I'm taking chunks of size 512, the previous issue is re-occurring. If I increase the chunk size, the flicker is far less frequent - almost negligible for an average word, but can be heard when pronouncing fairly long words or like humming a song. This issue persists even after using the norm_500h_saved_model

from dtln.

The block length and block shift are fixed. If you would like to use another length and shift, you have to retrain the model.

Another reason for flickering can be the sound card and the library for interacting with the sound card. But I can’t deliver support on that one. Did you check if a pass through, copying the block from the microphone directly to the output, is flickering free?

from dtln.

Yes, the pass through, directly copying the input block to the output without any processing, is flickering free.

from dtln.

Are you using 16k sampling frequency?

The model needs some CPU time and so there is maybe not enough for the audio driver. Real time audio is always tricky handling all the resources.

I added a file for real time processing today, check if you doing it similar.

After which time occurs the flickering?

The model can handle silence relatively well, but maybe integrate a gate, so the processing is only performed if the energy in the current block coming from the mic is above a threshold.

from dtln.

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

from dtln.

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

directly playing back after the model inference

from dtln.

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

directly playing back after the model inference

Does it have flickering, if you write it directly to a file?

from dtln.

Are you using 16k sampling frequency?

The model needs some CPU time and so there is maybe not enough for the audio driver. Real time audio is always tricky handling all the resources.

I added a file for real time processing today, check if you doing it similar.

After which time occurs the flickering?

The model can handle silence relatively well, but maybe integrate a gate, so the processing is only performed if the energy in the current block coming from the mic is above a threshold.

Yes, using 16k sampling rate and the real-time code you posted, which made the flicker sparse, but unfortunately not totally negligible.

No issues during silence.

I think the flicker is due to the process delay before each block is glued back into the output stream. Because, it is happening at a fixed interval, and the length of this interval is directly proportional to the block size.

from dtln.

@vinod1234567890 perhaps the flicker is because of the audio loop ? After you process the input from mic saving to a file ? Or directly playing back?

directly playing back after the model inference

Does it have flickering, if you write it directly to a file?

Yes. Exaclty similar to the live playback

from dtln.

Please post the code of your loop.
Did you check against the code I added to the repo? It is a bit different to the code here in the issue.

from dtln.

Does it work now?

from dtln.

Does it work now?

No. The earlier code, which was posted here in the issue, was giving lesser flicker. The one in the repo is resulting in a chopped kind of feel.

Here is what I'm currently using and working better than the one in the repo:

        #out_put container
        self.out_file = np.zeros((4096),dtype='float32')
        # iterate over the number of blcoks        
        for idx in range(self.num_blocks):
            # take the block
            in_block = audio[idx*self.block_shift:(idx*self.block_shift)+self.block_len] #here the audio size is 4096
            # create a batch dimension of one
            in_block = tf.expand_dims(in_block,axis=0)
            # process one block
            out_block= self.infer(in_block)['conv1d_1']
            # write block to output file
            self.out_file[idx*self.block_shift:(idx*self.block_shift)+self.block_len] = self.out_file[idx*self.block_shift:(idx*self.block_shift)+self.block_len]  + out_block
        out_bytes = self.out_file.tobytes()

from dtln.

I added a file: real_time_dtln_audio.py. Check it out. The audio works perfectly on my really old Macbook.

from dtln.