Trouble training model without Crepe f0 estimation about ddsp HOT 3 CLOSED

Hi, thanks for the in-depth study and posting all the resources.

This is actually expected behavior at the moment. As we said in the paper, when training on a full dataset like NSynth the f0 encoder model can get a small loss and learn to generate audio that a CREPE model classifies as having the right f0, but does not currently estimate the correct f0 internally. It often falls into the local minima of predicting an interger multiple of f0 and then doing the best to match the data by manipulating the harmonic distribution. Unlike other neural networks, this problem will be even more exacerbated in fitting a single datapoint (not having the stochasticity of SGD to help in optimization).

We have some follow-up work that overcomes these challenges, and are working on getting in prepared for a conference submission next month, at which time I'll clean it up and submit it to the repo. Sorry for the delay, or if the original paper was misleading, but I think there are actually several ways to tackle this challenge and we should hopefully have them robust and added soon.

from ddsp.

Understood. That's good to know and thanks for all the amazing work. I'm really excited about the developments. Should I close this issue for now?

from ddsp.

Yah, and I look forward to posting more when I have it :).

from ddsp.