vectominist / spin Goto Github PK

Hi,

Thanks for this great paper. I have a multilingual dataset where I am extracting features using one of the SSL methods HuBERT/contentvec for my voice conversion task. The problem I m facing with my current model is losing content for a few converted audios. I wanted to try the SPIN approach for my SSL to validate the content loss. I cannot depend on phonemes as the dataset is multilingual. Could you please guide me to skip the phonemes to extract SSL features for my custom dataset which are wav files.

Hey!

Using this repository requires the s3prl repo to be modified, which makes for a bit of an ugly dependency chain if one wants to use spin inside another project -- would it be possible to either have the model be directly usable from huggingface or torch.hub?

Hi!

Nice work :) I'm working on implementing my own quantizer. I'm trying to use your pnmi.py to calculate the PNMI.

I'm trying to use compute_show_pnmi(uid2refs, uid2hyps, upsample=1, show_results: bool = False).
If I understand correctly, uid2hyps is supposed to be the output from my quantizer?
However, I'm a bit unclear about how to generate uid2refs, the ground truth against which the outputs are compared.

Could you please guide me on how to calculate or acquire the uid2refs? Specifically, how do I obtain the true phones to make this comparison?

I really appreciate any help you can offer.

When running the following code:

from s3prl.nn import S3PRLUpstream

model = S3PRLUpstream("spin_hubert_2048")

I get the error:
wandb.errors.UsageError: Unable to attach to run zcoo137r

Good job! Congratulations!
If I want to use it to extract the ssl feature for downstream tasks, should I use the layer12 output (768dim), or the layer12 output after pred_head (256dim)?