Fine-Tuning TANGO 2 about tango HOT 4 CLOSED

Hey!

Thanks. You may just run this: accelerate launch train.py
--train_file="data/train_audiocaps.json" --validation_file="data/valid_audiocaps.json" --test_file="data/test_audiocaps_subset.json"
--hf_model "declare-lab/tango2" --unet_model_config="configs/diffusion_model_config.json" --freeze_text_encoder
--gradient_accumulation_steps 4 --per_device_train_batch_size=2 --per_device_eval_batch_size=2 --augment
--learning_rate=3e-5 --num_train_epochs 40 --snr_gamma 5
--text_column captions --audio_column location --checkpointing_steps="best"

Replace the train_file with your file. You will need to maintain the format accepted by the code: https://raw.githubusercontent.com/declare-lab/tango/master/data/train_audiocaps.json

from tango.

@soujanyaporia thank you very much for the reply.

In this case would I need to train the model from scratch with Audiocaps + Alpaca + My dataset?
Or is it pulling the tango2 checkpoint and would just finetune the trained over my new train.json file?

Edit:
Same goes for the valid_audiocaps.json and test_audiocaps_subset.json files: do I need to replace them with my own dataset files?

And souldn't I be running tango2-train.py instead?
Or is the pipeline to finetune a model with train.py and only later use that checkpoint to finetune with tango2?

from tango.

is it pulling the tango2 checkpoint and would just finetune the trained over my new train.json file?

Yes!

Same goes for the valid_audiocaps.json and test_audiocaps_subset.json files: do I need to replace them with my own dataset files?

Yes, you can but not necessary.

And souldn't I be running tango2-train.py instead?

No! because tango2 is trained using DPO. For fine-tuning, you do not need that.

from tango.

Thank you very much for your help, I will be closing the issue!

One other question would be, how many hours of recordings would you deem necessary for finetuning?
I know this could be subjective depending on the task at hand, but any estimates would be much appreciated.

from tango.