kinwaicheuk / demucs_lightning Goto Github PK

Hello, i am getting this error when i try to train. Please do you have any advice on how to fix this? I have three folders inside my training dir, train, valid and test, but i still get this error.

Validation sanity check: 0it [00:00, ?it/s]Error executing job with overrides: ['trainer.precision=16', 'batch_size=1', 'data_augmentation=False', 'model=HDemucs']
Traceback (most recent call last):
File "train.py", line 130, in main
trainer.fit(model, train_loader,valid_loader)
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 741, in fit
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 777, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1199, in _run
self._dispatch()
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in _dispatch
self.training_type_plugin.start_training(self)
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 202, in start_training
self._results = trainer.run_stage()
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1289, in run_stage
return self._run_train()
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1311, in _run_train
self._run_sanity_check(self.lightning_module)
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1371, in _run_sanity_check
self._evaluation_loop._reload_evaluation_dataloaders()
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 170, in _reload_evaluation_dataloaders
self.trainer.reset_val_dataloader()
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py", line 548, in reset_val_dataloader
RunningStage.VALIDATING, model=pl_module
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py", line 504, in _reset_eval_dataloader
if has_len_all_ranks(dataloader, self.training_type_plugin, module)
File "/usr/local/lib/python3.7/site-packages/pytorch_lightning/utilities/data.py", line 119, in has_len_all_ranks
"Total length of Dataloader across ranks is zero. Please make sure that it returns at least 1 batch."
pytorch_lightning.utilities.exceptions.MisconfigurationException: Total length of Dataloader across ranks is zero. Please make sure that it returns at least 1 batch.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Thank you for your work.

1. I trained the model you provided. I want to know whether can provide test code. I write one myself, but the effect is not very good. I only use 20 epoch at present, and I don't know whether epoch is enough .
2. when I test ，would I need to normalize the test wav？
   I hope you can help me with the question

Hi sorry for my issues, I have trained a model and have a e=21-TRAIN_loss=0.10.ckpt file. How can i convert this file to a model or how do i use the checkpoint for separation on my own files?

Thanks for your help !

Thank you for your work.
Hello, I tried to use your source code to train HDemucs, adopted 8 3090s, the parameters did not change, and trained 360 epochs. The test results are as follows:
Test/nsdr: 5.5024
Test/nsrd_bass:5.4150
Test/nsrd_drums: 6.2600
Test/nsdr_other: 3.5410
Test/nsrd_vocals: 6.7935
paper result:
Test/nsdr: 7.68
Test/nsrd_bass:6.06
Test/nsrd_drums: 8.24
Test/nsdr_other: 5.59
Test/nsrd_vocals: 8.13
I would like to ask you whether you have reproduced the original close result, whether there are other points I did not notice, whether 360epoch is not enough. Maybe I should change monitor loss to nsdr

Hi, thank you for this lightning version of demucs.

Can you tell me what precision demucs trains with? Is it possible to change this to fp16/fp32? Please could you show me how to do this.

Thanks!

Hi,

Yesterday the Demucs team released V4 Demucs Hybrid Transformer. Please can i request for this to be added to your lightning version repo?

Thank you for your time.