Corrupted 1b_lyrics checkpoint? about jukebox HOT 11 CLOSED

I had this problem after prematurely exiting a sampling execution before the prior models could finish downloading. You might have a truncated prior model file already saved in your cache folder.

Try clearing the cache found at /root/.cache/jukebox-assets (in google colab). For the 5b model I had to delete /root/.cache/jukebox-assets/models/5b/prior_level_0.pth.tar so that a fresh download would start instead of trying to read the existing file.

from jukebox.

@Desm0nt can you post the solution before you close it?

from jukebox.

@Jovonni on local machine I just run out of free space in /home drive and it's abort the downloading process. But I still don't know what the problem happens in CoLab and how to fix it.

from jukebox.

@Desm0nt @Jovonni Sorry, I still couldn't fix this problem. Does anybody have a solution? I have 50GB free space on the system, 128GB CPU RAM and 32 GB GPU memory.

from jukebox.

@apeguero1 Thank you for the prompt response -- it works now! :)

from jukebox.

I am getting this error when following the main instructions:

(jukebox) C:\Users\james\jukebox>python jukebox/sample.py --model=5b_lyrics --name=sample_5b --levels=3 --sample_length_in_seconds=20 --total_sample_length_in_seconds=180 --sr=44100 --n_samples=6 --hop_fraction=0.5,0.5,0.125 C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\librosa\util\decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location. Import requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0. from numba.decorators import jit as optional_jit C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\librosa\util\decorators.py:9: NumbaDeprecationWarning: An import was requested from a module that has moved location. Import of 'jit' requested from: 'numba.decorators', please update to use 'numba.core.decorators' or pin to Numba version 0.48.0. This alias will not be present in Numba version 0.50.0. from numba.decorators import jit as optional_jit Using cuda True {'name': 'sample_5b', 'levels': 3, 'sample_length_in_seconds': 20, 'total_sample_length_in_seconds': 180, 'sr': 44100, 'n_samples': 6, 'hop_fraction': (0.5, 0.5, 0.125)} Setting sample length to 881920 (i.e. 19.998185941043083 seconds) to be multiple of 128 Downloading from gce Traceback (most recent call last): File "jukebox/sample.py", line 279, in <module> fire.Fire(run) File "C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\fire\core.py", line 127, in Fire component_trace = _Fire(component, args, context, name) File "C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\fire\core.py", line 366, in _Fire component, remaining_args) File "C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\fire\core.py", line 542, in _CallCallable result = fn(*varargs, **kwargs) File "jukebox/sample.py", line 276, in run save_samples(model, device, hps, sample_hps) File "jukebox/sample.py", line 181, in save_samples vqvae, priors = make_model(model, device, hps) File "c:\users\james\jukebox\jukebox\make_models.py", line 191, in make_model vqvae = make_vqvae(setup_hparams(vqvae, dict(sample_length=hps.get('sample_length', 0), sample_length_in_seconds=hps.get('sample_length_in_seconds', 0))), device) File "c:\users\james\jukebox\jukebox\make_models.py", line 95, in make_vqvae restore_model(hps, vqvae, hps.restore_vqvae) File "c:\users\james\jukebox\jukebox\make_models.py", line 55, in restore_model checkpoint = load_checkpoint(checkpoint_path) File "c:\users\james\jukebox\jukebox\make_models.py", line 37, in load_checkpoint checkpoint = t.load(restore, map_location=t.device('cpu')) File "C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\torch\serialization.py", line 386, in load return _load(f, map_location, pickle_module, **pickle_load_args) File "C:\Users\james\Anaconda3\envs\jukebox\lib\site-packages\torch\serialization.py", line 563, in _load magic_number = pickle_module.load(f, **pickle_load_args) EOFError: Ran out of input

I have deleted the cache and still get it. I have 30gb of space. Any ideas?

from jukebox.

I am on Windows, my vqvae.pth.tar is 0KB. Happens with both 5b and 1b model.

from jukebox.

I have the same issue. I've tried deleting the file found at C:\Users\rfnoi\.cache\jukebox-assets\models\5b\vqvae.pth.tar, but it re-creates it and crashes with the same error EOFError: Ran out of input.

E: I have also attempted this with the 1b model. Nothing new is found in C:\Users\rfnoi\.cache\jukebox-assets\models\. Should I expect a /1b/ directory?

from jukebox.

I have the same issue. I've tried deleting the file found at C:\Users\rfnoi\.cache\jukebox-assets\models\5b\vqvae.pth.tar, but it re-creates it and crashes with the same error EOFError: Ran out of input.

E: I have also attempted this with the 1b model. Nothing new is found in C:\Users\rfnoi\.cache\jukebox-assets\models\. Should I expect a /1b/ directory?

did you get your answer? I'm having that problem too

from jukebox.

same issue here, WIndows 10

from jukebox.

same issue here, WIndows 10

No issue here, Windows 8.1

from jukebox.