(pytorch-c) yuandong@devfair0306:~/dl_theory2$ python two_layer_new2.py -s
[2019-08-12 11:28:50,983][hydra_plugins.fairtask.fairtask_launcher][INFO] - Sweep output dir : /checkpoint/yuandong/outputs/2019-08-12_11-28-50
[2019-08-12 11:28:51,039][hydra_plugins.fairtask.fairtask_launcher][INFO] - Launching 1 jobs to slurm queue
[2019-08-12 11:28:51,039][hydra_plugins.fairtask.fairtask_launcher][INFO] - #0 :
/private/home/yuandong/.local/lib/python3.7/site-packages/distributed/deploy/local.py:112: UserWarning: The ip keyword has been moved to host
warnings.warn("The ip keyword has been moved to host")
Dask dashboard for "slurm" at http://localhost:8006.
Traceback (most recent call last):
File "two_layer_new2.py", line 222, in <module>
main()
File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/hydra.py", line 165, in decorated_main
run_hydra(task_function, config_path, strict)
File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/hydra.py", line 145, in run_hydra
hydra.sweep(overrides=args.overrides)
File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/hydra.py", line 88, in sweep
return sweeper.sweep(arguments=cfg.hydra.overrides.task)
File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/sweeper/step_sweeper.py", line 61, in sweep
results = self.launcher.launch(batch)
File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra_plugins/fairtask/fairtask_launcher.py", line 96, in launch
return loop.run_until_complete(self.run_sweep(queue, job_overrides))
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
return future.result()
File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra_plugins/fairtask/fairtask_launcher.py", line 78, in run_sweep
return await gatherl(runs)
File "/private/home/yuandong/.local/lib/python3.7/site-packages/fairtask/util.py", line 26, in gatherl
return await multi(iters)
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
result_list.append(f.result())
File "/private/home/yuandong/.local/lib/python3.7/site-packages/fairtask/queue.py", line 418, in __call__
return await self.submit_fn(*args, **kwargs)
File "/private/home/yuandong/.local/lib/python3.7/site-packages/fairtask/queue.py", line 248, in submit
pure=pure,
File "/private/home/yuandong/.local/lib/python3.7/site-packages/distributed/client.py", line 1412, in submit
key = funcname(func) + "-" + tokenize(func, kwargs, *args)
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/dask/base.py", line 661, in tokenize
return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest()
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/dask/utils.py", line 486, in __call__
return meth(arg, *args, **kwargs)
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/dask/base.py", line 705, in normalize_object
method = getattr(o, "__dask_tokenize__", None)
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 72, in __getattr__
return self.get(key=key, default_value=None)
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 83, in get
return self._resolve_with_default(key=key, value=self.get_node(key), default_value=default_value)
File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 88, in get_node
raise KeyError("Accessing unknown key in a struct : {}".format(self.get_full_key(key)))
KeyError: 'Accessing unknown key in a struct : __dask_tokenize__'