The refill from awasthiabhijeet

multiprocess error when computing tree edit distance

When I ran the following command:

python sql-to-text/refill/compute_neighbours.py \
	--source_json=data/spider/train_spider.json \
	--target_json=data/spider/train_spider.json	 \
	--output_json=data/sql-to-text/refill/jsons/train_train_nbrs.json

I got the error because smbop is not installed via requirements.txt.

Traceback (most recent call last):
  File "sql-to-text/refill/compute_neighbours.py", line 14, in <module>
    import smbop.utils.ra_preproc as ra_preproc
ModuleNotFoundError: No module named 'smbop'

This can be fixed by adding the following two liens before import smbop.utils.ra_preproc as ra_preproc:

import sys
sys.path.append("./")

Then as is the title, the scripts show massive error messages by repeating the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/envs/refil/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/envs/refil/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/envs/refil/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/envs/refil/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/envs/refil/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/envs/refil/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/envs/refil/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/refill/sql-to-text/refill/compute_neighbours.py", line 104, in <module>
    source_data = parse_data(source_data)
  File "/refill/sql-to-text/refill/compute_neighbours.py", line 82, in parse_data
    result = Parallel(n_jobs=-1)(delayed(parse_item)(item) for item in tqdm(
  File "/envs/refil/lib/python3.8/site-packages/joblib/parallel.py", line 1942, in __call__
    next(output)
  File "/envs/refil/lib/python3.8/site-packages/joblib/parallel.py", line 1580, in _get_outputs
    self._start(iterator, pre_dispatch)
  File "/envs/refil/lib/python3.8/site-packages/joblib/parallel.py", line 1563, in _start
    if self.dispatch_one_batch(iterator):
  File "/envs/refil/lib/python3.8/site-packages/joblib/parallel.py", line 1454, in dispatch_one_batch
    self._dispatch(tasks)
  File "/envs/refil/lib/python3.8/site-packages/joblib/parallel.py", line 1376, in _dispatch
    job = self._backend.apply_async(batch, callback=batch_tracker)
  File "/envs/refil/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 600, in apply_async
    future = self._workers.submit(func)
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/reusable_executor.py", line 225, in submit
    return super().submit(fn, *args, **kwargs)
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 1248, in submit
    self._ensure_executor_running()
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 1220, in _ensure_executor_running
    self._adjust_process_count()
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 1209, in _adjust_process_count
    p.start()
  File "/envs/refil/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/backend/process.py", line 45, in _Popen
    return Popen(process_obj)
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 48, in __init__
    self._launch(process_obj)
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/backend/popen_loky_posix.py", line 99, in _launch
    prep_data = spawn.get_preparation_data(
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/backend/spawn.py", line 61, in get_preparation_data
    _check_not_importing_main()
  File "/envs/refil/lib/python3.8/site-packages/joblib/externals/loky/backend/spawn.py", line 39, in _check_not_importing_main
    raise RuntimeError(
RuntimeError: An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:

    if __name__ == '__main__':
        freeze_support()
        ...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.

I think this is because this repo assumes Linux OS which uses fork backend of multiprocess, https://docs.python.org/3/library/multiprocessing.html; I'm not using Linux, so it uses another backend, spawn. So adding multiprocessing.set_start_method("fork") before calling multiprocessing.Pool might resolve the error but I'm not sure whether or not this is correct approach.

Thanks.

awasthiabhijeet / refill Goto Github PK

refill's People

Contributors

Stargazers

Watchers

Forkers

refill's Issues

multiprocess error when computing tree edit distance

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent