karrlab / wc_env_manager Goto Github PK
View Code? Open in Web Editor NEWTools for managing computing environments for WC modeling
License: MIT License
Tools for managing computing environments for WC modeling
License: MIT License
Trace below. The first 2 lines starting config
are produced by my debugging code.
$ wc-env-manager pull
config['repo_unsquashed'] karrlab/wc_env_dependencies_unsquashed
config['tags'] ['latest', '0.0.51']
Traceback (most recent call last):
File "/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 229, in _raise_for_status
response.raise_for_status()
File "/anaconda3/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/karrlab/wc_env_dependencies_unsquashed:0.0.51/json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/anaconda3/bin/wc-env-manager", line 10, in <module>
sys.exit(main())
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 277, in main
app.run()
File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
return func()
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 236, in pull
mgr.pull_image(config['repo_unsquashed'], config['tags'])
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 491, in pull_image
image = self._docker_client.images.pull(image_repo, tag=tag)
File "/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 415, in pull
repository, tag, '@' if tag.startswith('sha256:') else ':'
File "/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 295, in get
return self.prepare_model(self.client.api.inspect_image(name))
File "/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/anaconda3/lib/python3.6/site-packages/docker/api/image.py", line 245, in inspect_image
self._get(self._url("/images/{0}/json", image)), True
File "/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 235, in _result
self._raise_for_status(response)
File "/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/anaconda3/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("no such image: karrlab/wc_env_dependencies_unsquashed:0.0.51: No such image: karrlab/wc_env_dependencies_unsquashed:0.0.51")
See 'single element list' and the docs. This would apply to all uses of configobj
.
Optimize the images using these and/or other strategies. In no particular order:
alpine
build-essential
cmake
wget
rm -r /var/lib/apt/lists/*
): not necessary if using multistage build: DONEnano
apt-get install -y --no-install-recommends ...
): DONE.dockerignore
(although this is unnecessary because the context is already minimal): IGNOREdocker build --squash
, docker-squash, or multi-stage builds (i.e. build dependencies in one image and then copy the result to a second image). Squashing is simpler, but multi-stage building offers the ability to share intermediate layers.: DONEI believe that the wc_env_manager Requirements should say Pip >= 19.0
wc-env-manager container build
:
$ wc-env-manager container build
/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py:443: UserWarning: Third party configuration files will not be copied to the image because no index of third party configuration files was found at "/Users/arthur_at_sinai/.wc/third_party/paths.yml". This path should be a YAML-encoded dictionary which maps the names of files within "{}" to locations in the image where each file should be copied. See "https://github.com/KarrLab/karr_lab_build_config/tree/master/third_party" for an example.
UserWarning)
Creating directory /root/.config/bioservices
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.ddl.postgresql] Detected sequence named 'cell_line_id_seq' as owned by integer column 'cell_line(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'method_id_seq' as owned by integer column 'method(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'characteristic_id_seq' as owned by integer column 'characteristic(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'experiment_design_id_seq' as owned by integer column 'experiment_design(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'experiment_id_seq' as owned by integer column 'experiment(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO [alembic.env] No changes in schema detected.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Traceback (most recent call last):
File "/usr/local/bin/datanator", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
app.run()
File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
return func()
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
verbose=True)
File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
verbose=verbose)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
exit_on_error=restore_backup_exit_on_error)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
manager.download(system_path=path, sym_links=True)
File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
quilt.login_with_token(self.token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
auth = _update_auth(team, refresh_token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401
Traceback (most recent call last):
File "/anaconda3/bin/wc-env-manager", line 10, in <module>
sys.exit(main())
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 274, in main
app.run()
File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
return func()
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 175, in build
mgr.setup_container()
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 668, in setup_container
self.run_process_in_container(['bash', '-c', cmd], container_user=WcEnvUser.root)
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 810, in run_process_in_container
result.output.decode('utf-8')))
wc_env_manager.core.WcEnvManagerError: Command not successfully executed in Docker container:
command: ['bash', '-c', '\n if [ -x "$(command -v datanator)" ]; then\n datanator db create\n datanator db migrate\n datanator db restore --restore-schema --do-not-exit-on-error\n fi\n ']
working directory: /root
environment:
exit code: 1
output: Creating directory /root/.config/bioservices
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.ddl.postgresql] Detected sequence named 'cell_line_id_seq' as owned by integer column 'cell_line(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'method_id_seq' as owned by integer column 'method(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'characteristic_id_seq' as owned by integer column 'characteristic(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'experiment_design_id_seq' as owned by integer column 'experiment_design(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'experiment_id_seq' as owned by integer column 'experiment(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO [alembic.env] No changes in schema detected.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Traceback (most recent call last):
File "/usr/local/bin/datanator", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
app.run()
File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
return func()
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
verbose=True)
File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
verbose=verbose)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
exit_on_error=restore_backup_exit_on_error)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
manager.download(system_path=path, sym_links=True)
File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
quilt.login_with_token(self.token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
auth = _update_auth(team, refresh_token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401
diff
between default and custom wc_env_manager.cfg
:
$ diff wc_env_manager.cfg ~/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/config/core.default.cfg
32c32
< postgresql_client_install = False
---
> postgresql_client_install = True
109,110c109
< # Comment out ones that are not needed
< # git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
---
> git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
112,113c111,112
< # git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
< # git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
---
> git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
> git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
117,120c116,119
< # git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]
< # git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
< # git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
< # git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
---
> git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]
> git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
> git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
> git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
122c121
< # git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
---
> git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
131,138c130,136
< # Not needed
< # name = wc
< # [[[containers]]]
< # [[[[postgres_service]]]]
< # image = circleci/postgres:11.2-alpine-ram
< # shm_size = 1G
< # [[[[[environment]]]]]
< # POSTGRES_USER = postgres
---
> name = wc
> [[[containers]]]
> [[[[postgres_service]]]]
> image = circleci/postgres:11.2-alpine-ram
> shm_size = 1G
> [[[[[environment]]]]]
> POSTGRES_USER = postgres
143,154c141,147
< # Do not setup datanator
< # setup_script = '''
< # if [ -x "$$(command -v datanator)" ]; then
< # datanator db create
< # datanator db migrate
< # datanator db restore --restore-schema --do-not-exit-on-error
< # fi
< # '''
< [[[paths_to_mount]]]
< [[[[${HOME}/gitOnMyLaptopLocal/wc_dev_repos]]]]
< bind = /root/host/Documents
[arthur_at_sinai/.wc]$ diff wc_env_manager.cfg ~/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/config/core.default.cfg|m
32c32
< postgresql_client_install = False
---
> postgresql_client_install = True
109,110c109
< # Comment out WC modeling components not needed
< # git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
---
> git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
112,113c111,112
< # git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
< # git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
---
> git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
> git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
117,120c116,119
< # git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]
< # git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
< # git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
< # git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
---
> git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]
> git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
> git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
> git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
122c121
< # git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
---
> git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
131,138c130,136
< # Not needed
< # name = wc
< # [[[containers]]]
< # [[[[postgres_service]]]]
< # image = circleci/postgres:11.2-alpine-ram
< # shm_size = 1G
< # [[[[[environment]]]]]
< # POSTGRES_USER = postgres
---
> name = wc
> [[[containers]]]
> [[[[postgres_service]]]]
> image = circleci/postgres:11.2-alpine-ram
> shm_size = 1G
> [[[[[environment]]]]]
> POSTGRES_USER = postgres
143,154c141,147
< # Do not setup datanator
< # setup_script = '''
< # if [ -x "$$(command -v datanator)" ]; then
< # datanator db create
< # datanator db migrate
< # datanator db restore --restore-schema --do-not-exit-on-error
< # fi
< # '''
< [[[paths_to_mount]]]
< [[[[${HOME}/gitOnMyLaptopLocal/wc_dev_repos]]]]
< bind = /root/host/Documents
< mode = rw
---
> setup_script = '''
> if [ -x "$$(command -v datanator)" ]; then
> datanator db create
> datanator db migrate
> datanator db restore --restore-schema --do-not-exit-on-error
> fi
> '''
But I can kill the postgres_service
container, and use wc_env
, so I can keep working.
wc-env-manager container build
fails with the default config/core.default.cfg
:
$ docker image ls -a; docker container ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
circleci/postgres 11.2-alpine-ram 9bba4a631f6d 3 months ago 70.8MB
karrlab/wc_env 0.0.51 3d35a1beb777 4 months ago 5.81GB
karrlab/wc_env latest 3d35a1beb777 4 months ago 5.81GB
karrlab/wc_env_dependencies latest 953edb5668b3 4 months ago 5.72GB
karrlab/wc_env_dependencies_unsquashed latest d4322c96ccd6 4 months ago 6.52GB
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ wc-env-manager container build
/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py:443: UserWarning: Third party configuration files will not be copied to the image because no index of third party configuration files was found at "/Users/arthur_at_sinai/.wc/third_party/paths.yml". This path should be a YAML-encoded dictionary which maps the names of files within "{}" to locations in the image where each file should be copied. See "https://github.com/KarrLab/karr_lab_build_config/tree/master/third_party" for an example.
UserWarning)
Creating directory /root/.config/bioservices
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.ddl.postgresql] Detected sequence named 'synonym_id_seq' as owned by integer column 'synonym(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'conditions_id_seq' as owned by integer column 'conditions(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO [alembic.env] No changes in schema detected.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Traceback (most recent call last):
File "/usr/local/bin/datanator", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
app.run()
File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
return func()
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
verbose=True)
File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
verbose=verbose)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
exit_on_error=restore_backup_exit_on_error)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
manager.download(system_path=path, sym_links=True)
File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
quilt.login_with_token(self.token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
auth = _update_auth(team, refresh_token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401
Traceback (most recent call last):
File "/anaconda3/bin/wc-env-manager", line 10, in <module>
sys.exit(main())
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 274, in main
app.run()
File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
return func()
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 175, in build
mgr.setup_container()
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 668, in setup_container
self.run_process_in_container(['bash', '-c', cmd], container_user=WcEnvUser.root)
File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 810, in run_process_in_container
result.output.decode('utf-8')))
wc_env_manager.core.WcEnvManagerError: Command not successfully executed in Docker container:
command: ['bash', '-c', '\n if [ -x "$(command -v datanator)" ]; then\n datanator db create\n datanator db migrate\n datanator db restore --restore-schema --do-not-exit-on-error\n fi\n ']
working directory: /root
environment:
exit code: 1
output: Creating directory /root/.config/bioservices
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.ddl.postgresql] Detected sequence named 'synonym_id_seq' as owned by integer column 'synonym(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'conditions_id_seq' as owned by integer column 'conditions(id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO [alembic.env] No changes in schema detected.
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
Traceback (most recent call last):
File "/usr/local/bin/datanator", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
app.run()
File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
return func()
File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
verbose=True)
File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
verbose=verbose)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
exit_on_error=restore_backup_exit_on_error)
File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
manager.download(system_path=path, sym_links=True)
File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
quilt.login_with_token(self.token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
auth = _update_auth(team, refresh_token)
File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401
status
command that lists wc images, containers, and config files--debug
optionAutomatically remove consistent duplicates.
For example, this
pkg == 0.2
pkg >= 0.1
pkg <= 0.4
should be simplified to
pkg == 0.2
Requires figuring out how to recursively test docker in CircleCI
We want mounted repos to access each other before installed repos.
This code makes the bash export command that needs to be executed in the container
import os
# todo: distinguish mounted repos from other mounted directories
def paths_to_mounted_repos(mounted_repo_dirs):
paths = []
for mounted_repo_dir in mounted_repo_dirs:
for entry in os.scandir(mounted_repo_dir):
# ignore . dirs, and files
if not entry.name.startswith('.') and entry.is_dir():
paths.append(os.path.join(mounted_repo_dir, entry.name))
return paths
def python_path_cmd(dirs, prepend=False):
""" make bash export command to add dirs to python path"""
joined_dirs = ':'.join(dirs)
if prepend:
bash_cmd = 'export PYTHONPATH={}:$PYTHONPATH'.format(joined_dirs)
else:
bash_cmd = 'export PYTHONPATH=$PYTHONPATH:{}'.format(joined_dirs)
return bash_cmd
MOUNTED_REPO_DIRS = config['container']['paths_to_mount']
python_path_cmd(paths_to_mounted_repos(MOUNTED_REPO_DIRS), prepend=True)
But I don't know how to automatically have bash execute this export command when bash starts in the container. Perhaps docker's ENTRYPOINT command would help. Or perhaps the export could be added to one of bash's initialization files. Our docker container has .bashrc and .profile in root's home dir.
what happens:
what's expected:
'wc_env_manager container build' has stopped working on my Mac. I don't know why, as I don't believe that I've changed any of the code it uses.
This is a critical problem because I'm relying on docker containers for my wc_sim development. I will try reinstalling wc_utils and wc_env_manager now.
Here's a trace:
Script started on Sat Oct 20 08:19:40 2018
bash-3.2$ wc_env_manager container build
Traceback (most recent call last):
File "/anaconda3/bin/wc_env_manager", line 11, in <module>
sys.exit(main())
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 230, in main
app.run()
File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
return func()
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 140, in build
mgr = wc_env_manager.core.WcEnvManager({'verbose': VERBOSE})
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/core.py", line 81, in __init__
'wc_env_manager': config or {}})['wc_env_manager']
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/config/core.py", line 39, in get_config
return wc_utils.config.ConfigManager(paths).get_config(extra=extra, context=context)
File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 127, in get_config
raise ExtraValuesError(config)
File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 284, in __init__
the_section = config[section]
File "/anaconda3/lib/python3.6/site-packages/configobj.py", line 554, in __getitem__
val = dict.__getitem__(self, key)
KeyError: 'container'
bash-3.2$ wc_env_manager -h
usage: wc_env_manager [-h] [-d] [-q] [-v]
{container,image,base-image,build,pull,push,remove} ...
Whole-cell modeling environment manager
optional arguments:
-h, --help show this help message and exit
-d, --debug full application debug mode
-q, --quiet suppress all console output
-v, --version show program's version number and exit
sub-commands:
{container,image,base-image,build,pull,push,remove}
container container controller
image image controller
base-image base-image controller
build Build base image, image, and container
pull Pull base image and image
push Push base image and image
remove Remove base image, image, and containers
bash-3.2$ wc_env_manager container
usage: wc_env_manager container [-h] {build,remove} ...
Build and remove containers of `wc_env`
optional arguments:
-h, --help show this help message and exit
sub-commands:
{build,remove}
build Build container
remove Remove container
bash-3.2$ pip show wc_utils
Name: wc-utils
Version: 0.0.1a5
Summary: Utilities for whole-cell modeling
Home-page: https://github.com/KarrLab/wc_utils
Author: Karr Lab
Author-email: [email protected]
License: MIT
Location: /anaconda3/lib/python3.6/site-packages
Requires: mendeleev, requests, log, diskcache, enum34, six, numpy, qualname, gitpython, setuptools, pyexcel, attrdict, pyexcel-io, openpyxl, pint, pyyaml, configobj, abduct
Required-by: wc-env-manager
bash-3.2$ pip show wc_env_manager
Name: wc-env-manager
Version: 0.0.1
Summary: Tools for managing computing environments for whole-cell modeling
Home-page: https://github.com/KarrLab/wc_env_manager
Author: Karr Lab
Author-email: [email protected]
License: MIT
Location: /Users/arthur_at_sinai/.local/lib/python3.6/site-packages
Requires: pkg-utils, gitpython, cement, configobj, wc-utils, python-dateutil, requirements-parser, docker, pyyaml, jinja2
Required-by:
bash-3.2$ pip show cement
Name: cement
Version: 3.0.0
Summary: CLI Framework for Python
Home-page: http://builtoncement.org
Author: Data Folk Labs, LLC
Author-email: [email protected]
License: BSD
Location: /anaconda3/lib/python3.6/site-packages
Requires:
Required-by: wc-env-manager
bash-3.2$ wc_env_manager container remove
Traceback (most recent call last):
File "/anaconda3/bin/wc_env_manager", line 11, in <module>
sys.exit(main())
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 230, in main
app.run()
File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
return_val = self.controller._dispatch()
File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
return func()
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 147, in remove
mgr = wc_env_manager.core.WcEnvManager({'verbose': VERBOSE})
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/core.py", line 81, in __init__
'wc_env_manager': config or {}})['wc_env_manager']
File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/config/core.py", line 39, in get_config
return wc_utils.config.ConfigManager(paths).get_config(extra=extra, context=context)
File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 127, in get_config
raise ExtraValuesError(config)
File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 284, in __init__
the_section = config[section]
File "/anaconda3/lib/python3.6/site-packages/configobj.py", line 554, in __getitem__
val = dict.__getitem__(self, key)
KeyError: 'container'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.