Code Monkey home page Code Monkey logo

wc_env_manager's Introduction

PyPI package Documentation Test results Test coverage Code analysis License Analytics

wc_env_manager: Tools for managing computing environments for whole-cell modeling

wc_env_manager helps modelers and software developers setup customizable computing environments for developing, testing, and running whole-cell (WC) models and WC modeling software. This eliminates the need for each modeler and software developer to install and configure the numerous dependencies required for WC modeling. This helps modelers and software developers focus on developing WC models and software tools, rather than on installing and maintaining complicated dependencies.

In addition, wc_env_manager facilitates collaboration by helping WC modelers and software developers share a common computing environment. Furthermore, wc_env_manager helps software developers anticipate and debug issues in deployment by enabling developers to replicate the same environment used to test and deploy WC models and tools in systems such as Amazon EC2, CircleCI, and Heroku.

wc_env_manager uses Docker to setup customizable computing environments that contains all of the software packages needed to run WC models and WC modeling software. This includes

  • Required non-Python packages
  • Required Python packages from PyPI and other sources
  • WC models and WC modeling tools
  • Optionally, local packages on the user's machine such as clones of these WC models and WC modeling tools

wc_env_manager supports two modes:

  • Development: wc_env_manager can run WC models and WC modeling software that is located on the user's machine. This is useful for testing WC models and WC modeling software before committing it to GitHub.
  • Deployment: wc_env_manager can run WC models and WC modeling software from external sources such as GitHub.

Installation and usage

Please see the documentation.

Documentation

Please see the API documentation.

License

The package is released under the MIT license.

Development team

This package was developed by the Karr Lab at the Icahn School of Medicine at Mount Sinai in New York, USA.

Questions and comments

Please contact the Karr Lab with any questions or comments.

wc_env_manager's People

Contributors

artgoldberg avatar jonrkarr avatar yinhoon avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wc_env_manager's Issues

Optimize base image

Optimize the images using these and/or other strategies. In no particular order:

  • Start image from alpine
  • Remove packages that are only needed to build images such as
    • build-essential
    • cmake
    • wget
  • Remove temporary files needed for installing dependencies with apt (i.e. rm -r /var/lib/apt/lists/*): not necessary if using multistage build: DONE
  • Remove debugging tools such as nano
  • Don't install recommended packages (i.e. apt-get install -y --no-install-recommends ...): DONE
  • Minimize layers: obviated by squashing: IGNORE
    • Combine consecutive ENV directives into single layer: DONE
  • Utilize .dockerignore (although this is unnecessary because the context is already minimal): IGNORE
  • Build separate images for each wc package that only contain the necessary dependencies: SKIP
  • Use Dockerfile lint tool: https://www.fromlatest.io: DONE
  • Use docker build --squash, docker-squash, or multi-stage builds (i.e. build dependencies in one image and then copy the result to a second image). Squashing is simpler, but multi-stage building offers the ability to share intermediate layers.: DONE

Test in CircleCI

Requires figuring out how to recursively test docker in CircleCI

`wc-env-manager container build` fails with a customized config designed to not use datanator or a DBMS

wc-env-manager container build:

$ wc-env-manager container build
/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py:443: UserWarning: Third party configuration files will not be copied to the image because no index of third party configuration files was found at "/Users/arthur_at_sinai/.wc/third_party/paths.yml".  This path should be a YAML-encoded dictionary which maps the names of files within "{}" to locations in the image where each file should be copied. See "https://github.com/KarrLab/karr_lab_build_config/tree/master/third_party" for an example.
  UserWarning)

Creating directory /root/.config/bioservices 
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.ddl.postgresql] Detected sequence named 'cell_line_id_seq' as owned by integer column 'cell_line(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'method_id_seq' as owned by integer column 'method(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'characteristic_id_seq' as owned by integer column 'characteristic(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'experiment_design_id_seq' as owned by integer column 'experiment_design(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'experiment_id_seq' as owned by integer column 'experiment(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO  [alembic.env] No changes in schema detected.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/datanator", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
    app.run()
  File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
    return func()
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
    verbose=True)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
    verbose=verbose)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
    exit_on_error=restore_backup_exit_on_error)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
    manager.download(system_path=path, sym_links=True)
  File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
    quilt.login_with_token(self.token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
    auth = _update_auth(team, refresh_token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
    raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401
Traceback (most recent call last):
  File "/anaconda3/bin/wc-env-manager", line 10, in <module>
    sys.exit(main())
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 274, in main
    app.run()
  File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
    return func()
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 175, in build
    mgr.setup_container()
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 668, in setup_container
    self.run_process_in_container(['bash', '-c', cmd], container_user=WcEnvUser.root)
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 810, in run_process_in_container
    result.output.decode('utf-8')))
wc_env_manager.core.WcEnvManagerError: Command not successfully executed in Docker container:
  command: ['bash', '-c', '\n            if [ -x "$(command -v datanator)" ]; then\n                datanator db create\n                datanator db migrate\n                datanator db restore --restore-schema --do-not-exit-on-error\n            fi\n            ']
  working directory: /root
  environment:

  exit code: 1
  output: Creating directory /root/.config/bioservices 
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.ddl.postgresql] Detected sequence named 'cell_line_id_seq' as owned by integer column 'cell_line(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'method_id_seq' as owned by integer column 'method(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'characteristic_id_seq' as owned by integer column 'characteristic(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'experiment_design_id_seq' as owned by integer column 'experiment_design(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'experiment_id_seq' as owned by integer column 'experiment(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO  [alembic.env] No changes in schema detected.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/datanator", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
    app.run()
  File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
    return func()
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
    verbose=True)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
    verbose=verbose)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
    exit_on_error=restore_backup_exit_on_error)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
    manager.download(system_path=path, sym_links=True)
  File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
    quilt.login_with_token(self.token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
    auth = _update_auth(team, refresh_token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
    raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401

diff between default and custom wc_env_manager.cfg:

$ diff wc_env_manager.cfg ~/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/config/core.default.cfg
32c32
<             postgresql_client_install = False
---
>             postgresql_client_install = True
109,110c109
<             # Comment out ones that are not needed
<             # git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
---
>             git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
112,113c111,112
<             # git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
<             # git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
---
>             git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
>             git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
117,120c116,119
<             # git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]           
<             # git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
<             # git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
<             # git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
---
>             git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]           
>             git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
>             git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
>             git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
122c121
<             # git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
---
>             git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
131,138c130,136
<         # Not needed
<         #    name = wc
<         #    [[[containers]]]
<         #        [[[[postgres_service]]]]
<         #            image = circleci/postgres:11.2-alpine-ram
<         #            shm_size = 1G
<         #            [[[[[environment]]]]]
<         #                POSTGRES_USER = postgres
---
>         name = wc
>         [[[containers]]]
>             [[[[postgres_service]]]]
>                 image = circleci/postgres:11.2-alpine-ram
>                 shm_size = 1G
>                 [[[[[environment]]]]]
>                     POSTGRES_USER = postgres
143,154c141,147
<         # Do not setup datanator
<         # setup_script = '''
<         #   if [ -x "$$(command -v datanator)" ]; then
<         #       datanator db create
<         #       datanator db migrate
<         #       datanator db restore --restore-schema --do-not-exit-on-error
<         #   fi
<         #   '''
<         [[[paths_to_mount]]]
<             [[[[${HOME}/gitOnMyLaptopLocal/wc_dev_repos]]]]
<                 bind = /root/host/Documents
[arthur_at_sinai/.wc]$ diff wc_env_manager.cfg ~/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/config/core.default.cfg|m
32c32
<             postgresql_client_install = False
---
>             postgresql_client_install = True
109,110c109
<             # Comment out WC modeling components not needed
<             # git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
---
>             git+https://github.com/KarrLab/datanator.git#egg=datanator[all]
112,113c111,112
<             # git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
<             # git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
---
>             git+https://github.com/KarrLab/wc_lang.git#egg=wc_lang[all]
>             git+https://github.com/KarrLab/wc_rules.git#egg=wc_rules[all]
117,120c116,119
<             # git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]           
<             # git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
<             # git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
<             # git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
---
>             git+https://github.com/KarrLab/wc_sim.git#egg=wc_sim[all]           
>             git+https://github.com/KarrLab/wc_analysis.git#egg=wc_analysis[all]
>             git+https://github.com/KarrLab/wc_test.git#egg=wc_test[all]
>             git+https://github.com/KarrLab/rand_wc_model_gen.git#egg=rand_wc_model_gen[all]
122c121
<             # git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
---
>             git+https://github.com/KarrLab/mycoplasma_pneumoniae.git#egg=mycoplasma_pneumoniae[all]
131,138c130,136
<         # Not needed
<         #    name = wc
<         #    [[[containers]]]
<         #        [[[[postgres_service]]]]
<         #            image = circleci/postgres:11.2-alpine-ram
<         #            shm_size = 1G
<         #            [[[[[environment]]]]]
<         #                POSTGRES_USER = postgres
---
>         name = wc
>         [[[containers]]]
>             [[[[postgres_service]]]]
>                 image = circleci/postgres:11.2-alpine-ram
>                 shm_size = 1G
>                 [[[[[environment]]]]]
>                     POSTGRES_USER = postgres
143,154c141,147
<         # Do not setup datanator
<         # setup_script = '''
<         #   if [ -x "$$(command -v datanator)" ]; then
<         #       datanator db create
<         #       datanator db migrate
<         #       datanator db restore --restore-schema --do-not-exit-on-error
<         #   fi
<         #   '''
<         [[[paths_to_mount]]]
<             [[[[${HOME}/gitOnMyLaptopLocal/wc_dev_repos]]]]
<                 bind = /root/host/Documents
<                 mode = rw
---
>         setup_script = '''
>             if [ -x "$$(command -v datanator)" ]; then
>                 datanator db create
>                 datanator db migrate
>                 datanator db restore --restore-schema --do-not-exit-on-error
>             fi
>             '''

But I can kill the postgres_service container, and use wc_env, so I can keep working.

Two enhancements

  • status command that lists wc images, containers, and config files
  • output progress, final status, config file(s) used and their interpretation, especially with the --debug option

configure PYTHONPATH to access mounted repos

We want mounted repos to access each other before installed repos.
This code makes the bash export command that needs to be executed in the container

import os
# todo: distinguish mounted repos from other mounted directories
def paths_to_mounted_repos(mounted_repo_dirs):
    paths = []
    for mounted_repo_dir in mounted_repo_dirs:
        for entry in os.scandir(mounted_repo_dir):
            # ignore . dirs, and files
            if not entry.name.startswith('.') and entry.is_dir():
                paths.append(os.path.join(mounted_repo_dir, entry.name))
    return paths

def python_path_cmd(dirs, prepend=False):
    """ make bash export command to add dirs to python path"""
    joined_dirs = ':'.join(dirs)
    if prepend:
        bash_cmd = 'export PYTHONPATH={}:$PYTHONPATH'.format(joined_dirs)
    else:
        bash_cmd = 'export PYTHONPATH=$PYTHONPATH:{}'.format(joined_dirs)
    return bash_cmd

MOUNTED_REPO_DIRS = config['container']['paths_to_mount']
python_path_cmd(paths_to_mounted_repos(MOUNTED_REPO_DIRS), prepend=True)

But I don't know how to automatically have bash execute this export command when bash starts in the container. Perhaps docker's ENTRYPOINT command would help. Or perhaps the export could be added to one of bash's initialization files. Our docker container has .bashrc and .profile in root's home dir.

`wc-env-manager container build` fails with the default config

wc-env-manager container build fails with the default config/core.default.cfg:

$ docker image ls -a; docker container ls -a
REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
circleci/postgres                        11.2-alpine-ram     9bba4a631f6d        3 months ago        70.8MB
karrlab/wc_env                           0.0.51              3d35a1beb777        4 months ago        5.81GB
karrlab/wc_env                           latest              3d35a1beb777        4 months ago        5.81GB
karrlab/wc_env_dependencies              latest              953edb5668b3        4 months ago        5.72GB
karrlab/wc_env_dependencies_unsquashed   latest              d4322c96ccd6        4 months ago        6.52GB
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ wc-env-manager container build
/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py:443: UserWarning: Third party configuration files will not be copied to the image because no index of third party configuration files was found at "/Users/arthur_at_sinai/.wc/third_party/paths.yml".  This path should be a YAML-encoded dictionary which maps the names of files within "{}" to locations in the image where each file should be copied. See "https://github.com/KarrLab/karr_lab_build_config/tree/master/third_party" for an example.
  UserWarning)

Creating directory /root/.config/bioservices 
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.ddl.postgresql] Detected sequence named 'synonym_id_seq' as owned by integer column 'synonym(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'conditions_id_seq' as owned by integer column 'conditions(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO  [alembic.env] No changes in schema detected.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/datanator", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
    app.run()
  File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
    return func()
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
    verbose=True)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
    verbose=verbose)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
    exit_on_error=restore_backup_exit_on_error)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
    manager.download(system_path=path, sym_links=True)
  File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
    quilt.login_with_token(self.token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
    auth = _update_auth(team, refresh_token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
    raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401
Traceback (most recent call last):
  File "/anaconda3/bin/wc-env-manager", line 10, in <module>
    sys.exit(main())
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 274, in main
    app.run()
  File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
    return func()
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 175, in build
    mgr.setup_container()
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 668, in setup_container
    self.run_process_in_container(['bash', '-c', cmd], container_user=WcEnvUser.root)
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 810, in run_process_in_container
    result.output.decode('utf-8')))
wc_env_manager.core.WcEnvManagerError: Command not successfully executed in Docker container:
  command: ['bash', '-c', '\n            if [ -x "$(command -v datanator)" ]; then\n                datanator db create\n                datanator db migrate\n                datanator db restore --restore-schema --do-not-exit-on-error\n            fi\n            ']
  working directory: /root
  environment:

  exit code: 1
  output: Creating directory /root/.config/bioservices 
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Creating directory /root/migrations ... done
Creating directory /root/migrations/versions ... done
Generating /root/migrations/script.py.mako ... done
Generating /root/migrations/README ... done
Generating /root/migrations/alembic.ini ... done
Generating /root/migrations/env.py ... done
Please edit configuration/connection/logging settings in '/root/migrations/alembic.ini' before proceeding.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.ddl.postgresql] Detected sequence named 'synonym_id_seq' as owned by integer column 'synonym(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'dna_binding_data_position_id_seq' as owned by integer column 'dna_binding_data(position_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'cell_compartment_id_seq' as owned by integer column 'cell_compartment(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'reaction_reaction_id_seq' as owned by integer column 'reaction(reaction_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'conditions_id_seq' as owned by integer column 'conditions(id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'parameter_parameter_id_seq' as owned by integer column 'parameter(parameter_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'abundance_data_abundance_id_seq' as owned by integer column 'abundance_data(abundance_id)', assuming SERIAL and omitting
INFO  [alembic.ddl.postgresql] Detected sequence named 'data_format__id_seq' as owned by integer column 'data_format(_id)', assuming SERIAL and omitting
INFO  [alembic.env] No changes in schema detected.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "/usr/local/bin/datanator", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 840, in main
    app.run()
  File "/usr/local/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/usr/local/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 808, in _dispatch
    return func()
  File "/usr/local/lib/python3.6/site-packages/datanator/__main__.py", line 773, in _default
    verbose=True)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/common_schema.py", line 64, in __init__
    verbose=verbose)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 110, in __init__
    exit_on_error=restore_backup_exit_on_error)
  File "/usr/local/lib/python3.6/site-packages/datanator/core/data_source.py", line 187, in restore_backup
    manager.download(system_path=path, sym_links=True)
  File "/usr/local/lib/python3.6/site-packages/wc_utils/quilt.py", line 104, in download
    quilt.login_with_token(self.token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 423, in login_with_token
    auth = _update_auth(team, refresh_token)
  File "/usr/local/lib/python3.6/site-packages/quilt/tools/command.py", line 199, in _update_auth
    raise CommandException("Authentication error: %s" % response.status_code)
quilt.tools.command.CommandException: Authentication error: 401

gracefully handel paths.yml

what happens:

  1. run 'wc_env_manager container build' with a wc_env_manager.cfg in ~/.wc
  2. get:
    Traceback (most recent call last):
    File "/anaconda3/bin/wc_env_manager", line 11, in
    sys.exit(main())
    File "/Users/arthur_at_sinai/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/main.py", line 268, in main
    app.run()
    File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
    File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
    return func()
    File "/Users/arthur_at_sinai/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/main.py", line 170, in build
    mgr.setup_container()
    File "/Users/arthur_at_sinai/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/core.py", line 643, in setup_container
    self.get_config_file_paths_to_copy_to_image()
    File "/Users/arthur_at_sinai/gitOnMyLaptopLocal/wc_dev_repos/wc_env_manager/wc_env_manager/core.py", line 439, in get_config_file_paths_to_copy_to_image
    with open(filename, 'r') as file:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/arthur_at_sinai/.wc/third_party/paths.yml'

what's expected:

  1. error message which should a) explain why the file ~/.wc/third_party/paths.yml is expected, b) point to documentation on how to avoid the error

'wc_env_manager container build' is failing

'wc_env_manager container build' has stopped working on my Mac. I don't know why, as I don't believe that I've changed any of the code it uses.

This is a critical problem because I'm relying on docker containers for my wc_sim development. I will try reinstalling wc_utils and wc_env_manager now.

Here's a trace:

Script started on Sat Oct 20 08:19:40 2018
bash-3.2$ wc_env_manager container build
Traceback (most recent call last):
  File "/anaconda3/bin/wc_env_manager", line 11, in <module>
    sys.exit(main())
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 230, in main
    app.run()
  File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
    return func()
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 140, in build
    mgr = wc_env_manager.core.WcEnvManager({'verbose': VERBOSE})
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/core.py", line 81, in __init__
    'wc_env_manager': config or {}})['wc_env_manager']
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/config/core.py", line 39, in get_config
    return wc_utils.config.ConfigManager(paths).get_config(extra=extra, context=context)
  File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 127, in get_config
    raise ExtraValuesError(config)
  File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 284, in __init__
    the_section = config[section]
  File "/anaconda3/lib/python3.6/site-packages/configobj.py", line 554, in __getitem__
    val = dict.__getitem__(self, key)
KeyError: 'container'
bash-3.2$ wc_env_manager -h
usage: wc_env_manager [-h] [-d] [-q] [-v]
                      {container,image,base-image,build,pull,push,remove} ...

Whole-cell modeling environment manager

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           full application debug mode
  -q, --quiet           suppress all console output
  -v, --version         show program's version number and exit

sub-commands:
  {container,image,base-image,build,pull,push,remove}
    container           container controller
    image               image controller
    base-image          base-image controller
    build               Build base image, image, and container
    pull                Pull base image and image
    push                Push base image and image
    remove              Remove base image, image, and containers
bash-3.2$ wc_env_manager container
usage: wc_env_manager container [-h] {build,remove} ...

Build and remove containers of `wc_env`

optional arguments:
  -h, --help      show this help message and exit

sub-commands:
  {build,remove}
    build         Build container
    remove        Remove container
bash-3.2$ pip show wc_utils
Name: wc-utils
Version: 0.0.1a5
Summary: Utilities for whole-cell modeling
Home-page: https://github.com/KarrLab/wc_utils
Author: Karr Lab
Author-email: [email protected]
License: MIT
Location: /anaconda3/lib/python3.6/site-packages
Requires: mendeleev, requests, log, diskcache, enum34, six, numpy, qualname, gitpython, setuptools, pyexcel, attrdict, pyexcel-io, openpyxl, pint, pyyaml, configobj, abduct
Required-by: wc-env-manager
bash-3.2$ pip show wc_env_manager
Name: wc-env-manager
Version: 0.0.1
Summary: Tools for managing computing environments for whole-cell modeling
Home-page: https://github.com/KarrLab/wc_env_manager
Author: Karr Lab
Author-email: [email protected]
License: MIT
Location: /Users/arthur_at_sinai/.local/lib/python3.6/site-packages
Requires: pkg-utils, gitpython, cement, configobj, wc-utils, python-dateutil, requirements-parser, docker, pyyaml, jinja2
Required-by: 
bash-3.2$ pip show cement
Name: cement
Version: 3.0.0
Summary: CLI Framework for Python
Home-page: http://builtoncement.org
Author: Data Folk Labs, LLC
Author-email: [email protected]
License: BSD
Location: /anaconda3/lib/python3.6/site-packages
Requires: 
Required-by: wc-env-manager
bash-3.2$ wc_env_manager container remove
Traceback (most recent call last):
  File "/anaconda3/bin/wc_env_manager", line 11, in <module>
    sys.exit(main())
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 230, in main
    app.run()
  File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
    return func()
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 147, in remove
    mgr = wc_env_manager.core.WcEnvManager({'verbose': VERBOSE})
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/core.py", line 81, in __init__
    'wc_env_manager': config or {}})['wc_env_manager']
  File "/Users/arthur_at_sinai/.local/lib/python3.6/site-packages/wc_env_manager/config/core.py", line 39, in get_config
    return wc_utils.config.ConfigManager(paths).get_config(extra=extra, context=context)
  File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 127, in get_config
    raise ExtraValuesError(config)
  File "/anaconda3/lib/python3.6/site-packages/wc_utils/config/core.py", line 284, in __init__
    the_section = config[section]
  File "/anaconda3/lib/python3.6/site-packages/configobj.py", line 554, in __getitem__
    val = dict.__getitem__(self, key)
KeyError: 'container'

`wc-env-manager pull` fails

Trace below. The first 2 lines starting config are produced by my debugging code.

$ wc-env-manager pull

config['repo_unsquashed'] karrlab/wc_env_dependencies_unsquashed
config['tags'] ['latest', '0.0.51']
Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/anaconda3/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/karrlab/wc_env_dependencies_unsquashed:0.0.51/json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/bin/wc-env-manager", line 10, in <module>
    sys.exit(main())
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 277, in main
    app.run()
  File "/anaconda3/lib/python3.6/site-packages/cement/core/foundation.py", line 916, in run
    return_val = self.controller._dispatch()
  File "/anaconda3/lib/python3.6/site-packages/cement/ext/ext_argparse.py", line 806, in _dispatch
    return func()
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/__main__.py", line 236, in pull
    mgr.pull_image(config['repo_unsquashed'], config['tags'])
  File "/anaconda3/lib/python3.6/site-packages/wc_env_manager/core.py", line 491, in pull_image
    image = self._docker_client.images.pull(image_repo, tag=tag)
  File "/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 415, in pull
    repository, tag, '@' if tag.startswith('sha256:') else ':'
  File "/anaconda3/lib/python3.6/site-packages/docker/models/images.py", line 295, in get
    return self.prepare_model(self.client.api.inspect_image(name))
  File "/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/anaconda3/lib/python3.6/site-packages/docker/api/image.py", line 245, in inspect_image
    self._get(self._url("/images/{0}/json", image)), True
  File "/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 235, in _result
    self._raise_for_status(response)
  File "/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/anaconda3/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("no such image: karrlab/wc_env_dependencies_unsquashed:0.0.51: No such image: karrlab/wc_env_dependencies_unsquashed:0.0.51")

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.