materialsproject / propjockey Goto Github PK

Propjockey helps you queue up workflows and dynamically change their priorities for running; it's like having a slipmat for scientific computing.

Propjockey is a tool for staff managing scientific computing resources to facilitate a user community voting to prioritize calculation of properties across a database of known entities. For example, across a database of entities corresponding to known crystalline materials, the full elastic tensor may not already be calculated for each material because such a calculation is computationally expensive, not of interest for all materials, etc.

The staff managing the growth and dissemination of this database of material structures and properties wishes to empower the community of users consuming the data to help prioritize computational jobs in a way that will dovetail with in-house priorities. Propjockey helps with this, connecting users to details of running workflows and providing email notification when property calculations complete.

• Former name: Interactive Leaderboard for Property Requests and Notification (ILPRN).

Example deployment: http://elastic.pj.materialsproject.org/.

Science Gateways 2016 conference: extended abstract and talk slides.

In a fresh virtualenv:

pip install -e .
export FLASK_APP=propjockey
export FLASK_DEBUG=1
export PROPJOCKEY_SETTINGS=$(pwd)/local_settings.py
flask run

There is a provided local_settings.example.py to serve as a template for your local_settings.py, which if stored at the root level will be .gitignored.

Ensure local settings are appropriate for your use case. Function definitions are part of the settings. Hopefully, many of the various settings, including some of the functions, will not need to change for you to get started.

You can make a test database derived from your real, live data. The database will be located at mongodb://localhost:27017/propjockey_test and have entries, votes, and workflows collections. All user emails in your real data will be aliased to protect user identities. To drop any existing test database and generate a new one using your data:

flask make_test_db

Then, to test the code against the test database:

python setup.py test

The USE_TEST_CLIENTS local setting enables you to use your test database during development and not just when running automated tests. This is nice when lacking a reliable/fast network connection.

To save your test database to a file for backup/sharing:

mongodump --db=propjockey_test --gzip --archive=propjockey_test.gz

and to restore it during development / on a testing server:

mongorestore --drop db=propjockey_test --gzip --archive=propjockey_test.gz

There are many officially documented options for deploying a Flask app such as this, for example gunicorn behind an nginx proxy, e.g.

# activate the virtualenv
gunicorn -w 4 -b 0.0.0.0:4000 propjockey:app
# and ensure your nginx configuration `proxy_pass`es to 0.0.0.0:4000

An example proxy setup is described at the official Flask documentation here.

# activate the virtualenv
# cd to directory with local_settings.py
# Ensure `USE_TEST_CLIENTS` local setting is False
export PROPJOCKEY_SETTINGS=$(pwd)/local_settings.py
python -m propjockey.notify