cyverse / atmosphere Goto Github PK

View Code? Open in Web Editor NEW

326.0 48.0 102.0 177.47 MB

Where cloud lives.

Home Page: https://atmo.cyverse.org

License: Other

Python 93.23% Shell 1.63% Ruby 0.06% HTML 0.52% Gherkin 3.32% Dockerfile 0.11% Jinja 1.14%

atmosphere's Introduction

THIS REPOSITORY IS READONLY AND NO LONGER ACTIVE

atmosphere ☁️

Atmosphere addresses the growing needs for highly configurable and customized computational resources to support research efforts in plant sciences. Atmosphere is an integrative, private, self-service cloud computing platform designed to provide easy access to preconfigured, frequently used analysis routines, relevant algorithms, and data sets in an available-on-demand environment designed to accommodate computationally and data-intensive bioinformatics tasks.

Installation

Install the required python packages

pip install -r requirements.txt

A separate environment is provided for developers

pip install -r dev_requirements.txt

The *requirements.txt files are generated using pip-tools. See REQUIREMENTS.md for instructions on using pip-tools and upgrading packages in Atmosphere.

Some Features

A powerful web client for management and administration of virtual machines
A fully RESTful API service for integrating with existing infrastructure components
Virtual machine images preconfigured for computational science and iPlant's infrastructure

Running scripts

There are several utility scripts in ./scripts. To run these:

cd <path to atmosphere>
export DJANGO_SETTINGS_MODULE='atmosphere.settings'
export PYTHONPATH="$PWD:$PYTHONPATH"
python scripts/<name of script>

Contributing

See HACKING.md.

Coding Style

We rely on a tool to format the codebase. If you install from dev_requirements.txt, the yapf binary will be in your environment.

yapf --in-place --parallel   -- $(git ls-files | grep '\.py$')

When master is pulled, it's helpful to know if a pip install or a manage.py migrate is necessary. To get other helpful warnings:

ln -s $(pwd)/contrib/post-merge.hook $(pwd)/.git/hooks/post-merge

Coding Conventions

Import ordering

Imports should be grouped into the sections below and in sorted order.

Standard libraries
Third-party libraries
External project libraries
Local libraries

License

See LICENSE.txt for license information

Lead

Edwin Skidmore [email protected]

Authors

The following individuals who have help/helped make ☁️ great appear in alphabetic order, by surname.

Evan Briones [email protected]
Tharon Carlson [email protected]
Joseph Garcia [email protected]
Steven Gregory [email protected]
Jason Hansen [email protected]
Christopher James LaRose [email protected]
Amit Juneja [email protected]
Andrew Lenards [email protected]
Monica Lent [email protected]
Chris Martin [email protected]
Calvin Mclean [email protected]
Andre Mercer [email protected]
Connor Osborn [email protected]
J. Matt Peterson [email protected]
Julian Pistorius [email protected]

Where the cloud lives!

atmosphere's People

Contributors

Stargazers

Watchers

Forkers

nickeddy zhanghaihua prodigeni tomasbruno consueloau spencerx louisabitbol shelleyfeng imclab xingningdu arunkgupta chromabinary meliyo landreamer vijdata spenceruog 420reich kmeiern hariomrana eandhy malabady phruederi chtang7 cnbeining2 mmontgomeryii jscottwagner angelfirenze emilycressey singkorn ardicool cdosborn lenards hwijung ethanchen-field transformersprimeabcxyz bollig nfaction davianlou steve-gregory catdewey george-wu509 duke-gcb amercer1 cyberlusion znaznazna-backup julianpistorius hoangtv1899 amit4111989 jmlowe jchansen c-mart eriksf spezi94 liuzhenhzong anothergithubbase jextends ramega xuhang57 calvinmclean yixiaoj frankhe123 saharjs eewahn ifarhankhan waterstroke cci-moc mohan1e j143-zz daveromana wetinx naveentrtumkur apachipa reddydodda mcutshall thanglq123 tharon-c baoadrian michaelbarkdoll simpsonw xiaoruiguo seanmalbert benlazarine todun zhxu73 usapp-io edwins npurcella lxngoddess5321 hkberlin joskid evolution-2045 wook2014 kontum99

atmosphere's Issues

More dead code

I think that _get_init_script() in service/instance.py, and code near it, is also dead. Should be confirmed and removed from repo if appropriate. I can do this eventually.

Problem: Upgrade New Relic agent

Create link to our contributor guide

Related to #208

Probably missing packages in requirements.txt

@steve-gregory Here is how to reproduce:

rm -rf /opt/env/atmo
virtualenv atmo
source /opt/env/atmo/bin/activate
pip install -r /opt/dev/atmosphere/requirements.txt
cd /opt/dev/atmopshere
./run_test.sh

And you would see the test cases failed due to a few "no module" error.

Not sure if you could reproduce this.
Lucas

Idea: Read-only proxy to live clouds

Problem: Testing against live clouds can be scary. We don't want to accidentally break something.

Remove unnecessary uses of get_or_create

In the code base there are quite a few instance of get_or_create. In most cases this is an anti-pattern. It's used like so:

status, _ = StatusType.objects.get_or_create(name="closed")

The status model belongs in a migration. The app should be able to make the assumption that it has the status types. The implication of this sort of coding is that the correct initial state of the application is sort of unknown so you have to hit random lines like the one above to make sure your models are right.

Change activity from N/A to Done/Ready?

@steve-gregory

Just an idea:

I don't know whether it is just on my end or not. It would be great that the activity could be changed from N/A to some more meaningful word such as Ready?

Add new scenarios

Script for backward compatibility of new renewed and supplement events

Problem: UserAllocationSource should have unique `user_id` + `allocation_source_id` key

Reported by @cdosborn

Extend Python style guide

Steal/borrow from here: http://cosdev.readthedocs.io/en/latest/style_guides/python.html

Remove `EventTableUpdated` and just use `EventTable`. Use it only for migrations. It's going to cause bugs. Export current `event_table` for posterity.

Add an EditorConfig file

http://editorconfig.org/

Mock tas_api.tacc_api_post & tas_api.tacc_api_get at the scenario level (or higher) - Maybe using VCR

Problem: USE_ALLOCATION_SOURCES=False will cause problems

Solution: Remove USE_ALLOCATION_SOURCES setting from the application, as well as corresponding logic and resulting dead code

issue installing pygraphviz

When I am trying to download pygraphviz then following error occurs.

ammar@ammar-HP-15-Notebook-PC:~/Plants$ pip install pygraphviz
Collecting pygraphviz
Using cached pygraphviz-1.3.1.tar.gz
Building wheels for collected packages: pygraphviz
Running setup.py bdist_wheel for pygraphviz ... error
Complete output from command /home/ammar/anaconda2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-Ryh18C/pygraphviz/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" bdist_wheel -d /tmp/tmpK4uCKupip-wheel- --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-i686-2.7
creating build/lib.linux-i686-2.7/pygraphviz
copying pygraphviz/agraph.py -> build/lib.linux-i686-2.7/pygraphviz
copying pygraphviz/version.py -> build/lib.linux-i686-2.7/pygraphviz
copying pygraphviz/init.py -> build/lib.linux-i686-2.7/pygraphviz
copying pygraphviz/graphviz.py -> build/lib.linux-i686-2.7/pygraphviz
copying pygraphviz/release.py -> build/lib.linux-i686-2.7/pygraphviz
creating build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_readwrite.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_attributes.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_html.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_edge_attributes.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_drawing.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_string.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_node_attributes.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_graph.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_unicode.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/init.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_clear.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_subgraph.py -> build/lib.linux-i686-2.7/pygraphviz/tests
copying pygraphviz/tests/test_layout.py -> build/lib.linux-i686-2.7/pygraphviz/tests
running egg_info
writing pygraphviz.egg-info/PKG-INFO
writing top-level names to pygraphviz.egg-info/top_level.txt
writing dependency_links to pygraphviz.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found

reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '~' found anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '.svn' found anywhere in distribution
no previously-included directories found matching 'doc/build'
writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
copying pygraphviz/graphviz.i -> build/lib.linux-i686-2.7/pygraphviz
copying pygraphviz/graphviz_wrap.c -> build/lib.linux-i686-2.7/pygraphviz
running build_ext
building 'pygraphviz._graphviz' extension
creating build/temp.linux-i686-2.7
creating build/temp.linux-i686-2.7/pygraphviz
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ammar/anaconda2/include/python2.7 -c pygraphviz/graphviz_wrap.c -o build/temp.linux-i686-2.7/pygraphviz/graphviz_wrap.o
pygraphviz/graphviz_wrap.c:2954:29: fatal error: graphviz/cgraph.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

Failed building wheel for pygraphviz
Running setup.py clean for pygraphviz
Failed to build pygraphviz
Installing collected packages: pygraphviz
Running setup.py install for pygraphviz ... error
Complete output from command /home/ammar/anaconda2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-Ryh18C/pygraphviz/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-Rh8hBS-record/install-record.txt --single-version-externally-managed --compile:
running install
Trying pkg-config
Package libcgraph was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcgraph.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libcgraph' found
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-Ryh18C/pygraphviz/setup.py", line 87, in
tests_require=['nose>=0.10.1', 'doctest-ignore-unicode>=0.1.0',],
File "/home/ammar/anaconda2/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/ammar/anaconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/ammar/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-build-Ryh18C/pygraphviz/setup_commands.py", line 44, in modified_run
self.include_path, self.library_path = get_graphviz_dirs()
File "/tmp/pip-build-Ryh18C/pygraphviz/setup_extra.py", line 121, in get_graphviz_dirs
include_dirs, library_dirs = _pkg_config()
File "/tmp/pip-build-Ryh18C/pygraphviz/setup_extra.py", line 44, in _pkg_config
output = S.check_output(['pkg-config', '--libs-only-L', 'libcgraph'])
File "/home/ammar/anaconda2/lib/python2.7/subprocess.py", line 573, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['pkg-config', '--libs-only-L', 'libcgraph']' returned non-zero exit status 1

----------------------------------------

Command "/home/ammar/anaconda2/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-Ryh18C/pygraphviz/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-Rh8hBS-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-Ryh18C/p

Problem: `story.feature` broken

UnboundLocalError for monitor_sizes_for()

In [3]: monitor_sizes_for(4)
/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/httplib_ssl.py:229: UserWarning: SSL certificate verification is disabled, this can pose a security risk. For more information how to enable the SSL certificate verification, please visit the libcloud documentation.
  warnings.warn(libcloud.security.VERIFY_SSL_DISABLED_MSG)

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-3-18bd2a358b11> in <module>()
----> 1 monitor_sizes_for(4)

/opt/env/atmo/local/lib/python2.7/site-packages/celery/local.pyc in __call__(self, *a, **kw)
    186
    187     def __call__(self, *a, **kw):
--> 188         return self._get_current_object()(*a, **kw)
    189
    190     def __len__(self):

/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.pyc in __call__(self, *args, **kwargs)
    418             if self.__self__ is not None:
    419                 return self.run(self.__self__, *args, **kwargs)
--> 420             return self.run(*args, **kwargs)
    421         finally:
    422             self.pop_request()

/opt/dev/atmosphere/service/tasks/monitoring.pyc in monitor_sizes_for(provider_id, print_logs)
    708     # Non-End dated sizes on this provider
    709     db_sizes = Size.objects.filter(only_current(), provider=provider)
--> 710     all_sizes = admin_driver.list_sizes()
    711     seen_sizes = []
    712     for cloud_size in all_sizes:

/opt/env/atmo/local/lib/python2.7/site-packages/rtwo/driver.pyc in list_sizes(self, *args, **kwargs)
    274         return self.provider.sizeCls.get_sizes(
    275             self.provider,
--> 276             super(EshDriver, self).list_sizes)
    277
    278     def list_locations(self, *args, **kwargs):

/opt/env/atmo/local/lib/python2.7/site-packages/rtwo/models/size.pyc in get_sizes(cls, provider, lc_list_sizes_method)
     67         cached_sizes = cls.sizes.get(identifier)
     68         if not cached_sizes or not cls.lc_sizes:
---> 69             cls.lc_sizes = lc_list_sizes_method()
     70             logger.debug("Caching %s sizes for identifier:%s" %
     71                          (len(cls.lc_sizes), identifier))

/opt/env/atmo/local/lib/python2.7/site-packages/rtwo/driver.pyc in list_sizes(self, *args, **kwargs)
    168                      % (self.provider.identifier,
    169                      self.identity.credentials['key']))
--> 170         return self._connection.list_sizes()
    171
    172     def list_locations(self, *args, **kwargs):

/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/compute/drivers/openstack.pyc in list_sizes(self, location)
    297     def list_sizes(self, location=None):
    298         return self._to_sizes(
--> 299             self.connection.request('/flavors/detail').object)
    300
    301     def list_locations(self):

/opt/env/atmo/local/lib/python2.7/site-packages/rtwo/drivers/openstack_facade.pyc in request(self, action, params, data, headers, method, max_attempts)
     98                         action=action,
     99                         params=params, data=data,
--> 100                         method=method, headers=headers)
    101                 return response
    102             except (BaseHTTPError, httplib.HTTPException, socket.error,

/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/common/openstack.pyc in request(self, action, params, data, headers, method, raw)
    200                                                             method=method,
    201                                                             headers=headers,
--> 202                                                             raw=raw)
    203
    204     def _get_auth_url(self):

/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/common/base.pyc in request(self, action, params, data, headers, method, raw)
    707                                        False) or RETRY_FAILED_HTTP_REQUESTS
    708
--> 709         action = self.morph_action_hook(action)
    710         self.action = action
    711         self.method = method

/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/common/openstack.pyc in morph_action_hook(self, action)
    267
    268     def morph_action_hook(self, action):
--> 269         self._populate_hosts_and_request_paths()
    270         return super(OpenStackBaseConnection, self).morph_action_hook(action)
    271

/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/common/openstack.pyc in _populate_hosts_and_request_paths(self)
    300                 kwargs = {}
    301
--> 302             osa = osa.authenticate(**kwargs)  # may throw InvalidCreds
    303
    304             self.auth_token = osa.auth_token

/opt/env/atmo/local/lib/python2.7/site-packages/libcloud/common/openstack_identity.pyc in authenticate(self, force)
   1050             body = 'code: %s body:%s' % (response.status, response.body)
   1051         else:
-> 1052             raise MalformedResponseError('Malformed response', body=body,
   1053                                          driver=self.driver)
   1054

UnboundLocalError: local variable 'body' referenced before assignment

Remove `pip-tools` from dev_requirements.in

Don't let pip-tools manage pip-tools.

[DUPE] Figure out what to do with running instances when an associated UserAllocationSource is deleted

Problem: Inconsistent OpenStack and Atmosphere usage reporting

Email from George:

This is the issue I was mentioning on the call today. Our goal is to report both API and Atmo usage in a consistent format. It is up to us to determine that format.

I took a look at the information in the XDCDB and noted that the API and Atmosphere appear to report
usage differently. Non-Atmosphere jobs appear consistent between 1 SU = 1 processor = 1 node with
usage reported each hour using the uuid + event for the local job id but Atmosphere jobs don't
follow this format. Atmosphere jobs appear to report on the entire job and don't use the uuid but
rather a string such as "use.jetstream-cloud.org-28760796" (for example job_id=51110252). It looks
like I can calculate the number of virtual cores for Atmosphere jobs as round(3600.0/wallduration *
local_charge, 0) but does this mean that there is no intermediate reporting for long-running jobs
through Atmosphere? Ideally, the reporting should be as consistent as possible. Let me know your
thoughts.

Problem: Tests broken on `allocations`

/manage.py test --keepdb
Using existing test database for alias 'default'...
 ...
This can take a while...
Created 0 machines - 0:00:00.001172
Updated 0 machine_requests - 0:00:00.000936
Created 0 Volumes 0:00:00.000774
Updated 0 Instances 0:00:00.000916
Completed!
.....EE............................s..........................sssE.....................sFF.........................s.........s.s....
======================================================================
ERROR: test_invalid_provider_token_update (api.tests.v2.test_token_update.TokenUpdateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/test/utils.py", line 205, in inner
    return func(*args, **kwargs)
  File "/opt/dev/atmosphere/api/tests/v2/test_token_update.py", line 22, in test_invalid_provider_token_update
    url = reverse('api:v2:token_update-list')
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/urls/base.py", line 91, in reverse
    return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 392, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns)
NoReverseMatch: Reverse for 'token_update-list' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []

======================================================================
ERROR: test_valid_data_token_update (api.tests.v2.test_token_update.TokenUpdateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev/atmosphere/api/tests/v2/test_token_update.py", line 39, in test_valid_data_token_update
    url = reverse('api:v2:token_update-list')
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/urls/base.py", line 91, in reverse
    return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs)))
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 392, in _reverse_with_prefix
    (lookup_view_s, args, kwargs, len(patterns), patterns)
NoReverseMatch: Reverse for 'token_update-list' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: []

======================================================================
ERROR: test_invalid_query_params (api.tests.v2.test_reporting.ReportingTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev/atmosphere/api/tests/v2/test_reporting.py", line 126, in test_invalid_query_params
    response = self.view(request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/viewsets.py", line 83, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/dev/atmosphere/api/v2/views/reporting.py", line 270, in list
    results = super(ReportingViewSet, self).list(request, *args, **kwargs)
AttributeError: 'super' object has no attribute 'list'

======================================================================
FAIL: test_loggedin_user_can_list_allocation_sources (api.tests.v2.test_allocation_sources.AllocationSourceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev/atmosphere/api/tests/v2/test_allocation_sources.py", line 94, in test_loggedin_user_can_list_allocation_sources
    self.assertEqual(response.data['count'], len(expected_values))
AssertionError: 3 != 2

======================================================================
FAIL: test_loggedin_user_with_no_sources_cant_see_allocation_sources (api.tests.v2.test_allocation_sources.AllocationSourceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev/atmosphere/api/tests/v2/test_allocation_sources.py", line 72, in test_loggedin_user_with_no_sources_cant_see_allocation_sources
    self.assertEqual(response.data['count'], 0)
AssertionError: 3 != 0

----------------------------------------------------------------------
Ran 132 tests in 5.600s

FAILED (failures=2, errors=3, skipped=8)
Jetstream not installed -- Skipping migration 0079... Preserving test database for alias 'default'...

Image metrics test broken

Failure
Traceback (most recent call last):
  File "/opt/dev/atmosphere/api/tests/v2/test_image_metrics.py", line 159, in test_accurate_application_statistics
    self.assertEquals(response.status_code, 200)
AssertionError: 404 != 200

Suspend/Stop(stutoff) Instance doesn't work as expected

Hey Steve,

I tried the suspend/stop instance action on atmosphere and it turns out that it removes my fixed ip and floating ip after I suspend/stop the instance.

However, I think a user might want to keep their IPs when they want to suspend/stop their instances? I am not sure how Atmosphere is supposed to work here. Do you think it should keep the IPs as well?

Staging & test server email whitelist

Problem: We don't want to accidentally send out emails to users when using test environments.

Proposed solution:

my staging server has a whitelist of email addresses and phone numbers owned by me. Through the magic of monkeypatching, attempting to contact anyone else raises an exception. That sounded a little paranoid until that day when I accidentally created an infinite loop and rang every number in the database a hundred times.

See:
http://www.kalzumeus.com/2010/12/12/staging-servers-source-control-deploy-workflows-and-other-stuff-nobody-teaches-you/

Use pip-tools to lock down requirements.txt

For reproducible, reliable deployments.

Currently when I run pip-compile --verbose requirements.in I run into a problem where Atmosphere and its various dependencies require different versions of Python libraries (e.g. requests, djangorestframework, etc.)

Cannot Redeploy

@steve-gregory
Not exactly sure that whether this bug is a local one or not. But I think it worths to mention.

How to Reproduce:

After an instance turns into active (with a floating ip assigned), go ahead and click redeploy button to redeploy that instance.
monitor the instance status and it stays in Networking state
Celery/flower gives me this in the add_floating_ip task:

Traceback (most recent call last):
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "service/tasks/driver.py", line 1365, in add_floating_ip
    countdown=countdown)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 696, in retry
    raise ret
Retry: Retry in 64s: Conflict()

Remove drf SearchFilter monkey patch

The next release of drf includes a fix to avoid the exponential search problem.

encode/django-rest-framework#5264

This is a self-reminder to remove the patch, when we upgrade.

Problem: Missing snapshot will break monitor_allocation_sources

See: 0ddc641#commitcomment-23437938

Problem: How we use pip-compile creates package conflicts

Example: cyverse/troposphere#582

This affects Troposphere and Atmosphere

Wish List: Push-based UI updates (channels?) combined with Instance Events

Problem 1: The UI is often quite out of sync with the state of the cloud and relies on polling the Atmosphere API. (Refer to ATMO-1379)
Problem 2: The Atmosphere service itself is often out of sync with the state of the cloud and relies on polling the cloud provider. (Refer to ATMO-519)

Both these scenarios cause unnecessary load and lag.

Proposed Solution for 1: Instead of polling, the UI registers for events on instances and the Atmosphere service pushes these events to the UI. (Django Channels, WAMP, etc.)

Proposed Solution for 2: Instead of polling the cloud, Atmosphere registers for cloud events (Nova Notifications 1, 2, AWS CloudWatch events)

Problem: `KeyError: 'chargeCode'` in `create_reports`

/opt/dev/atmosphere/jetstream/tasks.pyc in create_reports()
     46                                                                                                     allocation_name))
     47             continue
---> 48         project_name = driver.get_allocation_project_name(allocation_name)
     49         try:
     50             project_report = _create_tas_report_for(

/opt/dev/atmosphere/jetstream/allocation.pyc in get_allocation_project_name(self, allocation_name)
    156 
    157     def get_allocation_project_name(self, allocation_name):
--> 158         allocation = self.get_allocation(allocation_name)
    159         if not allocation:
    160             return

/opt/dev/atmosphere/jetstream/allocation.pyc in get_allocation(self, allocation_name)
    174         filtered_list = [
    175             a for a in self.get_all_allocations()
--> 176             if str(a['chargeCode']) == str(allocation_name)]
    177         if len(filtered_list) > 1:
    178             logger.error(">1 value found for allocation %s" % allocation_name)

KeyError: 'chargeCode'

Investigate using Celery 'chords' for bulk instance actions

https://celery.readthedocs.io/en/latest/userguide/canvas.html#chords

DeviceBusyException error when detaching volume

We see this error in the celery logs when a user tries to detach a volume. This happens when there are processes locking the volume.

 File "service/exceptions.py", line 133, in __str__
    return "%s:\n%s" % (self.message, repr(self.process_list))
AttributeError: 'DeviceBusyException' object has no attribute 'process_list'

Problem: Upgrade Sentry

Problem: Found 8 events with empty ('') `entity_id`

See Jetstream database.

SELECT t.* FROM public.event_table t WHERE entity_id = '';

Deleted instance would not disappear unless on refresh.

Have already confirmed with Steve @steve-gregory this is a bug and it might on the backboneJS. Issue this to keep track of it.

Thanks!

Problem: New user on initial login are not receiving an allocation source

This issue has already acknowledged by @steve-gregory and a solution will be associating an allocation source with the user at the create new accounts step.

Still issuing this so Lucas can keep track of the changes and adopt it in GIJI. Since GIJI has a different use case and GIJI sets AUTO_CREATE_NEW_ACCOUNTS = False , another solution will be implemented downstream. (Thinking of putting the code in the token_update?)

Thanks
Lucas

Problem: Jetstream should be able to check TAS from Atmosphere

Saves me having to check the TAS API when Jeremy is trying to find out why a user can't log in to Atmosphere.

Maybe an admin panel app.

Integrate Amit's BDD tests from `cyverse_allocations`

Explain how we use pip-tools

Problem: pip-sync and pip-compile behavior can be surprising.

Proposed solution: Explain that indeed it is surprising, in addition to being awesome, and link to the pip-tools README.

Handle `user_allocation_source_deleted` when calculating reports.

Problem: Any usage between the last report for a user allocation source and the delete event will not be reported.

Fixed?

Problem: Cannot launch a new volume using OpenStackAuthBackbend

This may be an issue by using the OpenStackAuthBackend. Will investigate this and hopefully I could send a PR to fix this soon. @steve-gregory

Error message:

Internal Server Error: /api/v1/provider/3bab1cd5-bd77-40b5-8adf-28d98ee66075/identity/0f4f0131-1aa9-4977-b1b9-1a6b7d996fd2/volume
Traceback (most recent call last):
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 249, in _legacy_get_response
    response = self._get_response(request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/env/atmo/lib/python2.7/site-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/opt/env/atmo/lib/python2.7/site-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/env/atmo/lib/python2.7/site-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/dev/atmosphere/api/v1/views/volume.py", line 318, in post
    snapshot=snapshot, image=image)
  File "/opt/dev/atmosphere/service/volume.py", line 159, in create_esh_volume
    **conn_kwargs)
  File "/opt/env/atmo/lib/python2.7/site-packages/rtwo/driver.py", line 330, in create_volume
    return super(EshDriver, self).create_volume(*args, **kwargs)
  File "/opt/env/atmo/lib/python2.7/site-packages/rtwo/driver.py", line 197, in create_volume
    return self._connection.create_volume(*args, **kwargs)
  File "/opt/env/atmo/lib/python2.7/site-packages/rtwo/drivers/openstack_facade.py", line 63, in service_catalog_switch
    new_service = lc_conn.service_catalog.get_endpoint(
AttributeError: 'NoneType' object has no attribute 'get_endpoint'

Change `jetstream.allocation.get_or_create_allocation_source` to return a tuple again.

The created return value is used by jetstream.allocation.fill_allocation_sources

celery init.d files should be generic?

Given this page: http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#generic-init-scripts

Maybe we could have one generic celeryd and celerydbeat file?

https://github.com/cyverse/atmosphere/blob/master/extras/init.d/ubuntu.celerybeat
https://github.com/cyverse/atmosphere/blob/master/extras/init.d/centos.celeryd
https://github.com/cyverse/atmosphere/blob/master/extras/init.d/ubuntu.celeryd

test_valid_data_token_update fails when TESTING = True

Problem: test_valid_data_token_update fails when TESTING = True

I suspect this is due to Celery being set to 'eager' in local.py

Output:

======================================================================
ERROR: test_valid_data_token_update (api.tests.v2.test_token_update.TokenUpdateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/dev/atmosphere/api/tests/v2/test_token_update.py", line 48, in test_valid_data_token_update
    response = self.view(request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/viewsets.py", line 83, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/mixins.py", line 21, in create
    self.perform_create(serializer)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/mixins.py", line 26, in perform_create
    serializer.save()
  File "/opt/env/atmo/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 214, in save
    self.instance = self.create(validated_data)
  File "/opt/dev/atmosphere/api/v2/serializers/post/token_update.py", line 33, in create
    identity = self._create_identity(validated_data['provider'], validated_data['username'], validated_data['project_name'], validated_data['token'])
  File "/opt/dev/atmosphere/api/v2/serializers/post/token_update.py", line 66, in _create_identity
    cred_key=username, cred_ex_project_name=project_name, cred_ex_force_auth_token=token)
  File "/opt/dev/atmosphere/core/models/identity.py", line 200, in create_identity
    id_membership = identity.share(group, allocation=allocation)
  File "/opt/dev/atmosphere/core/models/identity.py", line 131, in share
    allocation=allocation)[0]
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/db/models/query.py", line 475, in get_or_create
    return self._create_object_from_params(lookup, params)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/db/models/query.py", line 505, in _create_object_from_params
    obj = self.create(**params)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/django/db/models/query.py", line 399, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/dev/atmosphere/core/models/group.py", line 221, in save
    set_provider_quota.apply_async(args=[str(self.identity.uuid)])
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 522, in apply_async
    link=link, link_error=link_error, **options)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 739, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 381, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/dev/atmosphere/service/tasks/admin.py", line 19, in set_provider_quota
    set_provider_quota.retry(exc=exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 678, in retry
    S.apply().get()
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/canvas.py", line 192, in apply
    return self.type.apply(args, kwargs, **options)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 739, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 381, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/dev/atmosphere/service/tasks/admin.py", line 19, in set_provider_quota
    set_provider_quota.retry(exc=exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 678, in retry
    S.apply().get()
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/canvas.py", line 192, in apply
    return self.type.apply(args, kwargs, **options)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 739, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 381, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/dev/atmosphere/service/tasks/admin.py", line 19, in set_provider_quota
    set_provider_quota.retry(exc=exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 678, in retry
    S.apply().get()
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/canvas.py", line 192, in apply
    return self.type.apply(args, kwargs, **options)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 739, in apply
    ret = tracer(task_id, args, kwargs, request)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 381, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/dev/atmosphere/service/tasks/admin.py", line 19, in set_provider_quota
    set_provider_quota.retry(exc=exc)
  File "/opt/env/atmo/local/lib/python2.7/site-packages/celery/app/task.py", line 668, in retry
    raise_with_context(exc)
  File "/opt/dev/atmosphere/service/tasks/admin.py", line 13, in set_provider_quota
    return spq(identity_uuid)
  File "/opt/dev/atmosphere/service/quota.py", line 116, in set_provider_quota
    return set_openstack_quota(identity)
  File "/opt/dev/atmosphere/service/quota.py", line 139, in set_openstack_quota
    % identity.provider.get_type_name())
Exception: Cannot set provider quota on type: mock

See: api.tests.v2.test_token_update.TokenUpdateTests#test_valid_data_token_update

Organization URL 404s

Not sure which repo to put this in but the URL for the iPlant Collaborative GitHub Organization at https://github.com/iPlantCollaborativeOpenSource goes to http://www.iplantcollaborative.org/opensource, which doesn't exist.

Problem: `behave` tests fail when not `settings.USE_ALLOCATION_SOURCE`

Problem: Duplicate `user_allocation_source` Events

Description

Duplicate user_allocation_source events were created every time update_snapshot periodic task ran

Tests
Migration to remove previous duplicates

Fixed by: #351

Problem: Not handling running instances when user removed from allocation

What happens to instances associated with an allocation source when the user is removed from an allocation source?

From Steve: "ugly project modal and just go through the same user behavior as launched in openstack and found by api?"