slipstream / slipstreamconnectors Goto Github PK

open-source cloud connectors for SlipStream

License: Apache License 2.0

Java 25.01% Clojure 21.20% Python 53.78%

slipstreamconnectors's Introduction

SlipStream is now deprecated -> replaced by https://github.com/nuvla

SlipStream

Developed by SixSq, SlipStream is a multi-cloud application management platform. See the SlipStream product page for more detailed information.

Release Notes

The release notes are available from the main SlipStream documentation website. Stable releases are those that have been validated and are supported. Candidate releases are (usually) the result of each development iteration and may or may not be stable. These can be used but are not supported.

Getting started

See the Developer Guide on the main SlipStream documentation website. This contains the full, supported build procedure for SlipStream, including the required build tools and dependencies.

License and copyright

The code in the public repositories is licensed under the Apache license.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

slipstreamconnectors's People

Contributors

Stargazers

Watchers

Forkers

cjdcordeiro jolorenzo

slipstreamconnectors's Issues

The CloudStack connector doesn't use the parameter "zone"

I discovered that the CloudStack connector doesn't use the parameter "zone".

In the create_node function of libcloud this is called location but it's not documented.

PhysicalHost: migrate connector to utilise new Java->Python CLI model

"Unknown" VM state is confusing on the clouds where the same named state exists

On StratusLab "Unknown" means ONE couldn't get the state of the VM. When troubleshooting runs, in this case it's difficult to understand what is the actual state of the VM.
Suggestion is to change the default VM state when SS doesn't know the actual state of the VM to for example "SS-Unknown".

Instantiation of PhysicalHost connector via the configuration/ doesn't create a corresponding section

It's a bit misleading that actually the instance gets created, but the corresponding section is missing. Connector's configuration related factory should create a dummy parameter for the section to appear in the configuration/ resource.

Add missing root disk metrics for StratusLab

Opennebula connector with ability to run multi-component application

Will be done after ticket #59

[occi] getters on User object no longer raise exceptions if attribute is not found - update the code accordingly

When testing, Enol (EGI) stumbled upon the following issue:

ss:abort- Exception with detail: sequence item 8: expected string, NoneType found

This turned out to be related to None returned by user_info.user_info.get_cloud('proxy_file')
https://github.com/slipstream/SlipStreamConnectors/blob/master/occi/python/tar/slipstream_occi/OcciClientCloud.py#L146
i.e. when the attribute is not found user_info.get_cloud('proxy_file') doesn't raise, but rather returns None (by default).

Simple image are not deployed with requested CPU/RAM

Linked with https://github.com/SixSq/support/issues/211

Test/check physical connector

Unknown" VM state is confusing on the clouds where the same named state exists

The StratusLab connector is using his custom machanisme to build image instead of ...

... using the SS one like all other connectors _build_image_increment.

This prevent it to benefit from update in the SS _build_image_increment.
Example:
The addition of export DEBIAN_FRONTEND in get_packages_install_command which allow to install some packages like MySQL on Debian based OS.
Without this, the installation will hang because it will be waiting forever a user input.

StratusLab build image: orch should terminate VM if it's still running after the timeout on wait for the new image ID

Follow the state of the VM. It should go from Running to Shutdown -> Epilog -> Done. Otherwise, we leave running VMs in case of problems on StratusLab cloud.

[Cloudstack] authn error is not logged when running run-instances

[root@ip-10-73-188-103 ~]# grep Unauthorized /opt/slipstream/server/logs/slipstream.log.8.bak
[root@ip-10-73-188-103 ~]#

Also, below is the example of the POST at 2015-12-01T05:09:00.329 not followed by run-instances. The run failed with Unauthorized see the screenshot.

This is the ticket https://portal.exoscale.ch/tickets/1008395 on Exoscale side to investigate the reason for the Unauthorized issue.

This might actually be more of an issue of the server and client code base-lines. I'm creating it for Cloudstack for the moment until it gets understood.

[root@ip-10-73-188-103 ~]# grep -E "(test.*POST.*curl|run-instances.*MEKtgh8sw)" /opt/slipstream/server/logs/slipstream.log.8.bak | sed -e "s/$pass/HIDDEN/g"
 2015-12-01T03:02:25.819+0000 INFO com.sixsq.slipstream.util.ProcessUtils execGetOutputAsArray Calling: sh -c /usr/bin/cloudstack-run-instances --password 'HIDDEN' --endpoint 'https://api.exoscale.ch/compute' --native-contextualization 'linux-only' --zone 'CH-GV2' --instance-type 'Micro' --security-groups 'slipstream_managed' --networks '' --login-username '' --zone-type 'Basic' --username 'MEKtgh8swrU1wVU4Qf7aYKFVVxUOB_cLvsoW25pQMsJxYI6J6Y4UuK-SfeFdaZs-l8YEnblF3WXd0_4t9igmsA' --image-id '0d732f4f-d4f1-49ce-b4d1-905de65d06a8' --network-type 'Public'
 2015-12-01T03:12:00.354+0000 INFO org.restlet.engine.log.LogFilter afterHandle 2015-12-01      03:12:00        127.0.0.1       test    127.0.0.1       80      POST    /run    -       201     0       386     226     https://nuv.la  curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2        -
 2015-12-01T03:12:00.408+0000 INFO com.sixsq.slipstream.util.ProcessUtils execGetOutputAsArray Calling: sh -c /usr/bin/cloudstack-run-instances --password 'HIDDEN' --endpoint 'https://api.exoscale.ch/compute' --native-contextualization 'linux-only' --zone 'CH-GV2' --instance-type 'Micro' --security-groups 'slipstream_managed' --networks '' --login-username '' --zone-type 'Basic' --username 'MEKtgh8swrU1wVU4Qf7aYKFVVxUOB_cLvsoW25pQMsJxYI6J6Y4UuK-SfeFdaZs-l8YEnblF3WXd0_4t9igmsA' --image-id '0d732f4f-d4f1-49ce-b4d1-905de65d06a8' --network-type 'Public'
 2015-12-01T03:16:00.686+0000 INFO org.restlet.engine.log.LogFilter afterHandle 2015-12-01      03:16:00        127.0.0.1       test    127.0.0.1       80      POST    /run    -       201     0       131     755     https://nuv.la  curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2        -
 2015-12-01T03:16:00.743+0000 INFO com.sixsq.slipstream.util.ProcessUtils execGetOutputAsArray Calling: sh -c /usr/bin/cloudstack-run-instances --password 'HIDDEN' --endpoint 'https://api.exoscale.ch/compute' --native-contextualization 'linux-only' --zone 'CH-GV2' --instance-type 'Micro' --security-groups 'slipstream_managed' --networks '' --login-username '' --zone-type 'Basic' --username 'MEKtgh8swrU1wVU4Qf7aYKFVVxUOB_cLvsoW25pQMsJxYI6J6Y4UuK-SfeFdaZs-l8YEnblF3WXd0_4t9igmsA' --image-id '0d732f4f-d4f1-49ce-b4d1-905de65d06a8' --network-type 'Public'
 2015-12-01T05:09:00.329+0000 INFO org.restlet.engine.log.LogFilter afterHandle 2015-12-01      05:09:00        127.0.0.1       test    127.0.0.1       80      POST    /run    -       201     0       131     221     https://nuv.la  curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.2.3 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2        -
 2015-12-01T07:15:05.556+0000 INFO com.sixsq.slipstream.util.ProcessUtils execGetOutputAsArray Calling: sh -c /usr/bin/cloudstack-run-instances --password 'HIDDEN' --endpoint 'https://api.exoscale.ch/compute' --native-contextualization 'linux-only' --zone 'CH-GV2' --instance-type 'Micro' --security-groups 'slipstream_managed' --networks '' --login-username '' --zone-type 'Basic' --username 'MEKtgh8swrU1wVU4Qf7aYKFVVxUOB_cLvsoW25pQMsJxYI6J6Y4UuK-SfeFdaZs-l8YEnblF3WXd0_4t9igmsA' --image-id '0d732f4f-d4f1-49ce-b4d1-905de65d06a8' --network-type 'Public'
[root@ip-10-73-188-103 ~]#

hardcoded /usr/bin path

I'm trying to run SS from a development environment (i.e. not installed) and there are hardcoded pathes:

WARNING: Failed contacting cloud [SlipStreamRuntimeException]: [toto/myos] with 'sh: 1: /usr/bin/openstack-describe-instances: not found

I tried to change CLI_LOCATION from line 51 in:
SlipStreamServer/jar-connector/src/main/java/com/sixsq/slipstream/connector/CliConnectorBase.java

but that is not working, I think my modification is not taken...

use StratusLab Runner interface rather than OpenNebula Runner class

Moved from slipstream/SlipStreamClient#2

The StratusLab client has been refactored to allow backward compatibility with previous StratusLab deployments while supporting the newer CIMI interface for new deployments.

The SlipStream client relies directly on the Runner class for OpenNebula. This will fail when StratusLab infrastructures deploy a newer StratusLab release. The client should be refactored to use the interface and add a parameter to switch between which is active for a given infrastructure.

PhysicalHost: simple run uses Orch's IP (defined in User params), instead image ID from "Cloud Image Identifiers and Image Hierarchy" -> "Cloud image IDs" should be used

Simple run uses Orch's IP (defined in User params), instead image ID from "Cloud Image Identifiers and Image Hierarchy" -> "Cloud image IDs" should be used (which should be the actual IP of the host.)

OpenNebula connector allow disk size redefinition in image parameter

openstack connector dependencies not installed

The python dependencies for the OpenStack connector are not installed with the packages. The following needed to be installed manually (via yum or pip):

apache-libcloud (pip, version 0.14.1)
paramiko (pip)
scpclient (pip)
isodate (pip)
xmlwitch (pip)
python-devel (yum, needed by paramiko)
gcc (yum, needed by paramiko)
autoconf (yum, needed by paramiko)
The package installation for the connector should either provide/require these dependencies or the documentation should be updated including instructions about installing these dependencies.

move dependency versions to main pom.xml file

Move all dependency version definitions to the main pom.xml file. The OCCI connector and possibly others have defined local versions for the dependencies.

Extract classes from CLIs to be reused

OpenStack: Add support for Floating IPs

More help popups in configuration screen

Usability matters

Please add more help popups (small circled "?" icons) on each configuration items.

I'd personally go for mandatory help texts for ALL config items.
And don't forget the connectors ones.

For example:

In "SlipStream Basics"

"Cloud connector java class name(s) (comma separated for multi-cloud configuration)"
put the "(comma...)" in a help popup, and even add examples here...

Or in "openstack connector instance"

"Image Id of the orchestrator"
put the text, extracted from your blog:
"The image id of the Orchestrator needs to match a Linux image with wget and python installed. An Ubuntu 12.04 will do the job perfectly."

move connector code from SlipStreamClient into this repository

The python bits of the open source connectors are currently in SlipStreamCilent. Move that code into this repository in preparation for creating separate connector packages.

In StratusLab connector substitue SLIPSTREAM_PDISK_ENDPOINT env var with CLI param

OpenNebula connector have public network and private network capability

PhysicalConnector: erroneous double qualification of the ORCHESTRATOR_HOST parameter

This prevents from getting the IP of the orch and effectively fails to start the run.

create connector for OpenNebula

Create a SlipStream connector for OpenNebula infrastructures.

Revive PhysicalHost connector

move SlipStreamConnector-Occi code into this repository

Bring all of the open source SlipStream connectors under this repository. All of the ones residing in SlipStreamServer have been moved here. The only external one is the OCCI connector which now should be moved in.

Opennebula connector with ability to run single-component application

OpenNebula-describe-instances
OpenNebula-run-instances
OpenNebula-terminate-instances

create prototype docker connector

Create an initial prototype docker connector that wraps the docker CLI. Verify that the basic functionality of the connector works and then investigate what the best mechanism for handling image management for a docker resource would be.

[cloudstack] Termination of runs with cloudstack connector on Exoscale from time to time fails and the runs stay in Finilizing

This was noticed on nuv.la (SS v2.20) with exoscale-ch-gva:cloudstack connector instance, when attempting to manually terminate scalable runs from Ready state (successful or aborted). When the failure occurs, UI prints error message - (something like) failed to terminate the run. The VM of the deployment are always properly terminated.

The frequency is relatively high ~25%

SS logs are attached. For example look for run with uuid 2e4ccbbe-a3e1-4b4e-bcf1-5b7372bf8648
slipstream.logs.zip

SS server crashes after working some time with OCCI connector

Only OCCI connector was installed and three active instances were configured on the SS instance.

There is a number of OCCI CLIs is still running.

[root@machinef81e00ff-a18a-43b4-afa3-4c67c9e80a47 ~]#
...
30143 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450107787490-3794330107 --output-format json_extended --action describe --resource compute
30274 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450108776478-449789104 --output-format json_extended --action describe --resource compute
31821 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269249009-3375138242 --output-format json_extended --action describe --resource compute
32198 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269483893-3269184247 --output-format json_extended --action describe --resource compute
32226 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269526704-608208313 --output-format json_extended --action describe --resource compute
32254 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269591882-2970614110 --output-format json_extended --action describe --resource compute
32277 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269694751-4037480938 --output-format json_extended --action describe --resource compute
32400 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269934771-1826185782 --output-format json_extended --action describe --resource compute
32448 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450269967509-1001300010 --output-format json_extended --action describe --resource compute
  576 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450270414795-681737162 --output-format json_extended --action describe --resource compute
  663 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450270654818-3375521313 --output-format json_extended --action describe --resource compute
  814 ?        Sl     0:00 /opt/occi-cli/embedded/bin/ruby /opt/occi-cli/bin/occi --endpoint https://carach5.ics.muni.cz:11443 --auth x509 --voms --skip-ca-check --user-cred /opt/slipstream/server/tmp/proxy-1450271134838-3309199992 --output-format json_extended --action describe --resource compute
[root@machinef81e00ff-a18a-43b4-afa3-4c67c9e80a47 ~]# service slipstream status
Checking arguments to Jetty:
START_INI      =  /opt/slipstream/server/start.ini
START_D        =  /opt/slipstream/server/start.d
JETTY_HOME     =  /opt/slipstream/server
JETTY_BASE     =  /opt/slipstream/server
JETTY_CONF     =  /opt/slipstream/server/etc/jetty.conf
JETTY_PID      =  /var/run/slipstream.pid
JETTY_START    =  /opt/slipstream/server/start.jar
JETTY_LOGS     =  /opt/slipstream/server/logs
JETTY_STATE    =  /opt/slipstream/server/slipstream.state
CLASSPATH      =
JAVA           =  /etc/alternatives/jre_1.8.0/bin/java
JAVA_OPTIONS   =  -Djetty.logging.dir=/opt/slipstream/server/logs -Djetty.home=/opt/slipstream/server -Djetty.base=/opt/slipstream/server -Djava.io.tmpdir=/opt/slipstream/server/tmp
JETTY_ARGS     =  -Dslipstream.ui.util.localization.lang-default=en-US jetty.state=/opt/slipstream/server/slipstream.state --exec -Dslipstream.config.file=/etc/slipstream/slipstream.conf jetty-logging-ss.xml jetty-started.xml
RUN_CMD        =  /etc/alternatives/jre_1.8.0/bin/java -Djetty.logging.dir=/opt/slipstream/server/logs -Djetty.home=/opt/slipstream/server -Djetty.base=/opt/slipstream/server -Djava.io.tmpdir=/opt/slipstream/server/tmp -jar /opt/slipstream/server/start.jar -Dslipstream.ui.util.localization.lang-default=en-US jetty.state=/opt/slipstream/server/slipstream.state --exec -Dslipstream.config.file=/etc/slipstream/slipstream.conf jetty-logging-ss.xml jetty-started.xml
[root@machinef81e00ff-a18a-43b4-afa3-4c67c9e80a47 ~]#

Wrong path to connector Python lib and wrong connector URL in OpenStack and CloudStack connectors

Revealed just now during the validation of v2.15 :(

PhysicalHost: stop building RPM and remove currently built RPM artefacts from Nexus and YUM snapshots to prevent it to be released

Add cloud parameter on Exoscale connector to define the size of the root disk

This can either be

a separate connector for Exoscale with drop-down menu with 10,50,100,200,400 GB. (Those values are currently defined on Exoscale.)

if Exoscale actually allows to provide custom values, then simply update the generic CloudStack connector with ability to provide custom size via "edit box" (probably with a limit to 500GB).

make sure connectors delete tmp files they create

StratusLab connector leaves /tmp/stratuslab_*.{err,log} files (they come from SL API; need to disable their creation).

Improve stratuslabiter-terminate-instances to be faster

Currently this command is very slow, especially for big Run.

openstack-describe-instances command fails because of libcloud warning

The openstack-describe-instances command fails because libcloud is generating a warning on stderr:

WARNING: Failed contacting cloud: CC-IN2P3 on behalf of cavet with 'Error returned by describe command. Got: /usr/lib/python2.6/site-packages/libcloud/httplib_ssl.py:59: UserWarning: SSL certificate verification is disabled, this can pose a security risk. For more information how to enable the SSL certificate verification, please visit the libcloud documentation.
  warnings.warn(libcloud.security.VERIFY_SSL_DISABLED_MSG)
id state
045e2503-2646-44d1-9fce-2ad3906d85bd Running
36b80134-a603-404d-bda6-7bcf727b08c1 Running
'

The function actually returned the correct information, but the warning text is preventing the parsing of the output.

Opennebula connector with ability to build image

allow network identifier to be specified in connector parameters

For a pre-installed OpenStack cloud at IPHC, it is necessary to specify the network identifier associated with the public/private networks. The command with the nova interface is:

$ nova boot … --nic net-id=e9bb6250-6d0c-48e3-8788-900789258fd7 ID

Should public/private network IDs be added as parameters for the cloud connector? Or is there another way to support this functionality? (Requiring a specific configuration of the IPHC cloud for SlipStream is not a desired solution.)

create separate packages for each cloud connector

Continuation of slipstream/SlipStreamServer#186. This requires that connector code in SlipStreamClient be moved here first (see Issue #2).

extend openstack connector to manage floating IP addresses

Some OpenStack deployments use floating IP addresses for allocating public addresses to machines. For these infrastructures, it is necessary to manage the allocation, assignment, deallocation, and destruction of the IP addresses for machines.

For an initial implementation, I'd suggest that a boolean flag is added to the OpenStack system parameters that indicates whether floating IP addresses should be used. If so, then for each machine a new floating IP address is allocated and assigned to the machine after it has been created. When the machine terminates, then the IP address assignment should also be terminated and the IP address should be freed.

StratusLab connector: Extra disk doesn't work with simple run (run an image)

Extra disk is ignored when we start a simple run (run an image).
Top-level issue: slipstream/SlipStream#16

Calculation of the root disk size fails on stratuslabiter connector

On StratusLabIter connector, root disk size calculation should try to avoid taking into account extra disks. Especially, substituting the root disk size by the extra disk size (as this is seen below 86731, Running, 212.128.52.81, 1, 1536, 1, - should have been 6 instead of 1). In the case of stratuslabiter connector the volatile extra disks get created on PDisk, which complicates the matter.

[root@bluebox stratuslabiter-pdisk-742]# /usr/bin/stratuslabiter-describe-instances  -t 60 --password xxx --endpoint '154.48.152.10' --username sixsquser -vvv
id, state, ip, cpu [nb], ram [MB], root disk [GB], instance-type [name]
85709, Running, 212.128.48.49, 1, 1536, 6,
86712, Running, 212.128.52.62, 1, 1024, 6,
86713, Running, 212.128.52.63, 1, 1024, 6,
86731, Running, 212.128.52.81, 1, 1536, 1,
[root@bluebox stratuslabiter-pdisk-742]#

PhysicalHost: rework the way the IPs are provided in the image module and treated during the deployment instantiation

With the current model it's not possible to run concurrent deployments which use the same images, because the images always define the same IPs.

Solution:

do not require image IP to be set on the image module
instead, require IP to be set as a "mandatory" run parameter (something new to introduce) during the instantiation of the run on the physicalhost connector instance. The number of the required IPs should be coupled with the multiplicity parameter.

The latter requires rework of how the run parameters are provided and treaded both UI and server side.

OpenNebula connector allow user to define size of Ram and Cpu

Update CloudStack connector documentation

Advanced zone instantiation
Network parameters

Update README.md files

Update these files once the PR slipstream/SlipStreamDocumentation#63 has been merged.