slipstream / slipstreamserverdeps Goto Github PK

The missing package prevents from running libcloud dependant cloud connectors after the migration to libcloud 0.18.0

With the current 4096 it takes more than 18 min to generate and is done from the slipstream-server-nginx-conf RPM post-install script [1].

It takes too long and artificially increases deployment times in CI.

Proposed solution:

1. change from 4096 to 1024 in the RPM post-install.
2. If the need for a value higher 1024 is justified, update slipstream.sh with openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096 && systemctl restart nginx after the RPM installation, and probably only on enterprise edition.

[1]

  982 ?        Ssl    0:00 python /opt/slipstream/client/sbin/slipstream-node -v start
 1016 ?        S      0:00  \_ /bin/bash /var/lib/slipstream/module_slipstream_ss-ci-pipeline_ss-deployer_3535__deployment
 1021 ?        S      0:00      \_ /bin/bash ./ss-deployer.sh
22451 ?        S      0:00          \_ /bin/bash /tmp/ss-install-ref-conf.sh -r http://nexus.sixsq.com/service/local/artifact/maven/redirect?r=snapshots-enterprise-rhel7&g=com.sixsq.slipstream&a=SlipStre
22471 ?        S      0:00              \_ bash ./slipstream.sh -S -k snapshot -e community -d hsqldb
25012 ?        S      0:01                  \_ /usr/bin/python /usr/bin/yum install -y slipstream-server-nginx-conf-community
25137 ?        S      0:00                      \_ /bin/sh /var/tmp/rpm-tmp.Lzgroi 1
25153 ?        S      0:00                          \_ /bin/sh /opt/slipstream/server/sbin/generate-dh-params.sh
25154 ?        R     18:03                              \_ openssl dhparam -out /etc/nginx/ssl/dhparam.pem 4096

And indeed

[root@deployer1979c2a75-94bd-4990-9cba-2ecc2394cda1 ~]# openssl dhparam -out /tmp/blah.pem 4096
Generating DH parameters, 4096 bit long safe prime, generator 2
This is going to take a long time
.................................................................+......................................................................................+......................................................................................................................................................................^C
[root@deployer1979c2a75-94bd-4990-9cba-2ecc2394cda1 ~]#

The jar file name is currently SlipStreamServiceOfferAPI.jar. Change this to something more relevant.

Related to slipstream/SlipStream#215

Related to https://github.com/SixSq/tasklist/issues/895

Update the nginx webui configuration so that all URLs below /webui are routed to the single-page application at /webui/index.html.

At the moment by default it's on /var. If the partition gets full, nginx sends truncated files. Need to be more flexible with this.

Moving to /var/run/slipstream/nginx/cache

• Here is the ss-backup-get that uses 'boto' https://github.com/slipstream/SlipStreamServerDeps/blob/master/backup/src/utils/ss-backup-get
• Needed utility: ss-backup-put <s3-bucket-url> <backup-file> that would substitute s3curl.pl
• S3 account configuration should be migrated from ~/.s3curl to ~/.boto
• Update of SS installation script is required

package.os is provided by the root POM SlipStream/pom.xml.

Move the riemann dependency from the enterprise repository to the community repository.

The script uses set -e which forces it to return before the actual printout of the output from the S3 utility happens.

Backup timestamp file should be stored as /var/run/slipstream/slipstream-backup-timestamp.
Backup launched by cron is using slipstream user.
It should be ensured that /var/run/slipstream is there and writable by slipstream user.

/var/log/slipstream-backup-timestamp -> /var/log/slipstream/slipstream-backup-timestamp

Depends on slipstream/SlipStreamServer#543

slipstream user should be used. Backup script should check the effective user name and exit with error if it's not 'slipstream'.

• move build of jar file to boot
• clean up dependencies in main pom.xml file
• update dependencies to (if any) to repo
• add dependency on riemann to rpm (not done because riemann is not in an rpm repo)
• move sources into rpm module
• trigger rpm build on mac

On Mac Os , build would raise error because RPM utility is not present, but installing RPM would also make the build fail. Therefore the rpm build should be run conditionaly

The installation of phantomjs is missing a link to /usr/bin/phantomjs. The pricing information providers will not run without this link.

Ensure that requests proxied through nginx always have an empty value for the "slipstream-authn-info" header.

Create an RPM package to facilitate the installation of phantomjs for pricing scrappers. This is also needed for unit testing in clojurescript.

It would be useful to be able to associate a particular user session with a given virtual host. To do this, the SNI host information must be passed from nginx to the backend services. Add a request header to provide this information. The variable in nginx with this value is $ssl_server_name.

It looks like it fails since the end of Sept'16

Wed Sep 28 16:30:01 UTC 2016
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   179  100   179    0     0   5659      0 --:--:-- --:--:-- --:--:--  5774
{"error":{"root_cause":[{"type":"repository_missing_exception","reason":"[es_backup] missing"}],"type":"repository_missing_exception","reason":"[es_backup] missing"},"status":404}

The maximum size of a report is limited by nginx to be around 1MB by default. This should be increased as logs can be rather large. Discuss what is a good limit (2MB, 5MB?) and change the default configuration.

The file to change is /etc/nginx/conf.d/slipstream-extra/limit-reports.block:

location ~ ^/reports/ {
    include conf.d/slipstream-proxy.params;

    limit_conn reportslimit 10;
    client_max_body_size 5M;
}

Use 5MB as the maximum for now. Developer votes ranged from 2MB to 10MB.

The maintenance will do the following:

• Return the correct HTTP code (503). So that search engines can handle that case correctly.
• Display a meaningful message for users who try to connect to SlipStream.
• Allow some IPs to bypass the maintenance mode.

The custom error pages allow to return the error in the format which has been requested.
By default, nginx always serves default error pages as HTML.

As part of migration to centos 7.
Connected to https://github.com/SixSq/tasklist/issues/618

The riemann service as deployed on Nuvla (v3.22) currently fails. This is because the sysconfig file references the wrong jar file. The jar file has changed from SlipStreamServiceOfferAPI.jar to SlipStreamRiemann.jar.

The HSQLDB code seems to work correctly with Java 1.7. Move the dependency from 1.6 to 1.7.

The backup of the server should be split into two: one for the database and one for the reports. The database backup is critical. The report backup is less so.

Create an RPM that configures a machine to use the elasticsearch repository.

The StratusLab distribution has updated its package structure. We need to change the way we extract and repackage it here

At the moment, the link gets created in the postinstall script in slipstream-phantomjs RPM
https://github.com/slipstream/SlipStreamServerDeps/blob/master/phantomjs/pom.xml#L114
Instead it should be part of the RPM.

It should be done here, here and here

Currently using outdated s3curl.

The current nagios backup probe actually does the full backup. This causes a problem because the backup and probe take longer than 60 seconds to complete now. Although the timeout can be extended in the definition of the probe, it cannot exceed the maximum timeout given in the NRPE configuration file on the client. The default for this parameter is 60 seconds. In any case, it is not a good idea to do significant calculations in a probe.

The implementation should be refactored such that a cron job executes the backup and the nagios probe only verifies that the backup has been done recently.