flyingcircusio / batou_ext Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 2.0 512 KB

A library of batou extensions.

License: Other

Python 90.90% PHP 3.32% Shell 2.45% C 0.63% Nix 2.70%

batou_ext's People

Contributors

Stargazers

Watchers

Forkers

frlan sweh

batou_ext's Issues

fcio.Provison: Failing when a host is not having componets assigned

Scenario:
One want to provision VM without actually assigning components at the very moment (e.g. on initial bootstrapping of project)

When not having a component-section at all:

 ./appenv run fcio provision --diff staging
Traceback (most recent call last):
  File "mybatou/./.appenv/6e86ce17/bin/fcio", line 33, in <module>
    sys.exit(load_entry_point('batou-ext==0.1.dev0', 'console_scripts', 'fcio')())
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 315, in main
    return args.func(**func_args)
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 309, in <lambda>
    p.set_defaults(func=lambda **kw: Provision(**kw).apply())
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 181, in apply
    self.environment_ = self.load_env()
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou_ext/fcio.py", line 156, in load_env
    environment.load()
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 187, in load
    self.load_hosts(config)
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 289, in load_hosts
    self._load_hosts_multi_section(config)
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 325, in _load_hosts_multi_section
    config[section].as_list("components"))
  File "mybatou/.appenv/6e86ce17/lib/python3.9/site-packages/batou/environment.py", line 44, in as_list
    result = self[option]
KeyError: 'components'

If the section is empty:

./appenv run fcio provision --diff staging
ERROR: Missing component
 Component: 
      Host: myhost01
ERROR: Missing component
 Component: 
      Host: myhost02

batou_ext.rabbitmq.User: Check and update password if needed

batou_ext.rabbitmq.User should check, whether a given password for a user is set and set in case of not.

config.CustomizeYaml: Unsupported usage of yaml.load()

A deployment using CustomizeYaml giving this warning:

/deployment/.batou/3537682d/lib/python3.7/site-packages/batou_ext/config.py:96: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  data = yaml.load(f)

geo.GeoIPDatabase: Make download and extracting step more rebust against failures

With current implementation it might happen that a broken download is corrupting active used one

fcio.Provision: Cannot provision VM without assigned components

% .batou/bin/fcio provision –diff production
 ERROR: Missing component
       Component:
            Host: myhost10

VM configuration:

 [host:myhost10]
 data-cores = 1
 data-ram = 4
 data-disk = 60
 data-description = backend
 components =

Mailhog: Allow Mailhog to store mails outside of the docker container for persitency

With PR #65 we introduced the option to change the general storage engine for Mailhog.
When using maildir, it should be possible to store mails outside the docker container to keep them, even the container is destroyed. However, this will need also a strategy to clean this folder up at some point.

RabbitMQ: ErlangCookie cannot be updated

When changing content of .erlang.cookie btou cannot write the file:

    with open(self.path, "wb") as target:
PermissionError: [Errno 13] Permission denied: '/srv/xxx/.erlang.cookie'

.erlang.cookie is created with -r--------

fcio: Allow to set a RG to maintenance for xx time

Could be something like

./appenv run fcio set-maintenance --duration 1h production

PostgreSQL password must not contain dollar sign

Setting the password for a PostgreSQL user is done via subprocess.Popen with the parameter shell=True. So the call is using a shell which interprets the dollar sign as signal of a requested variable interpolation.

Example:

Using abc$efg as the password for a postgres user
leads to the following command issued: sudo -u postgres psql -d postgres -c "ALTER ROLE \\"username\\" WITH ENCRYPTED PASSWORD 'abc$efg' "
What actually sent to PostgreSQL is: ALTER ROLE "username" WITH ENCRYPTED PASSWORD 'abc' (as there is no variable named efg in the environment.)

This leads to passwords which can be way shorter than expected.
When using in combination with a .pgpass file no log-in is possible because the passwords stored in .pgpass probably does not match with the password stored in PostgreSQL.

`HTPasswd` requires external library `passlib`

The batou_ext.htpasswd module imports passlib.hash.sha512_crypt.

Unfortunately:

passlib is not an install requirement in setup.py
It fails with the newest version from pypi: Exception: No module named 'passlib.hash.sha512_crypt'; 'passlib.hash' is not a package

Importing passlib.hash instead of passlib.hash.sha512_crypt does the trick.

fcio.Provision: Error if service user is missing

With missing service user in environment-section I'm getting this traceback

 % ./batou appenv-run fcio provision production
Traceback (most recent call last):
  File "/my/project/.batou/567b4dd6/bin/fcio", line 33, in <module>
    sys.exit(load_entry_point('batou-ext==0.1.dev0', 'console_scripts', 'fcio')())
  File "/my/project/.batou/567b4dd6/lib/python3.9/site-packages/batou_ext/fcio.py", line 315, in main
    return args.func(**func_args)
  File "/my/project/.batou/567b4dd6/lib/python3.9/site-packages/batou_ext/fcio.py", line 309, in <lambda>
    p.set_defaults(func=lambda **kw: Provision(**kw).apply())
  File "/my/project/.batou/567b4dd6/lib/python3.9/site-packages/batou_ext/fcio.py", line 261, in apply
    pprint(self.api.apply(calls))
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 1116, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 1455, in __request
    request = dumps(params, methodname, encoding=self.__encoding,
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 975, in dumps
    data = m.dumps(params)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 508, in dumps
    dump(v, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 530, in __dump
    f(self, value, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 583, in dump_array
    dump(v, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 530, in __dump
    f(self, value, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 601, in dump_struct
    dump(v, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 530, in __dump
    f(self, value, write)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xmlrpc/client.py", line 534, in dump_nil
    raise TypeError("cannot marshal None unless allow_none is enabled")
TypeError: cannot marshal None unless allow_none is enabled

SSL `CertificateCheck` broken with Batou >= 2.1.

flyingcircusio/batou#65 prevents from setting unknown attributes to components.

When enabling a certificate check on a let's encrypt certificate this leads to the instantiation of CertificateCheck which calls ServiceCheck with a name kw-argument which is not defined on Service. This breaks now a customer deployment.

batou_ext/src/batou_ext/ssl.py

Lines 245 to 254 in c13a461

    
           self += batou.lib.nagios.ServiceCheck( 
        
               self.expand( 
        
                   'https://{{component.public_name}} certificate valid?'), 
        
               name=self.expand('ssl_cert_{{component.public_name}}'), 
        
               command='check_http', 
        
               args=self.expand( 
        
                   '-H {{component.public_name}} ' 
        
                   '-p {{component.port}} ' 
        
                   '-S --sni ' 
        
                   '-C {{component.warning_days}},{{component.critical_days}}'))

fcio.Provision: Allow to provision VM inside different RG

When having a parent RG, it's possible to use the same API credentials to provision VM in child rg.
The provision method should be capable of handling such a scenario

batou.ssl.Certificate: Ensure alternative_names are given as tuple (not a string only)

When using ssl.Certifcate() to get an Lets Encrypt certificate it might happen alternative_names is not set correctly to a tuple of strings (list of domains).

Given, that at this example self.alternative_name is a string with one FQDN only …

Good call:

    self.cert = batou_ext.ssl.Certificate(
          ....
          alternative_names=(self.alternative_name,),
          ....
      )

Wrong call:

    self.cert = batou_ext.ssl.Certificate(
          ....
          alternative_names=(self.alternative_name),
          ....
      )

As the default of alternative_names is an empty tupel, this error might only occur on specific configurations (like e.g. production deployment) during runtime of the deployment itself.

We should ensure that alternative_names is a really a tuple of strings.

Document meaning of `checkWarning` and `checkCritical` of the CronJob component

The current docstring is not really helpful for non-involved users of the library.

fcio.Provision: --diff is handling wrong VM name not correct

In cases of there is a typo in host: the -–diff subcommand is not handling the error correct.

Here I had I typo within VM name so i didn't belonged to RG anymore.

% ./batou appenv-run fcio provision –diff staging                              
Running unclean installation from requirements.txt
Ensuring unclean install ...
Traceback (most recent call last):
  File ".batou/unclean/bin/fcio", line 11, in <module>
    load_entry_point('batou-ext==0.1.dev0', 'console_scripts', 'fcio')()
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 309, in main
    return args.func(**func_args)
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 303, in <lambda>
    p.set_defaults(func=lambda **kw: Provision(**kw).apply())
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 230, in apply
    diff = self.get_diff(self.get_currently_provisioned_vms(), vms)
  File "/my/deployment/folder/.batou/unclean/lib/python3.7/site-packages/batou_ext/fcio.py", line 173, in get_currently_provisioned_vms
    return self.api.query('virtualmachine')
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1170, in single_request
    return self.parse_response(resp)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 1342, in parse_response
    return u.close()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault -32500: 'application error'>

Consider a better versioning schema and using actual releases.

The upgrade to batou 2.3 is a bit confusing/painful with respect to batou_ext.

I think we should consider adding real releases and a proper versioning scheme.

I suggest that we:

use the same minor version as batou when releasing for specific batou versions e.g. batou_ext 2.3.X is compatible with batou 2.3.X
make a batou dependency so that for batou_ext==x.y.z it requires batou<=x.y+1

Does that sound reasonable?

nix.PurgePackage: Add documentation

There is no documentation for PurgePackage available.

mail.Mailhog: Change on configuration does not restart service

When changing configuration of mailhog -- e.g. to make usage of feature introduced with 5432201 -- the service is not restarted automaticity

batou_ext.cron.CronJob: Explicit log if an timeout occurs

It would be nice to make it more visible whether a CronJob run into a timeout.

From timout manpage:

If the command times out, and --preserve-status is not set, then
exit with status 124.  Otherwise, exit with the status of
COMMAND.  If no signal is specified, send the TERM signal upon
timeout.  The TERM signal kills any process that does not block
or catch that signal.  It may be necessary to use the KILL (9)
signal, since this signal cannot be caught, in which case the
exit status is 128+9 rather than 124.

document transition from self signed to valid certificate

After deploying a new site for the first time, a self signed cert is created.

Please document

that a self signed cert is created in the first place (yes, it is documented in the source code )
how to manually trigger the generation of the LE cert (ie goto deployment/work/nginx/.../ and execute the bash script)
document when - without manual intervention - a new cert is created ( -> cronjob )

ScanHost.update breaks as ssh-keyscan changed its API

ssh-keyscan now returns 1 (previously 0) when no keys are found

cf openssh/openssh-portable@c2c18a3

This broke my build on Travis when I updated to focal (Ubuntu 20.04).

Ubuntu 18.04

jugmac00@odin:~$ ssh-keyscan -6 github.com
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
jugmac00@odin:~$ echo $?
0

Ubuntu 20.04

jugmac00@heimdall:~$ ssh-keyscan -6 github.com
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
getaddrinfo github.com: No address associated with hostname
jugmac00@heimdall:~$ echo $?
1

P.S.: This is certainly no Ubuntu issue, I just used those distros as they provide ssh-keyscan versions before and after the above linked API change.

git.Checkout(): Don't run SSH Scanhost when git url is https://

Was cloning a repo from github via it's https://-address:

testvm > Backend > GitCheckout > ScanHost('github.com') > File('/srv/test/.ssh') > Directory('.ssh')
testvm > Backend > GitCheckout > ScanHost('github.com') > File('/srv/test/.ssh') > Mode('.ssh')
testvm > Backend > GitCheckout > ScanHost('github.com')

I think we should not perform the ScanHost-part if there is a https-URL present

S3: Better error handling when not able to connect to s3 gateway

Something like this is not helpful:

oks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "/srv/s-gitlab/deployment/.appenv/e2e91147/lib/python3.10/site-packages/botocore/hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "/srv/s-gitlab/deployment/.appenv/e2e91147/lib/python3.10/site-packages/botocore/handlers.py", line 279, in validate_bucket_name
    if not VALID_BUCKET.search(bucket) and not VALID_S3_ARN.search(bucket):
TypeError: expected string or bytes-like object

Postgres.Extension: Predict with missing database not working

Code:
self += batou_ext.postgres.Extension('postgis', db=self.dbname)

Results in:

xxx00 > DBBackend > Extension('postgis')
ERROR: sudo -u postgres psql -d dbname -qtAX -c "SELECT extname FROM pg_extension WHERE extname = 'postgis';"
Return code: 2
STDOUT

STDERR
psql: FATAL:  database "dbname" does not exist

if database 'dbname' does not yet exists (e.g. on bootstrapping)

nix.rebuild - defer rebuild to the end

Could be useful deferred to the end, @dpausp experiences half failed builds sometimes

Cronjob name clash not detected

Given two cron jobs in the same component:

        self += batou_ext.cron.CronJob(
            "maintenance",
            command="command a", …)
        self += batou_ext.cron.CronJob(
            "maintenance",
            command="command b', …)

They both have the same name, hence the script will be overwritten. I think this should be prevented somehow.

fcio.Provision: Implicit creation of service user is not setting its description

Setting a description for a service user is now enforced. The implicit creation via provisioning is not setting it.

redis.Redis: Allow to configure providing name

Usecase:

Using multiple instances of Redis inside a project

RabbitMQ: ErlangCookie cannot handle prediction mode

If there is no .erlang.cookie-file already created, the ErlangCookie crases

ERROR: Traceback (most recent call last):
  File "/my.batou/.appenv/80e88ae5/lib/python3.10/site-packages/batou/remote_core.py", line 402, in <module>
  File "/my.batou/.appenv/80e88ae5/lib/python3.10/site-packages/batou/remote_core.py", line 366, in deploy
  File "/my.batou/.appenv/80e88ae5/lib/python3.10/site-packages/batou/remote_core.py", line 178, in deploy
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou/component.py", line 327, in deploy
    sub_component.deploy(predict_only)
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou/component.py", line 339, in deploy
    call_with_optional_args(
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou/utils.py", line 524, in call_with_optional_args
    return func(**call_kw)
  File "/remote/deployment/.appenv/33f09e0d/lib/python3.9/site-packages/batou_ext/rabbitmq.py", line 43, in verify
    with open(self.path, 'r') as target:
FileNotFoundError: [Errno 2] No such file or directory: '/remote/.erlang.cookie'

batou_ext.cron.SystemdTimer: Allow to supply multiple onCalendar expressions

Systemd allows to supply timers with multiple values for OnCalendar. This will trigger the service every time any timer matches. The SystemdTimer class does, however, only supports supplying a single string value.

NixOS seems to be able to handle multiple timer units. Local testing suggests that supplying a list should be able to generate a valid systemd timer:

systemd.timers.test_timer = {
    # …
    timerConfig.OnCalendar = ["*-*-01 01:00:00" "*-*-2 02:00:00"];
}

# …

[Timer]
OnCalendar=*-*-01 01:00:00
OnCalendar=*-*-02 02:00:00

An alternative would be to create multiple timers/services doing the same thing. But as each timer
is marked as Persistent, my understanding is that such jobs could be triggered multiple times after a system failure.

RabbitMQ: Permissions fails in prediction mode without propper user

ERROR: rabbitmqctl -q list_user_permissions myuser
    Return code: 69
STDOUT

STDERR
Error:
{:no_such_user, "myuser"}

git.GitCheckout: Allow to use clobber=True

In most cases we can assume that the git checkout used for git.GitCheckout should be clean. Therefore setting clobber=True got batou.lib.git.Clone() might prevent unwanted surprises.

s3.Download: Print URL tried for downloading in case of an failure

in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
ERROR: Unexpected exception

is not very helpful when debugging an error.

batou_ext.nix.UserEnv: Add a way to deinstall a UserEnv

Installing a package like this

        self += batou_ext.nix.UserEnv(
                "django",
                packages=[
                    "gcc",
                     … 
                    "yarn",
                    "zip",
                ],
                shellInit="# additional shell init")

will cause a package like django-1.bb2a973726635812db18475991c37f382495943257f0d77faaf80f0f4a7cce4b can be found via nix-env -q. To ensure it's deinstalled later you have to use the exact name at batou_ext.nix.PurgePackage().

It would be nice if there would be an easier way to do so.

fcio.Provision: Roles are not sorted correctly -> changes not correct detected

When running fcio provision --diff <environment> the list of assigned roles is in some cases not correct sorted. This is causing a message like this:

myvm00
    classes             : ['role::docker', 'role::external_net', 'role::webgateway'] → ['role::docker', 'role::webgateway', 'role::external_net']

Even though there is actually no role update needed.

ssh.SSHKeyPair: Allow to set name of key

Currently the component can only either deploy id_ed25519(.pub) or id_rsa(.pub). For deploying multiple SSH-keys it would be nice if this could be configurable (optional)

ssh.SSHKeyPair: Deletes local SSH keys when using local as deployment target

        # ED25519
        if self.id_ed25519:
            self += File(
                "~/.ssh/id_ed25519",
                content="{}\n".format(self.id_ed25519),
                mode=0o600,
                sensitive_data=True,
            )

	self += batou.lib.nagios.ServiceCheck(
	self.expand(
	'https://{{component.public_name}} certificate valid?'),
	name=self.expand('ssl_cert_{{component.public_name}}'),
	command='check_http',
	args=self.expand(
	'-H {{component.public_name}} '
	'-p {{component.port}} '
	'-S --sni '
	'-C {{component.warning_days}},{{component.critical_days}}'))