ansible-community / molecule-hetznercloud Goto Github PK

View Code? Open in Web Editor NEW

25.0 13.0 14.0 325 KB

A molecule driver for Hetzner Cloud

Home Page: https://ansible.readthedocs.io/projects/molecule/

License: GNU Lesser General Public License v3.0

Python 96.87% Makefile 3.13%

molecule ansible hetzner hetzner-cloud molecule-driver

molecule-hetznercloud's Introduction

Molecule Hetzner Cloud Driver

A Hetzner Cloud driver for Molecule.

This plugin allows you to use on-demand Hetzner Cloud servers for your molecule integration tests.

Install

$ pip install molecule-hetznercloud

Upgrade

This project adheres to Semantic Versioning.

$ pip install --upgrade molecule-hetznercloud

Upgrading to v2

In your molecule.yml files, rename the molecule driver name (driver.name) to molecule_hetznercloud:

 driver:
-  name: hetznercloud
+  name: molecule_hetznercloud

In your molecule.yml files, the volumes name (platforms[].volumes[].name) field is now required. If the field is missing, you MUST add it:

 platforms:
   - name: instance-1
     image: debian-12
     volumes:
-     - size: 20
+     - name: volume-1
+       size: 20

Each resource (servers, volumes, networks) name are prefixed with a hash (32 chars) based on the role and scenario path. This means you MAY reuse the same names (e.g. instance-1) across your scenarios. Resource names MUST not exceed their max length, for example the server name max length is 63 chars, with the prefix you only have 31 chars left for your name.
In your molecule.yml files, the platforms server type (platforms[].server_type) field now defaults to cx11. If you already use the default, you MAY remove the field:

 platforms:
   - name: instance-1
     image: debian-12
-    server_type: cx11

Usage

To communicate with the Hetzner Cloud API, you need to expose a HCLOUD_TOKEN environment variable. Find out more about how to get a Hetzner Cloud API token in the authentication documentation.

$ export HCLOUD_TOKEN="set_the_hcloud_token_here"

Then setup a new molecule scenario using the driver plugin.

$ molecule init scenario --driver-name molecule_hetznercloud

Warning

With molecule 6, the molecule init scenario command dropped support for driver provided configuration. If you are using molecule >=6, please copy the example below and paste it in your scenario molecule.yml file. See this commit for details.

Your molecule/default/molecule.yml should then look like the following.

---
driver:
  name: molecule_hetznercloud
platforms:
  - # Name of the Server to create (must be unique per Project and a valid hostname as per RFC 1123).
    # required
    name: instance-1
    # Name of the Image the Server is created from.
    # required
    image: debian-12
    # Name of the Server type this Server should be created with.
    # default: cx11
    server_type: cx11
    # Name of Location to create Server in (must not be used together with datacenter).
    # default: omit
    location: hel1
    # Name of Datacenter to create Server in (must not be used together with location).
    # default: omit
    datacenter: null
    # Cloud-Init user data to use during Server creation. This field is limited to 32KiB.
    # default: omit
    user_data: null

    # List of volumes to attach to the server.
    volumes:
      - # Name of the volume.
        # required
        name: volume-1
        # Size of the Volume in GB.
        # default: 10
        size: 10

    # Dictionary of private networks the server should be attached to.
    networks:
      # Name of the network
      network-1:
        # IP range of the whole network which must span all included subnets. Must be one of the private IPv4 ranges of RFC1918.
        # If multiple hosts using the same network, you may only define it once.
        # required
        ip_range: 10.0.0.0/16
        subnet:
          # IP to assign to the server.
          # required
          ip: 10.0.0.1/24
          # Type of subnetwork.
          # default: cloud
          type: cloud
          # Name of network zone.
          # default: eu-central
          network_zone: eu-central
      network-2:
        ip_range: 10.1.0.0/16
        subnet:
          ip: 10.1.0.1/24

Note

The networks.ip_range is important for creating. If you have multiple hosts, you may only define it once.

Note

You may list the server types and available images using the hcloud command line tool:

# List server types
$ hcloud server-type list --sort name
# List images for the x86 architecture
$ hcloud image list --type system --architecture x86 --sort name

Then test your role.

$ molecule test

To ease initial debugging for getting things started, also expose the following environment variables.

$ export MOLECULE_NO_LOG=False  # not so verbose, helpful
$ export MOLECULE_DEBUG=True  # very verbose, last ditch effort

You may also define a custom resource namespace by exposing the following environment variables, for example in CI workflows:

$ export RESOURCE_NAMESPACE=e121dc64ff615ccdfac71bb5c00296b9 # Ensure the value length is <= 32

Development

Testing

Run unit tests:

make test

Run integration tests

export HCLOUD_TOKEN="set_the_hcloud_token_here"
make integration

History

The project was initially maintained by @decentral1se. After a long period looking for new maintainers, the project was archived in early 2023.

In September 2023, the code has been rewritten by @jooola and the project was reactivated to continue development.

License

The LGPLv3 license.

molecule-hetznercloud's People

Contributors

Stargazers

Watchers

Forkers

ssbarnea cfra xlab-steampunk hetznercloud veselahouba konradasb tumbl3w33d de-it-krachten rakhithjk steinbrueckri ohdearaugustin seikexyz ciumabok

molecule-hetznercloud's Issues

Consolidate configuration into pyproject.toml

Finally the maintenance experience we need...

Add possibility to create and attach volumes

Sometimes it might be useful to create and attach a volume to the test VM e.g. in case you want to test a storage preparation role for LVM setups.

Support converge.yml embedding on core side

Follow from ansible/molecule#2675 since it got closed 🙄

https://github.com/ansible-community/molecule-hetznercloud#only-use-moleculeyml-for-configuration

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

ansible-galaxy

requirements.yml

github-actions

.github/workflows/release-please.yml

google-github-actions/release-please-action v4

.github/workflows/release.yml

actions/checkout v4

actions/setup-python v5

actions/upload-artifact v4

actions/download-artifact v4

pypa/gh-action-pypi-publish v1.8.14

.github/workflows/stale.yml

.github/workflows/test.yml

actions/checkout v4

actions/setup-python v5

codecov/codecov-action v4

actions/checkout v4

actions/setup-python v5

codecov/codecov-action v4

pep621

pyproject.toml

pip_setup

setup.py

molecule >=5.0.0,<24.3

python-dateutil >=2.7.5

requests >=2.20

tox >=4.11.3,<5.0

pytest-xdist >=3.3.1,<4.0

pytest >=7.4.0,<8.0

pytest-ansible >=4.1.0,<5.0

pytest-cov >=5,<5.1

pre-commit

.pre-commit-config.yaml

pre-commit/pre-commit-hooks v4.6.0

pre-commit/mirrors-prettier v3.1.0

asottile/pyupgrade v3.15.2

pycqa/isort 5.13.2

psf/black-pre-commit-mirror 24.4.2

pycqa/flake8 7.0.0

Check this box to trigger a request for Renovate to run again on this repository

Instance name length limit

When creating a hcloud VM you are limited in length when choosing a name. Usually there's enough space when just manually giving names via platforms definitions, however, when you run your scenarios in --parallel then molecule adds a UUID to the instance name. This happens here (util.py around line 329):

def _parallelize_platforms(config, run_uuid):
    def parallelize(platform):
        platform["name"] = "{}-{}".format(platform["name"], run_uuid)
        return platform

    return [parallelize(platform) for platform in config["platforms"]]

So I was thinking whether I should ask for this in the molecule repo, but maybe you have an idea how we could override this behavior in the driver to limit the name to whatever the hcloud API accepts. Shortening to the first part of the UUID would probably do the trick in most cases.

The actual error:

TASK [Wait for instance(s) creation to complete] *******************************
FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).
failed: [localhost] (item={'started': 1, 'finished': 0, 'ansible_job_id': '856426607467.349', 'results_file': '/home/jenkins/agent/workspace/myproject_master@2/.ansible_async/856426607467.349', 'changed': True, 'failed': False, 'item': {'image': 'debian-10', 'name': 'master-73-firewalld-blocked-7e316d97-359d-4d32-8c83-04d2123b39e9', 'server_type': 'cx11'}, 'ansible_loop_var': 'item'}) => {"ansible_job_id": "856426607467.349", "ansible_loop_var": "item", "attempts": 2, "changed": false, "finished": 1, "item": {"ansible_job_id": "856426607467.349", "ansible_loop_var": "item", "changed": true, "failed": false, "finished": 0, "item": {"image": "debian-10", "name": "master-73-firewalld-blocked-7e316d97-359d-4d32-8c83-04d2123b39e9", "server_type": "cx11"}, "results_file": "/home/jenkins/agent/workspace/myproject_master@2/.ansible_async/856426607467.349", "started": 1}, "msg": "invalid input in field 'name'"}

My platforms definition:

platforms:
  - name: "${VM_PREFIX:-$USER}-firewalld-blocked"
    server_type: cx11
    image: debian-10

💖 looking for hetznercloud driver maintainers

I am unable to maintain this driver as I am not using it, so I am looking for maintainers.

I can keep providing support for integration with molecule itself but not around vagrant bits.

I would work noting that the top priority would be to setup CI jobs.

VM name prefix not matching inventory_hostname causes vars not to be loaded

Hello,

TL;DR:

molecule knows instances by their name defined in the platforms section
molecule-hetznercloud prefixes instances (by default a dynamic string, computed by md5 of the scenario path)
if you use an additional inventory that works with the hcloud API, you must be aware the instances names in hcloud won't match the host names in your play, i.e. host_vars also won't match
my workaround is to define a static prefix (RESOURCE_NAMESPACE) and remove that prefix on-the-fly from the instance names in my dynamic inventory

Long story:

I define my instances in the platforms attribute like this:

platforms:
  - name: "${$USER}-app-01"
    groups:
      - my_servers
 […]

and in my converge.yml I target the hosts like

- name: Converge
  hosts: my_servers

This basically works. However, I also link my dynamic inventory

provisioner:
[…]
  inventory:
    links:
      hosts: "../../../../inventories/hcloud_inventory.py"

which picks up my hosts and sets some hostvars each. Now the problem is, my tests run against the group's hosts like tumble-app-01, but in the background the hcloud driver prefixes VMs with a resource namespace (with fallback to a md5 of the scenario path which effectively is the value, because I didn't set that env myself).

The result is that my dynamic inventory knows a host like <md5>-tumble-app-01 which is not the same as the one used in the molecule run (tumble-app-01) which in turn means that my hostvars are not loaded.

I added a workaround to my dynamic inventory to remove that prefix from the host, when a certain env var is set:

    # for molecule-based tests running in hcloud
    if "VM_PREFIX_TO_REMOVE" in os.environ:
        vm_name = vm_name.replace(os.environ["VM_PREFIX_TO_REMOVE"], "", 1)

That's not pretty, but I could live with it. However, I struggle to dynamically set that env for the run:

provisioner:
[…]
 env:
   ANSIBLE_ROLES_PATH: "../../../../roles:../../../../common-roles"
   INV_HCLOUD_TOKEN: "${HCLOUD_TOKEN}"
   #VM_PREFIX_TO_REMOVE: "{{ '${MOLECULE_SCENARIO_DIRECTORY}' | md5 }}-" # this does not work, no interpolation
   VM_PREFIX_TO_REMOVE: 19fa9d8fc0b584b2b5e2704a69929e2d-

The statically set prefix leads to my inv removing that part of the host name and my play picks up the vars as expected. What I can do is setting RESOURCE_NAMESPACE there, so I know the static string that I need to remove from the host names.

It's all a little awkward. I guess there's no way for you to influence the host names molecule works with. I tried targeting the hosts directly instead of the group like

  hosts:
    - "{{ lookup('env', 'MOLECULE_SCENARIO_DIRECTORY') | md5 }}-app-01"
    - "{{ lookup('env', 'MOLECULE_SCENARIO_DIRECTORY') | md5 }}-app-02"

and that basically works, however, it's somehow out of sync with the molecule universe so it doesn't know how to reach these hosts and the play fails with unreachable (permission denied).

I guess I will continue with the static prefix for now as that does the trick for me due to the workaround in my inventory. Feel free to close the issue if you think that's the best we can do.

Drop all the test fixture parametrizing

Will need to pass back and drop all that cruft from Molecule core testing.

We're only testing one driver now.

Wait for instance(s) creation to complete FAILED #bug

on deploy with molecule

this task failed

it seems happen when user testinfra verifier

molecule 3.4.0 using python 3.9 
    ansible:2.11.3
    delegated:3.4.0 from molecule
    docker:0.2.4 from molecule_docker
    hetznercloud:1.3.0 from molecule_hetznercloud
    vagrant:0.6.3 from molecule_vagrant

molecule.yml
    dependency:
      name: galaxy
    driver:
      name: hetznercloud
    platforms:
      - name: "s2"
        server_type: cx11
        image: debian-10
    provisioner:
      name: ansible
    verifier:
      name: testinfra

TASK [Wait for instance(s) creation to complete] *******************************
FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (299 retries left).
FAILED - RETRYING: Wait for instance(s) creation to complete (298 retries left).
failed: [localhost] (item={'started': 1, 'finished': 0, 'ansible_job_id': '563148533006.16193', 'results_file': '/home/spham/.ansible_async/563148533006.16193', 'changed': True, 'failed': False, 'item': {'image': 'debian-10', 'name': 's2', 'server_type': 'cx11'}, 'ansible_loop_var': 'item'}) => {"ansible_job_id": "563148533006.16193", "ansible_loop_var": "item", "attempts": 4, "changed": false, "cmd": "/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py", "data": "", "finished": 1, "item": {"ansible_job_id": "563148533006.16193", "ansible_loop_var": "item", "changed": true, "failed": false, "finished": 0, "item": {"image": "debian-10", "name": "s2", "server_type": "cx11"}, "results_file": "/home/spham/.ansible_async/563148533006.16193", "started": 1}, "msg": "Traceback (most recent call last):\n  File \"/tmp/ansible_ansible.legacy.async_wrapper_payload_o6_p3ip2/ansible_ansible.legacy.async_wrapper_payload.zip/ansible/modules/async_wrapper.py\", line 162, in _run_module\n  File \"/tmp/ansible_ansible.legacy.async_wrapper_payload_o6_p3ip2/ansible_ansible.legacy.async_wrapper_payload.zip/ansible/modules/async_wrapper.py\", line 90, in _filter_non_json_lines\nValueError: No start of json char found\n", "stderr": "Traceback (most recent call last):\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 699, in urlopen\n    httplib_response = self._make_request(\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 445, in _make_request\n    six.raise_from(e, None)\n  File \"<string>\", line 3, in raise_from\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 440, in _make_request\n    httplib_response = conn.getresponse()\n  File \"/usr/local/lib/python3.9/http/client.py\", line 1347, in getresponse\n    response.begin()\n  File \"/usr/local/lib/python3.9/http/client.py\", line 307, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/local/lib/python3.9/http/client.py\", line 268, in _read_status\n    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n  File \"/usr/local/lib/python3.9/socket.py\", line 704, in readinto\n    return self._sock.recv_into(b)\n  File \"/usr/local/lib/python3.9/ssl.py\", line 1241, in recv_into\n    return self.read(nbytes, buffer)\n  File \"/usr/local/lib/python3.9/ssl.py\", line 1099, in read\n    return self._sslobj.read(len, buffer)\nConnectionResetError: [Errno 104] Connection reset by peer\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/spham/.env/lib/python3.9/site-packages/requests/adapters.py\", line 439, in send\n    resp = conn.urlopen(\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 755, in urlopen\n    retries = retries.increment(\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 532, in increment\n    raise six.reraise(type(error), error, _stacktrace)\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/packages/six.py\", line 769, in reraise\n    raise value.with_traceback(tb)\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 699, in urlopen\n    httplib_response = self._make_request(\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 445, in _make_request\n    six.raise_from(e, None)\n  File \"<string>\", line 3, in raise_from\n  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 440, in _make_request\n    httplib_response = conn.getresponse()\n  File \"/usr/local/lib/python3.9/http/client.py\", line 1347, in getresponse\n    response.begin()\n  File \"/usr/local/lib/python3.9/http/client.py\", line 307, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/local/lib/python3.9/http/client.py\", line 268, in _read_status\n    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n  File \"/usr/local/lib/python3.9/socket.py\", line 704, in readinto\n    return self._sock.recv_into(b)\n  File \"/usr/local/lib/python3.9/ssl.py\", line 1241, in recv_into\n    return self.read(nbytes, buffer)\n  File \"/usr/local/lib/python3.9/ssl.py\", line 1099, in read\n    return self._sslobj.read(len, buffer)\nurllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 366, in _create_server\n  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/actions/client.py\", line 21, in wait_until_finished\n    self.reload()\n  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/core/client.py\", line 125, in reload\n    bound_model = self._client.get_by_id(self.data_model.id)\n  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/actions/client.py\", line 42, in get_by_id\n    response = self._client.request(\n  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/hcloud.py\", line 219, in request\n    response = self._requests_session.request(\n  File \"/home/spham/.env/lib/python3.9/site-packages/requests/sessions.py\", line 542, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/home/spham/.env/lib/python3.9/site-packages/requests/sessions.py\", line 655, in send\n    r = adapter.send(request, **kwargs)\n  File \"/home/spham/.env/lib/python3.9/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py\", line 100, in <module>\n    _ansiballz_main()\n  File \"/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py\", line 92, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.hetzner.hcloud.plugins.modules.hcloud_server', init_globals=dict(_module_fqn='ansible_collections.hetzner.hcloud.plugins.modules.hcloud_server', _modlib_path=modlib_path),\n  File \"/usr/local/lib/python3.9/runpy.py\", line 210, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/local/lib/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/local/lib/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 654, in <module>\n  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 635, in main\n  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 577, in present_server\n  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 386, in _create_server\nAttributeError: 'ConnectionError' object has no attribute 'message'\n", "stderr_lines": ["Traceback (most recent call last):", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 699, in urlopen", "    httplib_response = self._make_request(", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 445, in _make_request", "    six.raise_from(e, None)", "  File \"<string>\", line 3, in raise_from", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 440, in _make_request", "    httplib_response = conn.getresponse()", "  File \"/usr/local/lib/python3.9/http/client.py\", line 1347, in getresponse", "    response.begin()", "  File \"/usr/local/lib/python3.9/http/client.py\", line 307, in begin", "    version, status, reason = self._read_status()", "  File \"/usr/local/lib/python3.9/http/client.py\", line 268, in _read_status", "    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")", "  File \"/usr/local/lib/python3.9/socket.py\", line 704, in readinto", "    return self._sock.recv_into(b)", "  File \"/usr/local/lib/python3.9/ssl.py\", line 1241, in recv_into", "    return self.read(nbytes, buffer)", "  File \"/usr/local/lib/python3.9/ssl.py\", line 1099, in read", "    return self._sslobj.read(len, buffer)", "ConnectionResetError: [Errno 104] Connection reset by peer", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/home/spham/.env/lib/python3.9/site-packages/requests/adapters.py\", line 439, in send", "    resp = conn.urlopen(", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 755, in urlopen", "    retries = retries.increment(", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/util/retry.py\", line 532, in increment", "    raise six.reraise(type(error), error, _stacktrace)", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/packages/six.py\", line 769, in reraise", "    raise value.with_traceback(tb)", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 699, in urlopen", "    httplib_response = self._make_request(", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 445, in _make_request", "    six.raise_from(e, None)", "  File \"<string>\", line 3, in raise_from", "  File \"/home/spham/.env/lib/python3.9/site-packages/urllib3/connectionpool.py\", line 440, in _make_request", "    httplib_response = conn.getresponse()", "  File \"/usr/local/lib/python3.9/http/client.py\", line 1347, in getresponse", "    response.begin()", "  File \"/usr/local/lib/python3.9/http/client.py\", line 307, in begin", "    version, status, reason = self._read_status()", "  File \"/usr/local/lib/python3.9/http/client.py\", line 268, in _read_status", "    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")", "  File \"/usr/local/lib/python3.9/socket.py\", line 704, in readinto", "    return self._sock.recv_into(b)", "  File \"/usr/local/lib/python3.9/ssl.py\", line 1241, in recv_into", "    return self.read(nbytes, buffer)", "  File \"/usr/local/lib/python3.9/ssl.py\", line 1099, in read", "    return self._sslobj.read(len, buffer)", "urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 366, in _create_server", "  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/actions/client.py\", line 21, in wait_until_finished", "    self.reload()", "  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/core/client.py\", line 125, in reload", "    bound_model = self._client.get_by_id(self.data_model.id)", "  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/actions/client.py\", line 42, in get_by_id", "    response = self._client.request(", "  File \"/home/spham/.env/lib/python3.9/site-packages/hcloud/hcloud.py\", line 219, in request", "    response = self._requests_session.request(", "  File \"/home/spham/.env/lib/python3.9/site-packages/requests/sessions.py\", line 542, in request", "    resp = self.send(prep, **send_kwargs)", "  File \"/home/spham/.env/lib/python3.9/site-packages/requests/sessions.py\", line 655, in send", "    r = adapter.send(request, **kwargs)", "  File \"/home/spham/.env/lib/python3.9/site-packages/requests/adapters.py\", line 498, in send", "    raise ConnectionError(err, request=request)", "requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py\", line 100, in <module>", "    _ansiballz_main()", "  File \"/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py\", line 92, in _ansiballz_main", "    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)", "  File \"/home/spham/.ansible/tmp/ansible-tmp-1637197060.3134565-16174-162800436823256/AnsiballZ_hcloud_server.py\", line 40, in invoke_module", "    runpy.run_module(mod_name='ansible_collections.hetzner.hcloud.plugins.modules.hcloud_server', init_globals=dict(_module_fqn='ansible_collections.hetzner.hcloud.plugins.modules.hcloud_server', _modlib_path=modlib_path),", "  File \"/usr/local/lib/python3.9/runpy.py\", line 210, in run_module", "    return _run_module_code(code, init_globals, run_name, mod_spec)", "  File \"/usr/local/lib/python3.9/runpy.py\", line 97, in _run_module_code", "    _run_code(code, mod_globals, init_globals,", "  File \"/usr/local/lib/python3.9/runpy.py\", line 87, in _run_code", "    exec(code, run_globals)", "  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 654, in <module>", "  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 635, in main", "  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 577, in present_server", "  File \"/tmp/ansible_hcloud_server_payload_cb98r1vw/ansible_hcloud_server_payload.zip/ansible_collections/hetzner/hcloud/plugins/modules/hcloud_server.py\", line 386, in _create_server", "AttributeError: 'ConnectionError' object has no attribute 'message'"]}

other issue, it not remove ssh key on destroy

Destroy not remove server

Hi
When make molecule test.
It not remove server at the end.

And when make molecule destroy too

Configure PyPI package publishing

The CI is already configured to build the package and to publish it to PyPI on releases.

The only missing secrets are the following:

PYPI_USERNAME
PYPI_PASSWORD

We are missing the credentials or permissions to push the molecule-hetznercloud package on PyPI: https://pypi.org/project/molecule-hetznercloud/

@cidrblock Could not give access to the PyPI package, we might need to ask someone else: @ssbarnea ?

Ideally, we could add permissions to upload the package to the following user: https://pypi.org/user/HetznerCloudGmbH/

Once this issue is solved, I can cut a new release.

Creating mulitple volumes sometimes fails

When I do the following:

volumes:
  - name: "molecule-hetznercloud-volume-1-${INSTANCE_UUID}"
  - name: "molecule-hetznercloud-volume-2-${INSTANCE_UUID}"

It sometimes fails with some sort of JSON related error which I don't have right now. It is flaky.

This could be the underlying library or the play implementation. Unsure.

/cc @xoxys

Migrate away from Molecule specific directory structure

In all this copying/pasta'ing, I am following the Molecule directory structure. That is mostly for tests. It doesn't really make sense now that the plugin code will live outside of Molecule core. So, we should move away from that when we get a bit further. Creating this to track.

SSH Key Type

More modern OS are starting to deprecate the rsa key algorithms for ssh (Archlinux,...)

Therefore it would be nice to make the generated key type configurable, so more modern key types like ed25519.

All supported types can be found here. (https://docs.ansible.com/ansible/latest/collections/community/crypto/openssh_keypair_module.html#parameter-type)

I would for now implement it as a environment variable.

PR: OTW

Support latest molecule (3.2.1)

The ecosystem is moving. See ansible/molecule#2917.

Setup functional testing

Need to chat to Hetzner Cloud world to get the CI running over here.

Setup CI/CD that integrates with GH

@webknjaz has kindly offered to wire this up. Opening this to track.

Discussion in #36.

Enable Drone CI for PRs also

See #20.

Volume clean up does not delete attached volumes

See ansible-collections/hetzner.hcloud#47.

/cc @xoxys

molecule_to_yaml filter still present when role init

Filter molecule_to_yaml have been replaced with ansible native some time ago (#20) but somehow when I init new role I still get the "old ones"

$ pip freeze |grep molecule
molecule==3.4.0
molecule-hetznercloud==1.3.0

$ molecule init role myrolename -d hetznercloud
INFO     Initializing new role myrolename...
No config file found; using defaults
- Role myrolename was created successfully
INFO     Initialized role in /tmp/ansible-role-crowdsec/myrolename successfully.

$ grep from_yaml myrolename/* -R
myrolename/molecule/default/create.yml:      with_items: "{{ lookup('file', molecule_instance_config) | molecule_from_yaml }}"
myrolename/molecule/default/destroy.yml:            instance_conf: "{{ lookup('file', molecule_instance_config) | molecule_from_yaml }}"

which leads to errors like

TASK [Dump instance config] ******************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: No filter named 'molecule_header'.. String: {{ instance_conf | to_json | from_json | molecule_to_yaml | molecule_header }}"}

Debugging molecule runs which use this driver

We run into issues time and time again where tests don't complete because instances either fail to start at all or e.g. molecule (Ansible) fails find SSH on the instance:

https://github.com/pycontribs/molecule-hetznercloud/blob/e3d14e0e37587d490096b4efde5eb2d4ea55608a/molecule_hetznercloud/cookiecutter/scenario/driver/hetznercloud/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D/create.yml#L81-L87

... or later on, fails to connect to the running instance via SSH.

We starting using pipelining, which helped some. But generally, it's really hard to debug this.

I suppose some of this is a platform issue. I am unable to debug this though, as we run 6 (or so) scenarios in each PR/merge build and it's literally another "scenario" every time. Their support has been saying "no issues" even when their status feed scrolls by with a multiple issues on different "cloud nodes".

I also noticed this:
https://docs.hetzner.cloud/#overview-rate-limiting

Is this elevated somehow? Or is this something that would need to be added to hcloud_server?

Was hoping someone had insights into how this could be made more transparent.

pypi issue tracker point to private tracker instance

I discovered that the issue tracker listed on https://pypi.org/project/molecule-hetznercloud/ points to https://git.autonomic.zone/autonomic-cooperative/molecule-hetznercloud/issues where I want not able to login or to create an account.

Please address this issue as molecule-hetznercloud is causing some problems with molecule packaging.

CRITICAL Failed to find driver molecule_hetznercloud. Please ensure that the driver is correctly installed.

I have problems using the module with my molecule setup. Seems more related to my pip but I am clueless what is wrong. Maybe someone can tell me whats wrong.

I installed the plugin with pip install --upgrade molecule-hetznercloud and also pip3 install --upgrade molecule-hetznercloud to be sure, but when I run molecule test I get the following errors.

Failed to load driver entry point Traceback (most recent call last):
  File "/Users/jakoberpf/Library/Python/3.11/lib/python/site-packages/molecule/api.py", line 53, in drivers
    pm.load_setuptools_entrypoints("molecule.driver")
  File "/opt/homebrew/lib/python3.11/site-packages/pluggy/_manager.py", line 288, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/opt/homebrew/lib/python3.11/site-packages/pluggy/_manager.py", line 91, in register
    raise ValueError(
ValueError: Plugin already registered: delegated=<class 'molecule.driver.delegated.Delegated'>
{'default': <class 'molecule.driver.delegated.Delegated'>, 'azure': <class 'molecule_plugins.azure.driver.Azure'>, 'containers': <class 'molecule_plugins.containers.driver.Container'>, 'docker': <class 'molecule_plugins.docker.driver.Docker'>, 'ec2': <class 'molecule_plugins.ec2.driver.EC2'>, 'gce': <class 'molecule_plugins.gce.driver.GCE'>, 'podman': <class 'molecule_plugins.podman.driver.Podman'>, 'vagrant': <class 'molecule_plugins.vagrant.driver.Vagrant'>}

ERROR    Failed to load driver entry point Traceback (most recent call last):
  File "/Users/jakoberpf/Library/Python/3.11/lib/python/site-packages/molecule/api.py", line 53, in drivers
    pm.load_setuptools_entrypoints("molecule.driver")
  File "/opt/homebrew/lib/python3.11/site-packages/pluggy/_manager.py", line 288, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/opt/homebrew/lib/python3.11/site-packages/pluggy/_manager.py", line 91, in register
    raise ValueError(
ValueError: Plugin already registered: delegated=<class 'molecule.driver.delegated.Delegated'>
{'default': <class 'molecule.driver.delegated.Delegated'>, 'azure': <class 'molecule_plugins.azure.driver.Azure'>, 'containers': <class 'molecule_plugins.containers.driver.Container'>, 'docker': <class 'molecule_plugins.docker.driver.Docker'>, 'ec2': <class 'molecule_plugins.ec2.driver.EC2'>, 'gce': <class 'molecule_plugins.gce.driver.GCE'>, 'podman': <class 'molecule_plugins.podman.driver.Podman'>, 'vagrant': <class 'molecule_plugins.vagrant.driver.Vagrant'>}

CRITICAL Failed to find driver molecule_hetznercloud. Please ensure that the driver is correctly installed.

From the molecule --version command I am guessing that the driver is not correctly installed but am clueless what the issue is. I would be very thankfull about any hints.

molecule 6.0.2 using python 3.11 
    ansible:2.15.8
    azure:23.5.0 from molecule_plugins
    containers:23.5.0 from molecule_plugins requiring collections: ansible.posix>=1.3.0 community.docker>=1.9.1 containers.podman>=1.8.1
    default:6.0.2 from molecule
    docker:23.5.0 from molecule_plugins requiring collections: community.docker>=3.0.2 ansible.posix>=1.4.0
    ec2:23.5.0 from molecule_plugins
    gce:23.5.0 from molecule_plugins requiring collections: google.cloud>=1.0.2 community.crypto>=1.8.0
    podman:23.5.0 from molecule_plugins requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0
    vagrant:23.5.0 from molecule_plugins

💖 Project needs maintainers

That is a call for finding some people that can maintain this project, ensuring that there is a CI/CD pipeline running and that the project does work with latest versions of molecule.

Correctly handle ANSIBLE_FILTER_PLUGINS for plugin vendored filter functions

I've just started using this and was confused by some errors the generated default files emit.

The first one was as edge case that I, unfortunately, cannot reproduce anymore right now. It wouldn't hurt to handle it gracefully anyway, I guess

$ molecule test

# in destroy.yml

TASK [Populate instance config from file] **************************************
[WARNING]: Unable to find
'/home/myuser/.cache/molecule/myrole/default/instance_config.yml' in
expected paths (use -vvvvv to see paths)
fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /home/myuser/.cache/molecule/myrole/default/instance_config.yml"}

TASK [Populate instance config when file missing] ******************************
ok: [localhost]

It's not a big deal after all, because this happens in a rescue block, but it confused me at first until I found that out. I'd suggest to add a failed_when that catches this expected condition. Something like:

    - name: Populate the instance config
      block:
        - name: Populate instance config from file
          set_fact:
            instance_conf: "{{ lookup('file', molecule_instance_config) | from_yaml }}"
            skip_instances: false
          register: instance_config_lookup
          failed_when: instance_config_lookup is not success and 'could not locate file in lookup' not in instance_config_lookup.stderr

A second thing that caused problems was the generated create.yml as it uses the filter molecule_get_hetznercloud_networks that are not in the filter_path and there's no explanation in the documentation how to fix that.

TASK [Create private network(s)] ***********************************************
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: No filter named 'molecule_get_hetznercloud_networks'.. String: {{ molecule_yml.platforms|molecule_get_hetznercloud_networks('networks') }}"}

I could try pointing that to the virtualenv path but it's not really stable as the python version changes. When you rename/delete the create.yml then the "implicit" one is executed and it finds its filter. Copying this filter to the role does not seem like a maintainable solution to me either.

Pending removal of driver unless a maintainer step in

Few minutes ago I observed that this driver does not have any CI/CD pipeline configured and thus I will soon remove it from the list of drivers as molecule cores are not able to maintain CI for this driver.

I will manually merge the hotfix this time but if nobody steps in to takeover maintenance and setup CI pipelines for it that report on pull requests, I will mark it abandoned and remove from molecule own testing pipeline (eco job).

I hope someone steps in.

Revamp test suite to drop all old molecule core conventions

It's really showing now at this point when this plugin code has been out of molecule core for a while and there are a bunch of plugins and and conventions that don't reflect what is going on here in this plugin now. So, in short, I need to refactor/clean up the test suite here.

hetznercloud plugin causing requirment conflicts

Molecule packaging (the container building) reports:

ERROR: hcloud 1.7.0 has requirement requests<2.23,>=2.20, but you'll have requests 2.23.0 which is incompatible.

At this moment this conflict is just a warning but unless is is fixed soon I will be forced to remove molecule-hetzercloud from the list of plugins we bundle via https://github.com/ansible-community/molecule/blob/master/Dockerfile#L70