When deleting a cluster in AWS, it does not free up resources such as volumes, elastic IPs, etc. This needs to be fixed. This is most likely taken care of if/when moving to using AWS CloudFormation for provisioning, but if this doesn't happen, the delete role should ensure to free up all resources that were created for a cluster.

DNS / Route53 is another component that also needs to be cleaned up

@redhat-cop/casl

To allow for additional clarity for provisioning cluster, provide documentation into the inventory variables that are available for provisioning.

Currently, wildcard routes are not resolvable from within the private network. This makes invocations from within the OpenShift environments to routes impossible

As part of the task to check if a passed in template is a url or local file, we should use failed_when to avoid printing red failure messages, or even consider something like method: HEAD with the url module (as described here.

The variable used for dns domain in the OSP inventory example has different value in all.yml file and OSEv3.yml file:

https://github.com/redhat-cop/casl-ansible/blob/master/inventory/sample.osp.example.com.d/inventory/group_vars/all.yml#L52

https://github.com/redhat-cop/casl-ansible/blob/master/inventory/sample.osp.example.com.d/inventory/group_vars/OSEv3.yml#L27-L29

Currently we name heat stacks with env_id, like l1/d1/a1 etc. However, env_id often repeats across cluster_ids. This prevents me from standing up an l1 cluster in two different dns_domains.

Since we no longer deploy / manage a DNS server as part of the deployment, the sample inventory should be scrubbed for DNS info:

https://github.com/redhat-cop/casl-ansible/blob/master/inventory/sample.osp.example.com.d/inventory/hosts

For example, right now the DNS server is using the same flavor as everything else - this often means using a lot of resources for something that requires very little.

add the ability to have different vm sizes for master, infranodes and nodes.

The oc-apply role right now is very much undocumented - i.e.: we need a detailed description of the inventory and its dictionary definition.

Product direction is to steer away from individual clouds' provisioning/templating mechanisms. As such, we should replace our heat-based server provisioning with using the native ansible os_server, os_volume, etc.

latest 3.6.1 release does not https://github.com/redhat-cop/casl-ansible/blob/v3.6.1/roles/openshift-applier/tasks/main.yml#L11

Just like we have an image for OSP provisioning, we should have one for AWS as well.

Steps to reproduce:

• oc logout
• run the role

Output:

TASK [openshift-applier : Create/Apply 'ci-cd.yml'] ********************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "cmd": "oc create -f /home/jholmes/Code/cluster-contents/inventory/../namespaces/ci-cd.yml", "failed": true, "msg": "[Errno 2] No such file or directory", "rc": 2}

Expected Output:
something that tells me I'm not logged into oc

Ansible Version:

ansible 2.3.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Sep  5 2017, 08:53:59) [GCC 7.1.1 20170622 (Red Hat 7.1.1-3)]

The use of is defined in a conditional is not valid, as null variables still cause failure. we need to instead do some kind of boolean to decided whether tasks in a role should be run.

http://docs.ansible.com/ansible/playbooks_conditionals.html#applying-when-to-roles-and-includes

Something like:
when: users_to_create | default({}) | length >= 1

With the introduction of using heat for provisioning (PR #2), node labels aren't working correctly still and needs to be worked. This issue is to track this work.

Right now, due to some subtle inconsistencies with how ansible variables are loaded, we have to pass the location of the openshift-ansible codebase via a -e in the ansible command. We are not able to write the value in an inventory file.

the condition should be change to test the number of both masters and infranodes, also consider just always create the load balancer.

If I add or remote yum repos from the list, our role does not recognize that there are changes to be made. We currently do the enabling of repos using commands rather than modules because the modules don't support it yet.

There is an open issue in ansible-core-modules about adding it. ansible/ansible-modules-core#3300

The AWS instances (as most public cloud providers do) are using hostnames tied to their IP/region, etc. -i.e.: ip-172-31-8-56.us-west-2.compute.internal ... this isn't ideal when it comes to naming the hosts within OpenShift. Just like we do for OpenStack, it would be preferred to have the hostnames reflect the role of the instance - i.e.: master, infra-node, app-node, etc.

For example:

1. Deploy SonarQube - Takes a while because of provisioning and build
2. Deploy Jenkins - Requires a running instance of SonarQube in order to set up authentication token

Can we, and if so how, address these "order of operation" issues?

We need to add this to our automation, since it will not be added to the out of the box templates.

See https://bugzilla.redhat.com/show_bug.cgi?id=1503260

this will facilitate testing CNS.
the attached disk name should be predictable so that it can be used in the CNS topology configuration

The AWS provisioning needs to be enhanced and the dynamic hosts file needs to be removed as it doesn't work well + updating an inventory by automation isn't quite good.

When objects are created like:

TASK [openshift-applier : Create OpenShift objects based on static files for 'namespace'] *************************************************************************************************************************
changed: [master-0.sample.casl.example.com] => {"changed": true, "cmd": ["oc", "create", "-f", "https://raw.githubusercontent.com/redhat-cop/cluster-lifecycle/inception-poc/namespaces/myapp-dev.yml"], "delta": "0:00:00.215057", "end": "2017-10-05 22:52:22.301717", "failed": false, "failed_when_result": false, "rc": 1, "start": "2017-10-05 22:52:22.086660", "stderr": "Error from server (AlreadyExists): namespaces \"myapp-dev\" already exists\nError from server (AlreadyExists): rolebinding \"system:image-pullers\" already exists\nError from server (AlreadyExists): rolebinding \"system:image-builders\" already exists\nError from server (AlreadyExists): rolebinding \"system:deployers\" already exists\nError from server (AlreadyExists): rolebinding \"admin\" already exists", "stderr_lines": ["Error from server (AlreadyExists): namespaces \"myapp-dev\" already exists", "Error from server (AlreadyExists): rolebinding \"system:image-pullers\" already exists", "Error from server (AlreadyExists): rolebinding \"system:image-builders\" already exists", "Error from server (AlreadyExists): rolebinding \"system:deployers\" already exists", "Error from server (AlreadyExists): rolebinding \"admin\" already exists"], "stdout": "", "stdout_lines": []}

The task reports as changed, when no changes actually took place

It is expected that the provisioning sets up wildcard record(s) for the provisioned environment, e.g.: *.apps.example.com ==> infra node(s)

See https://github.com/redhat-cop/casl-ansible/blob/master/roles/openshift-applier/tasks/process-one-entry.yml#L8.

At the beginning of an inventory, there will usually be projectrequests whic need file_action = create. Unless file_action is reset, subsequent files (e.g. policies) will use create, which is unexpected behavior.

If we can set the default to create instead of omit I think that would be good.

The delete-cluster playbook should prompt the user for a confirmation on delete to avoid accidental deletions.

Per OpenShift docs [1], the following key/value needs to be set per cluster (and they need to be unique):

Key=kubernetes.io/cluster/xxxx,Value=clusterid where xxxx and clusterid are unique per cluster

[1] : https://docs.openshift.com/container-platform/3.6/install_config/persistent_storage/dynamically_provisioning_pvs.html#available-dynamically-provisioned-plug-ins

The openshift-applier should error out if the template entry doesn't point to a file - i.e: a directory. The file entry support directories, hence need to ensure that a file entry doesn't accidentally get changed to template and still pointing to a directory.

Update the repository README to reference the correct repository and properly callout the contents

It should be possible to source templates with the corresponding params file from a directory. This basically means that a well known structure of needs to be established. An example can be found here:
https://github.com/redhat-cop/cluster-lifecycle

I think this is actually a contrib issue now, but need to look into it further.

Failure summary:

  1. Host:     localhost
     Play:     localhost
     Task:     openshift-ansible-contrib/roles/dns-records : Add public master cluster hostname records to the private A records (single master)
     Message:  The conditional check 'hostvars[groups.masters[0]].openshift_master_cluster_public_hostname is defined' failed. The error was: error while evaluating conditional (hostvars[groups.masters[0]].openshift_master_cluster_public_hostname is defined): list object has no element 0
               
               The error appears to have been in '/tmp/src/casl-ansible/roles/openshift-ansible-contrib/roles/dns-records/tasks/main.yml': line 12, column 3, but may
               be elsewhere in the file depending on the exact syntax problem.
               
               The offending line appears to be:
               
               
               - name: "Add public master cluster hostname records to the private A records (single master)"
                 ^ here

the NFS role should allow to create an NFS server that then is configured by the openshift-ansible playbook

Instead of building a file for static DNS records, we should be using nsupdate (the dns-server role already supports it) to add records. I.e.: change these places within the code:

https://github.com/redhat-cop/casl-ansible/blob/master/playbooks/openshift/pre-install.yml#L17-L24
https://github.com/redhat-cop/casl-ansible/blob/master/playbooks/openshift/pre-install.yml#L32-L34
https://github.com/redhat-cop/casl-ansible/blob/master/playbooks/openshift/dns_records.yml

In addition to listing files/templates within the inventory, oc-apply should also handle processing a directory of files.

@redhat-cop/casl

this has been widely discussed, creating an issue to track it. we are waiting on ansible 2.5 to be released, which merged ansible/ansible#33386

oc-apply needs to check for the existence of files (local or remote/url) and error out with a proper error message. This should also be applied to a check for the NAMESPACE parameter to ensure it is set in the parms file.

Firstly, we could add support for file_action: delete in all cases.

Secondly, I would like to be able to have a way to globally set file_action: delete on the entire role for a run, as a way to clean up a previous run.

lb, dns and nfs (if it will be supported) could be all merged into a single vm, saving some resources.

When using oc-applier to create persistent databases with generate credentials, subsequent runs of the inventory will overwrite the secrets stored in OpenShift but have no effect on the database itself. I propose added an optional field in the inventory so that a particular template is applied once and only once on a given cluster. For example:

- object: someobjects
  content:
  - name: ephemeraldb
    template: openshift//mysql-ephemeral
    namespace: my-project
    params: "{{ inventory_dir }}/../params/ephemeraldb/build_params"
  - name: persistentdb
    template: openshift//mysql-persistent
    namespace: my-project
    params: "{{ inventory_dir }}/../params/persistentdb/build_params"
    idempotent: true

Inside of of this project, we create a copy of roles/oc-apply/tasks/process-content.yml and name it roles/oc-apply/tasks/process-idempotent.yml. This task will be modified to ONLY run when idempotent is defined and the original will be modified to ONLY run when idempotent is undefined. The new task will use oc create along with oc process instead of using oc apply. The new task would also swallow errors related to the objects already existing.

For example, we may not always want to re-run certain segments of the inventory on every run, so a way to perhaps set a flag to only run certain objects within the inventory would be ideal.

@sherl0cks and @InfoSec812 have both been experiencing issues around this.

this is done in /etc/sysconfig/docker-storage-setup, for example:

STORAGE_DRIVER=overlay2
DEVS=/dev/sdb
CONTAINER_ROOT_LV_NAME=dockerlv
CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
VG=dockervg

The second time running through an openshift-applier role on a cluster, ansible fails, citing "Project already exists" error message. Need to figure out how to handle this.

@oybed @day4skiing