manurodriguez / ansible_advance_homework Goto Github PK

This project forked from redhat-gpte-devopsautomation/ansible_advance_homework

0.0 2.0 0.0 102 KB

Fork The Repo for completing Ansible Advance Homework

ansible_advance_homework's Introduction

Ansible Tower Config

Table 1. List of Playbooks

Files or dir	Purpose
app-tier	Install application server role
db-tier	Install postgressql server for database role
lb-tier	Install HA proxy role
base-config	Setup yum repo and base packages role
setup-workstation	Setup workstation, create network, ssh keypair, security group etc. role
osp-servers	Provision OSP Instances role
osp-instance-delete	Delete OSP Instances role
osp-facts	Genrate in-memory inventory for OSP instances role
roles/config-tower/vars/main.yml	Very important file to review. All the variable values are set there. Please do not make any changes in the file
config-tower	Role to configure ansible tower job templates and workflow
aws_creds.yml	Fetch GUIDkey.pem from bastion of Three tier application env and create machine credential to connect to AWS instances
aws_provision.yml	Use `order_svc.sh` script to provision env
aws_status_check.yml	Check aws instances are up or not
site-3tier-app.yml	Playbook to deploy three tier app
site-install-isolated-node.yml	Playbook to install isolated node
site-config-tower.yml	Playbook to call role `config-tower`
site-osp-delete.yml	Playbook to call role
site-osp-instances.yml	Playbook to call role
site-setup-workstation.yml	Playbook to call role
site-smoke-osp.yml	Playbook to test three tier app on OSP
site-smoketest-aws.yml	Playbook to test three tier app on AWS
grading-script.yml	Self grading script
roles/config-tower/tasks/ec2_dynamic.yml	For creating Dynamic inventory in Ansible tower. Use `AWS Access Key` for credential
roles/config-tower/tasks/job_template.yml	For creating job templates
roles/config-tower/tasks/pre-config-tower.yml	Any pre config tasks needed
roles/config-tower/tasks/workflow_template.yml	genrate workflow from `workflow.yml` file
roles/config-tower/tasks/post-config-tower.yml	any post config jobs

Upload your opentlc ssh private key to /root/.ssh/mykey.pem on the bastion node and set it 400 permissions

[bastion]$ sudo -i
[bastion]# vi /root/.ssh/mykey.pem
[bastion]# chmod 400 /root/.ssh/mykey.pem

From the cloned repo run site-setup-workstation.yml playbook to setup workstation as an isolated node, provide vault pass for repo_vars.yml

[bastion]$ sudo -i
[bastion]# git clone https://github.com/manurodriguez/ansible_advance_homework.git
[bastion]# cd ansible_advance_homework
[bastion]# OSP_GUID=<Openstack for Ansible GUID from mail>
[bastion]# ansible-playbook site-setup-workstation.yml -e OSP_GUID=${OSP_GUID} -e @repo_vars.yml \
--private-key=/root/.ssh/mykey.pem -u <username-company.com> --ask-vault-pass

From the cloned repo run site-config-tower.yml playbook to create job templates and workflow template.

[bastion]$ sudo -i
[bastion]# cd ansible_advance_homework
[bastion]# TOWER_GUID=<Ansible Tower Homework GUID from mail>
[bastion]# OSP_GUID=<Openstack for Ansible GUID from mail>
[bastion]# OPENTLC_LOGIN=<username-company.com>
[bastion]# OPENTLC_PASSWORD=<your openlc account password>
[bastion]# GITHUB_REPO=https://github.com/<githubhandler>/ansible_advance_homework
[bastion]# JQ_REPO_BASE=http://www.opentlc.com/download/ansible_bootcamp
[bastion]# REGION=<enter region name example us-east-1>
[bastion]# RH_MAIL_ID=<your mail id for dynamic inventory tag>
[bastion]# ansible-playbook site-config-tower.yml -e tower_GUID=${TOWER_GUID} \
 -e osp_GUID=${OSP_GUID} -e opentlc_login=${OPENTLC_LOGIN} -e path_to_opentlc_key=/root/.ssh/mykey.pem \
 -e param_repo_base=${JQ_REPO_BASE} -e opentlc_password=${OPENTLC_PASSWORD} -e REGION_NAME=${REGION} \
 -e EMAIL=${RH_MAIL_ID} -e github_repo=${GITHUB_REPO}

From your web browser, connect to tower console using the admin username and known password
Navigate to Templates menu from the side panel and select the launch button of the cicd_workflow template.
Wait for the cicd_workflow to finish and finally grade your work

[bastion]# OSP_GUID=<Openstack GUID>
[bastion]# ANSIBLE_ADVANCED_GUID=<Prod Tier Three App GUID>
[bastion]# ansible-playbook grading-script.yml -e OSP_GUID=${OSP_GUID} -e ANSIBLE_ADVANCE_GUID=${ANSIBLE_ADVANCED_GUID}

ansible_advance_homework's People

Contributors

Watchers

ansible_advance_homework's Issues

Make playbooks consistent

Use a clear and consistent style
Make sure that plays and tasks are self-documenting, with clear and meaningful names
Use templates throughout, making sure they are clearly marked as "Ansible generated"
Use loop labels when possible to avoid printing all details
Do not use deprecated modules like os_server_facts
Use same space between tasks

ansible variables are for vault have unvalid characters

Variables are not found because they contain dashes in variable names

TASK [base-config : enable opentcl repos] **************************************
fatal: [10.10.10.5]: FAILED! => {"msg": "'vault_rhel' is undefined"}

Job template creation fails in tower - playbook not found

Task "Create Homework Assignment project" from roles/config-tower/tasks/post-config-tower.yml needed the git repo specifications, if not playbook roles/config-tower/tasks/job_template.yml will fail:

TASK [config-tower : Job template for OSP Instances] ***********************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["tower-cli", "job_template", "create", "--name", "Provision QA Env", "--job-type", "run", "--inventory", "scm_inventory", "--project"
, "Homework Assignment", "--playbook", "site-osp-instances.yml", "--credential", "Connect_to_workstation"], "delta": "0:00:00.797685", "end": "2020-04-18 21:49:10.145600", "msg": "non-zero r
eturn code", "rc": 40, "start": "2020-04-18 21:49:09.347915", "stderr": "Error: The Tower server claims it was sent a bad request.\n\nPOST https://tower1.XYZ/api/v2/job_
templates/\nParams: None\nData: {\"job_type\": \"run\", \"playbook\": \"site-osp-instances.yml\", \"inventory\": 2, \"credential\": 3, \"name\": \"Provision QA Env\", \"project\": 6}\n\nResp
onse: {\"playbook\":[\"Playbook not found for project.\"]}", ...

Wrong host group name for prod 3-tier-app instances

Playbook aws_status_check.yml doesn't found any instances since it uses wrong host group

PLAY [tag_instance_filter_three_tier_app_psrivast_redhat_com] ******************
skipping: no hosts matched

Job template association with instance group fails

There are multiple misspelled options in the tower-cli tasks making tasks fail[1]:

--job_template it should be --job-template
--instance_group it should be --instance-group

[1]

TASK [config-tower : Associate IG to OSP instances job template] **************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["tower-cli", "job_template", "associate_ig", "--job_template", "Provision QA Env", "--instance_group", "osp"], "delta": "0:00:00.231554", "end": "2020-04-19 01:28:46.793899", "msg": "non-zero return code", "rc": 2, "start": "2020-04-19 01:28:46.562345", "stderr": "Error: no such option: --job_template", "stderr_lines": ["Error: no such option: --job_template"], "stdout": "", "stdout_lines": []}


TASK [config-tower : Associate IG to OSP instances job template] **************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["tower-cli", "job_template", "associate_ig", "--job-template", "Provision QA Env", "--instance_group", "osp"], "delta": "0:00:00.2325
89", "end": "2020-04-19 01:40:47.918044", "msg": "non-zero return code", "rc": 2, "start": "2020-04-19 01:40:47.685455", "stderr": "Error: no such option: --instance_group", "stderr_lines":
["Error: no such option: --instance_group"], "stdout": "", "stdout_lines": []}

remove unused roles or files

After review from documentation, I noticed role osp-setup is not used by any playbook, is a duplicate of role setup-workstation without the pre-tasks to setup the workstation.

Image download task fails due to missing python deps

During the execution using ansible 2.9.6 and openstacksdk 0.46.0, we get the following error:

TASK [setup-workstation : Download RHEL image] *************************************************
fatal: [workstation-b651.rhpds.opentlc.com]: FAILED! => {"changed": false, "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"}

It seems selinux package is missing, we need to install it via pip3

Various issues in the 3-tier app deployment

During the deployment, the following errors occurred:

- No package matching 'httpie' found available, installed or updated
- haproxy template expression hostvars[host] didn't contain any data
- vault vars not found in all roles
- repositories url were printed during execution

Openstack tasks fail due to openstacksdk pkg not found, even if it's installed

During the playbook execution the first task using openstack module fails with the following error:

TASK [setup-workstation : Create m2.small flavor] ***************************************************************
fatal: [workstation-3186.rhpds.opentlc.com]: FAILED! => {"changed": false, "msg": "openstacksdk is required for this module"}

After troubleshooting it seems that the ansible version is not compatible with the openstack module anymore, openstacksdk latest version used was made for python3, these are the versions used:
ansible 2.9.6
openstacksdk (0.46.0)

Since the tasks are performed in the workstation node, we need to make sure to install python3 and pip3 there, and not just python-pip which defaults to python2, then modify the openstack tasks to use python3 as default interpreter.

playbooks fails when creating machine credentials

Tower doesn't seem to like the ansible header and footer from the managed block in the private key:

TASK [config-tower : Machine Credentail to connect to workstation using openstack.pub] **************************************************************************************
[DEPRECATION WARNING]: ssh_key_data should be a string, not a path to a file.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to update credential: The Tower server claims it was sent a bad request.\n\nPOST https://tower1.XYZ/api/
v2/credentials/\nParams: None\nData: {\"inputs\": {\"username\": \"some-user\", \"ssh_key_data\": \"# BEGIN ANSIBLE MANAGED BLOCK\\n-----B

Tower task not using variables

The following task works but it uses hardcoded values in parameters instead of variables:

- name: Tower Creds key file

Implement instance creation and cleanup via openstack module

Two main playbooks are setup to build instances via openstack and delete them, however the actions are not included.

- site-osp-instances.yml
- site-osp-delete.yml

Implement these functionalities, add variables.

Implement smoke test

Playboook site-smoke-osp.yml was provided as a skeleton to write tasks to validate 3-tier app, add the functionalities to validate the content of the web application

Include metadata on the roles

Update the metadata information in the implemented roles

Instance creation fails, ssh public keys not found

SSH public key files are not located in the isolated node where tasks are executed, hence tasks failed to pass these through the userdata during instance creation.

TASK [osp-servers : Create new server instances and attaches them a network and passes metadata to the instance] ***
[WARNING]: Unable to find '/root/.ssh/openstack.pub' in expected paths (use
-vvvvv to see paths)
fatal: [workstation-b651.rhpds.opentlc.com]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /root/.ssh/openstack.pub"}

Setup correct DNS

Openstack instances get a wrong DNS which makes instances to take time to response and pull packages since the resolv.conf loops in the list, then provide correct DNS IPs

- name: Configuration of Subnets for Public and Private SubNets
  os_subnet:
...
    dns_nameservers:
      - 8.8.8.7
      - 8.8.8.8

Provide documentation on implemented roles

Add instructions of how to use, metadata, dependencies, etc in the contributed roles.

privilege scalation timeout on prod 3-tier app plays

When running aws_creds.yml randomly I get errors like this:

Timeout (32s) waiting for privilege escalation prompt:

smoke test fails after curl time out

After site-3tier-app.yml playbook setup the QA 3-tier-app, smoke tests fails since frontend is not responding:

TASK [Curl Frontent website] ***************************************************
failed: [workstation-b651.rhpds.opentlc.com] (item=10.10.10.12) => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "ansible_loop_var": "item", "changed": false, 
"content": "", "elapsed": 30, "item": "10.10.10.12", "msg": "Status code was -1 and not [200]: Connection failure: timed out", "redirected": false, "status": -1, "url": "http://10.10.10.12"}

After looking at the logs, I noticed the tomcat server takes around 10 minutes to fully start, and answer requests:

Apr 20 17:03:40 app1.novalocal systemd[1]: Started Apache Tomcat Web Application Container.
Apr 20 17:03:40 app1.novalocal systemd[1]: Starting Apache Tomcat Web Application Container...
Apr 20 17:03:43 app1.novalocal server[10509]: Java virtual machine used: /usr/lib/jvm/jre/bin/java
Apr 20 17:03:43 app1.novalocal server[10509]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Apr 20 17:03:43 app1.novalocal server[10509]: main class used: org.apache.catalina.startup.Bootstrap
Apr 20 17:03:43 app1.novalocal server[10509]: flags used:
...
Apr 20 17:04:06 app1.novalocal server[10509]: INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.p
Apr 20 17:04:09 app1.novalocal server[10509]: Apr 20, 2020 5:04:09 PM org.apache.coyote.AbstractProtocol init
Apr 20 17:04:09 app1.novalocal server[10509]: INFO: Initializing ProtocolHandler ["http-bio-8080"]
Apr 20 17:04:09 app1.novalocal server[10509]: Apr 20, 2020 5:04:09 PM org.apache.coyote.AbstractProtocol init
Apr 20 17:04:09 app1.novalocal server[10509]: INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Apr 20 17:04:09 app1.novalocal server[10509]: Apr 20, 2020 5:04:09 PM org.apache.catalina.startup.Catalina load
Apr 20 17:04:09 app1.novalocal server[10509]: INFO: Initialization processed in 18513 ms
Apr 20 17:04:10 app1.novalocal server[10509]: Apr 20, 2020 5:04:10 PM org.apache.catalina.core.StandardService startInternal
Apr 20 17:04:10 app1.novalocal server[10509]: INFO: Starting service Catalina
Apr 20 17:04:10 app1.novalocal server[10509]: Apr 20, 2020 5:04:10 PM org.apache.catalina.core.StandardEngine startInternal
Apr 20 17:04:10 app1.novalocal server[10509]: INFO: Starting Servlet Engine: Apache Tomcat/7.0.69
Apr 20 17:04:11 app1.novalocal server[10509]: Apr 20, 2020 5:04:11 PM org.apache.catalina.startup.HostConfig deployDirectory
Apr 20 17:04:11 app1.novalocal server[10509]: INFO: Deploying web application directory /var/lib/tomcat/webapps/ROOT
Apr 20 17:04:25 app1.novalocal server[10509]: Apr 20, 2020 5:04:25 PM org.apache.catalina.startup.TldConfig execute
Apr 20 17:04:25 app1.novalocal server[10509]: INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were sc
Apr 20 17:12:52 app1.novalocal server[10509]: Apr 20, 2020 5:12:52 PM org.apache.catalina.util.SessionIdGeneratorBase createSecureRandom
Apr 20 17:12:52 app1.novalocal server[10509]: INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [505,831] milliseconds.
Apr 20 17:12:53 app1.novalocal server[10509]: Apr 20, 2020 5:12:53 PM org.apache.catalina.startup.HostConfig deployDirectory
Apr 20 17:12:53 app1.novalocal server[10509]: INFO: Deployment of web application directory /var/lib/tomcat/webapps/ROOT has finished in 522,591 ms
Apr 20 17:12:53 app1.novalocal server[10509]: Apr 20, 2020 5:12:53 PM org.apache.coyote.AbstractProtocol start
Apr 20 17:12:53 app1.novalocal server[10509]: INFO: Starting ProtocolHandler ["http-bio-8080"]
Apr 20 17:12:54 app1.novalocal server[10509]: Apr 20, 2020 5:12:54 PM org.apache.coyote.AbstractProtocol start
Apr 20 17:12:54 app1.novalocal server[10509]: INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Apr 20 17:12:54 app1.novalocal server[10509]: Apr 20, 2020 5:12:54 PM org.apache.catalina.startup.Catalina start
Apr 20 17:12:54 app1.novalocal server[10509]: INFO: Server startup in 524175 ms

We need to implement a timeout or wait condition to wait for tomcat to finish.

grade playbook not getting right output from QA 3-tier frontend

When running playbook grading-script.yml the validation fails when checking the QA 3-tier frontend since the IP is not in the variable, playbook uses os_server_facts module and it got deprecated for os_server_info module

Implement smoke test for Prod 3-tier app

Playbook site-smoketest-aws.ymlhas the skeleton and it's imported to tower with the correct credentials, tasks are missing, implement the test.

Missing task to create project on tower

When running ansible-playbook site-config-tower.yml -e ... we obtain the following error:

TASK [config-tower : include_tasks] ****************************************************
fatal: [localhost]: FAILED! => {"reason": "no module/action detected in task.\n\nThe error appears to be in '/root/ansible_advance_homework/roles/config-tower/tasks/post-config-tower.yml': l
ine 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Create Homework Assignment project\n  ^ here\n"}

Cause: Task is missing

- name: Create Homework Assignment project\
# Use tower module or tower-cli to create Ansible Tower project. Name of the project is specifed in ./vars/main.yml

Ansible block lines from keys break tower modules when uploading creds

These keys are provisioned during the deployment of the bastion node in the Homework tower lab, they are not create by this repo, however they break the deployment when ansible tower tasks upload them for credetials

/root/.ssh/openstack.pub
/root/.ssh/openstack.pem

Remove the following lines:

# BEGIN ANSIBLE MANAGED BLOCK
...
# END ANSIBLE MANAGED BLOCK

Clarify a few points in README files to run these playbooks

I deleted and start new Labs to confirm all runs as expected and instructions are correct,
a few minor points are missing:

initial setup of the private key
the vault for repo_vars.yaml
dependency of roles in the 3tier-app

Integrate 3tier app deployment

A playbook site-3tier-app.yml and role was started to deploy a three tier app, tasks, templates and variables are missing. Integrate accordingly to start testing deployment via tower

Job template with unknown parameter machine_credential

aws_creds.yml is using machine_credential parameter to update template jobs, per the documentation it should be credential

"Unsupported parameters for (tower_job_template) module: machine_credential Supported
 parameters include: ask_credential, ask_diff_mode, ask_extra_vars, ask_inventory, ask_job_type,
 ask_limit, ask_skip_tags, ask_tags, ask_verbosity, become_enabled, concurrent_jobs_enabled,
 credential, description, diff_mode_enabled, extra_vars_path, fact_caching_enabled,
 force_handlers_enabled, forks, host_config_key, inventory, job_tags, job_type, limit, name, 
 playbook, project, skip_tags, start_at_task, state, survey_enabled, survey_spec, timeout, 
 tower_config_file, tower_host, tower_password, tower_username, validate_certs, vault_credential,
 verbosity"