cloudera.exe -- an Ansible collection enabling runlevel management of CDP Public Cloud deployments as well as numerous utilities for deployments.
License: Apache License 2.0
The 3.x.x release of community.aws has removed the module, ec2_elb_info.
auto-repo-mirror/task/populate_from_upstream.yml
• name: Setup System Rhel8
Fails when Cloudera-Deploy container is CentOS Stream 8 and parcel_distro: el8 cat /etc/os-release NAME="CentOS Stream" VERSION="8" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Stream 8" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://centos.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
As per Minimal setup for cloud storage, the IDBROKER_ROLE requires two policies:
Current cloudera-deploy AWS setup here shows that the IDBroker_role only has the IDBroker Assume Role Policy and is lacking the CDP Log policy hence failing validations in cdpctl, specifically describing the actions under the aws-cdp-log-policy.
Receiving:
"msg": "Couldn't create policy XYZ-logs-policy: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::XYZ-ABC-DEFG\"."Previously, DW activation did not require worker subnets for public load balancing, but the endpoint now enforcing this behavior.
The data role is able to set up read-only/read-write access to external AWS S3 buckets for IDBroker and Datalake Admin role. However, an additional step is required for DWX -- the same policies need to be assigned to the NodeInstanceRole within the DW cluster. See the bottom of the following best practices document: https://community.cloudera.com/t5/Community-Articles/External-AWS-Bucket-Access-in-CDP-Public-Cloud/ta-p/302074
Currently, only pull from GH, not local project files.
Apparently a describe-datalake operation can return a transient HTTP 500 right after a delete-datalake (at least in an Azure environment).
This can cause a teardown to fail. I manually ran a describe-datalake after the failure and it was working fine, showing the datalake with status EXTERNAL_DATABASE_DELETION_IN_PROGRESS.
The module could possibly tolerate a few 500's.
TASK [cloudera.exe.platform : Tear down CDP Datalake aa-az-dl] ********************************************************************************************************************************************************************************************************************************************************************************************************************************
Friday 25 February 2022 05:05:35 +0000 (0:00:00.125) 0:02:41.320 *******
fatal: [localhost]: FAILED! => {"changed": false, "error": "{'base_error': ClientError('An error occurred: An internal error has occurred. Retry your request, but if the problem persists, contact us with details by posting a message on the Cloudera Community forums. (Status Code: 500; Error Code: UNKNOWN; Service: datalake; Operation: describeDatalake; Request ID: 2677014e-890a-4034-8bf0-194672891ad9;)'), 'ext_traceback': [' File \"/root/.ansible/tmp/ansible-tmp-1645765535.2918718-3248-198179142058362/AnsiballZ_datalake.py\", line 102, in <module>\\n _ansiballz_main()\\n', ' File \"/root/.ansible/tmp/ansible-tmp-1645765535.2918718-3248-198179142058362/AnsiballZ_datalake.py\", line 94, in _ansiballz_main\\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\\n', ' File \"/root/.ansible/tmp/ansible-tmp-1645765535.2918718-3248-198179142058362/AnsiballZ_datalake.py\", line 40, in invoke_module\\n runpy.run_module(mod_name=\\'ansible_collections.cloudera.cloud.plugins.modules.datalake\\', init_globals=None, run_name=\\'__main__\\', alter_sys=True)\\n', ' File \"/usr/lib64/python3.8/runpy.py\", line 207, in run_module\\n return _run_module_code(code, init_globals, run_name, mod_spec)\\n', ' File \"/usr/lib64/python3.8/runpy.py\", line 97, in _run_module_code\\n _run_code(code, mod_globals, init_globals,\\n', ' File \"/usr/lib64/python3.8/runpy.py\", line 87, in _run_code\\n exec(code, run_globals)\\n', ' File \"/tmp/ansible_cloudera.cloud.datalake_payload_0nryc88o/ansible_cloudera.cloud.datalake_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datalake.py\", line 621, in <module>\\n', ' File \"/tmp/ansible_cloudera.cloud.datalake_payload_0nryc88o/ansible_cloudera.cloud.datalake_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datalake.py\", line 611, in main\\n', ' File \"/tmp/ansible_cloudera.cloud.datalake_payload_0nryc88o/ansible_cloudera.cloud.datalake_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datalake.py\", line 402, in __init__\\n', ' File \"/tmp/ansible_cloudera.cloud.datalake_payload_0nryc88o/ansible_cloudera.cloud.datalake_payload.zip/ansible_collections/cloudera/cloud/plugins/module_utils/cdp_common.py\", line 42, in _impl\\n result = f(self, *args, **kwargs)\\n', ' File \"/tmp/ansible_cloudera.cloud.datalake_payload_0nryc88o/ansible_cloudera.cloud.datalake_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datalake.py\", line 469, in process\\n', ' File \"/tmp/ansible_cloudera.cloud.datalake_payload_0nryc88o/ansible_cloudera.cloud.datalake_payload.zip/ansible_collections/cloudera/cloud/plugins/modules/datalake.py\", line 520, in delete_datalake\\n', ' File \"/usr/local/lib/python3.8/site-packages/cdpy/common.py\", line 452, in wait_for_state\\n current = describe_func(**params)\\n', ' File \"/usr/local/lib/python3.8/site-packages/cdpy/datalake.py\", line 24, in describe_datalake\\n return self.sdk.call(\\n', ' File \"/usr/local/lib/python3.8/site-packages/cdpy/common.py\", line 559, in call\\n parsed_err = CdpError(err)\\n'], 'error_code': 'UNKNOWN', 'violations': 'An internal error has occurred. Retry your request, but if the problem persists, contact us with details by posting a message on the Cloudera Community forums.', 'message': 'Client request error', 'status_code': '500', 'rc': 1, 'service': 'datalake', 'operation': 'describeDatalake', 'request_id': '2677014e-890a-4034-8bf0-194672891ad9'}", "msg": "Client request error", "violations": "An internal error has occurred. Retry your request, but if the problem persists, contact us with details by posting a message on the Cloudera Community forums."}
The PR workflows are broken at the moment -- unable to download collections from Ansible Galaxy. Seems to be related to the pinned version of ansible-core.
setup_aws_networks.yml expects a dictionary of ad-hoc tags to update the VPC, but fails if no tags are defined.
if run__dw_dbc_configs is empty, run__dw_vw_configs is never set in Construct CDP DW Data Catalog configurations, which causes Check CDP DW Virtual Warehouse tags to fail.
TASK [cloudera.exe.runtime : Check CDP DW Virtual Warehouse tags] ************************************************************************************************************************************************************
task path: /opt/cldr-runner/collections/ansible_collections/cloudera/exe/roles/runtime/tasks/initialize_base.yml:229
Friday 23 December 2022 09:11:49 +0000 (0:00:00.035) 0:01:47.025 *******
The conditional check '__dw_vw_config.tags | length > 0' failed. The error was: error while evaluating conditional (__dw_vw_config.tags | length > 0): '__dw_vw_config' is undefined
The error appears to be in '/opt/cldr-runner/collections/ansible_collections/cloudera/exe/roles/runtime/tasks/initialize_base.yml': line 229, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Check CDP DW Virtual Warehouse tags
^ here
fatal: [localhost]: FAILED! => {
"msg": "'run__dw_vw_configs' is undefined"
}
The data role duplicates the listing of AWS Policy documents for Cloudbreak, and is incorrect.
The policy documents URL should be moved to common and referred to in both other roles.
During teardown of a CDP environment on GCP, get an error when "Tear down Operational GCP Service Accounts Policies" in the platform role.
Failing Ansible Task: cloudera.exe.platform : Tear down Operational GCP Service Accounts Policies. Code link below:
Error message (snippet):
{
"changed": true,
"stdout": "",
"stderr": "ERROR: (gcloud.projects.remove-iam-policy-binding) Policy bindings with the specified principal and role not found!",
"rc": 1,
"cmd": [
"gcloud",
"projects",
"remove-iam-policy-binding",
"<GCP_PROJECT>",
"--member=serviceAccount:jegcpt-logs-identity@<GCP_PROJECT>.iam.gserviceaccount.com",
"--role=projects/<GCP_PROJECT>/roles/jegcpt_logs_role",
"--all"
],
"start": "2022-02-24 16:50:01.181110",
"end": "2022-02-24 16:50:03.351317",
"delta": "0:00:02.170207",
"msg": "non-zero return code",
"invocation": {
"module_args": {
"_raw_params": "gcloud projects remove-iam-policy-binding <GCP_PROJECT> --member=serviceAccount:jegcpt-logs-identity@<GCP_PROJECT>.iam.gserviceaccount.com --role=projects/<GCP_PROJECT>/roles/jegcpt_logs_role --all\n",
"warn": true,
"_uses_shell": false,
"stdin_add_newline": true,
"strip_empty_ends": true,
"argv": null,
"chdir": null,
"executable": null,
"creates": null,
"removes": null,
"stdin": null
}
},
"stdout_lines": [],
"stderr_lines": [
"ERROR: (gcloud.projects.remove-iam-policy-binding) Policy bindings with the specified principal and role not found!"
],
"_ansible_no_log": false,
"failed_when_result": true,
"__gcp_binding_item": {
"member": "serviceAccount:jegcpt-logs-identity@<GCP_PROJECT>.iam.gserviceaccount.com",
"role": "projects/<GCP_PROJECT>/roles/jegcpt_logs_role"
},
"ansible_loop_var": "__gcp_binding_item",
"_ansible_item_label": "__gcp_binding_item.member"
}
This tasks attempts to remove GCP IAM role & policy bindings from the their associated service accounts. However these service accounts have already been removed in a previous task (L54-L66 Tear down Operational GCP Service Accounts)
I believe this is just an ordering problem - iam-policy-bindings should be removed before deleting the Service Accounts so that they can still be referenced in the gcloud projects remove-iam-policy-binding command.
The task which removes GCP Service Account policies task which uses a gcloud command now fails when the Service Account does not exist. There is a failed_when condition on this task which should prevent this happending:
cloudera.exe/roles/platform/tasks/teardown_gcp_authz.yml
Lines 54 to 62 in c211d37
This task uses the gcloud projects remove-iam-policy-binding command and it seems the the error message for a non-existent SA has changed slightly:
cldr full-v1.5.4 #> gcloud projects remove-iam-policy-binding <GCP_ACCOUNT> --member=serviceAccount:jenright-audit-identity@<GCP_ACCOUNT>.iam.gserviceaccount.com --role=roles/storage.objectAdmin --all
ERROR: (gcloud.projects.remove-iam-policy-binding) Policy bindings with the specified principal and role not found!
Change the failed_when condition on the Tear down Operational GCP Service Accounts Policies to catch the new error message.
With the use of -t teardown and no purge: yes key in the definition file, the utility VM is removed anyway.
Regardless of the infra_region configured in the profile, an Azure deployment always show:
TASK [cloudera.exe.infrastructure : Check Azure Region - westeurope]
Several of the AWS IAM policies published under the public cloudbreak repository now have "${ARN_PARTITION}" replacement prompts included in their Resource references. For example,
"Resource": "arn:aws:s3:::${DATALAKE_BUCKET}"
has been replaced with
"Resource": "arn:${ARN_PARTITION}:s3:::${DATALAKE_BUCKET}"
(I believe this change is related to work required to support GovCloud. )
As a result, environment deployment attempts have started failing with errors like the following:
TASK [cloudera.exe.platform : Create CDP Data Access Policies] *************************************************************************************************************************************************************
Wednesday 11 May 2022 22:53:56 +0000 (0:00:01.692) 0:05:02.625 *********
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.MalformedPolicyDocumentException: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition "${ARN_PARTITION}" is not valid for resource "arn:${ARN_PARTITION}:s3:::sup-rp-uet2".
failed: [localhost] (item=sup-rp-logs-policy) => {"__aws_policy_details_item": {"description": "CDP Log Location Access", "key": "log", "name": "sup-rp-logs-policy"}, "ansible_loop_var": "__aws_policy_details_item", "boto3_version": "1.21.40", "botocore_version": "1.24.40", "changed": false, "error": {"code": "MalformedPolicyDocument", "message": "Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2\".", "type": "Sender"}, "msg": "Couldn't create policy sup-rp-logs-policy: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2\".", "response_metadata": {"http_headers": {"connection": "close", "content-length": "370", "content-type": "text/xml", "date": "Wed, 11 May 2022 22:54:05 GMT", "x-amzn-requestid": "6d5d6b50-3917-479b-92c6-81618857a0ea"}, "http_status_code": 400, "request_id": "6d5d6b50-3917-479b-92c6-81618857a0ea", "retry_attempts": 0}}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.MalformedPolicyDocumentException: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition "${ARN_PARTITION}" is not valid for resource "arn:${ARN_PARTITION}:s3:::sup-rp-uet2/ranger/audit/*".
failed: [localhost] (item=sup-rp-audit-policy) => {"__aws_policy_details_item": {"description": "CDP Ranger Audit S3 Access", "key": "ranger_audit_s3", "name": "sup-rp-audit-policy"}, "ansible_loop_var": "__aws_policy_details_item", "boto3_version": "1.21.40", "botocore_version": "1.24.40", "changed": false, "error": {"code": "MalformedPolicyDocument", "message": "Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2/ranger/audit/*\".", "type": "Sender"}, "msg": "Couldn't create policy sup-rp-audit-policy: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2/ranger/audit/*\".", "response_metadata": {"http_headers": {"connection": "close", "content-length": "385", "content-type": "text/xml", "date": "Wed, 11 May 2022 22:54:12 GMT", "x-amzn-requestid": "483671df-ad62-48b8-9c53-bcb7a5c44714"}, "http_status_code": 400, "request_id": "483671df-ad62-48b8-9c53-bcb7a5c44714", "retry_attempts": 0}}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.MalformedPolicyDocumentException: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition "${ARN_PARTITION}" is not valid for resource "arn:${ARN_PARTITION}:s3:::sup-rp-uet2".
failed: [localhost] (item=sup-rp-dladmin-policy) => {"__aws_policy_details_item": {"description": "CDP Datalake Admin S3 Access", "key": "datalake_admin_s3", "name": "sup-rp-dladmin-policy"}, "ansible_loop_var": "__aws_policy_details_item", "boto3_version": "1.21.40", "botocore_version": "1.24.40", "changed": false, "error": {"code": "MalformedPolicyDocument", "message": "Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2\".", "type": "Sender"}, "msg": "Couldn't create policy sup-rp-dladmin-policy: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2\".", "response_metadata": {"http_headers": {"connection": "close", "content-length": "370", "content-type": "text/xml", "date": "Wed, 11 May 2022 22:54:20 GMT", "x-amzn-requestid": "06e9eb59-961e-4645-a0fa-3367f4b5132d"}, "http_status_code": 400, "request_id": "06e9eb59-961e-4645-a0fa-3367f4b5132d", "retry_attempts": 0}}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.MalformedPolicyDocumentException: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition "${ARN_PARTITION}" is not valid for resource "arn:${ARN_PARTITION}:s3:::sup-rp-uet2".
failed: [localhost] (item=sup-rp-storage-policy) => {"__aws_policy_details_item": {"description": "CDP Bucket S3 Access", "key": "bucket_access", "name": "sup-rp-storage-policy"}, "ansible_loop_var": "__aws_policy_details_item", "boto3_version": "1.21.40", "botocore_version": "1.24.40", "changed": false, "error": {"code": "MalformedPolicyDocument", "message": "Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2\".", "type": "Sender"}, "msg": "Couldn't create policy sup-rp-storage-policy: An error occurred (MalformedPolicyDocument) when calling the CreatePolicy operation: Partition \"${ARN_PARTITION}\" is not valid for resource \"arn:${ARN_PARTITION}:s3:::sup-rp-uet2\".", "response_metadata": {"http_headers": {"connection": "close", "content-length": "370", "content-type": "text/xml", "date": "Wed, 11 May 2022 22:54:26 GMT", "x-amzn-requestid": "d992095d-cf39-42d9-9ed0-3d5f4af3ee53"}, "http_status_code": 400, "request_id": "d992095d-cf39-42d9-9ed0-3d5f4af3ee53", "retry_attempts": 0}}
PLAY RECAP *****************************************************************************************************************************************************************************************************************
localhost : ok=212 changed=32 unreachable=0 failed=1 skipped=152 rescued=0 ignored=0
It will be necessary to replace "${ARN_PARTITION}" with "aws" (or a GovCloud equivalent).
I found that replacement tag in the following IAM policy templates:
aws-cdp-ranger-audit-s3-policy.json
aws-cdp-datalake-admin-s3-policy.json
aws-cdp-bucket-access-policy.json
aws-cdp-dynamodb-policy.json (although this one can probably be ignored)
aws-cdp-log-policy.json
Something along the lines of the following variables.
infra:
aws:
vpc:
existing:
security_groups:
default_id:
knox_id:
Trying to deploy an Azure environment is failing due to a missing Role Assignation that should be added at the setup_azure_authz.yml task in the Platform role. The Audit role needs access to the Log container.
Missing Role:
- role: "{{ __azure_storageblobdata_ctrb_role_id }}"
scope: "{{ plat__azure_logpath_uri }}"
assignee: "{{ __azure_ranger_audit_identity.properties.principalId }}"
cloudera.exe/roles/platform/tasks/setup_azure_authz.yml
Lines 200 to 203 in f47c8d3
Migration missed group_id, instance_profile_name, instance_tags
There is a minor bug in the setup_aws_network.yml task file of cloudera.exe.infrastructure where infra__security_group_vpce_name is used as a literal string instead of as a variable:
This causes the AWS VPC Endpoint Security Group Name tag to have a value of infra__security_group_vpce_name.
Remove quotes around infra__security_group_vpce_name in Line 256 of above file - i.e.
This code:
tags: "{{ infra__tags | combine({ 'Name': 'infra__security_group_vpce_name' }, recursive=True) }}"
Becomes:
tags: "{{ infra__tags | combine({ 'Name': infra__security_group_vpce_name }, recursive=True) }}"
When running Cloudera-Deploy to create an environment on the eu-1 CDP tenant the error below occurs in the platform role.
Failing Ansible Task: cloudera.exe.platform: Create CDP Admin group
Error message (snippet):
fatal: [localhost]: FAILED! => {"changed": false, "error": "{'base_error': ClientError(\"An error occurred: Role 'crn:altus:iam:us-west-1:altus:role:EnvironmentAdmin' does not exist in the account 'altus' (Status Code: 404; Error Code: NOT_FOUND; Service: iam; Operation: assignGroupRole; Request ID: ;)\"), "msg": "Client request error", "violations": "Role 'crn:altus:iam:us-west-1:altus:role:EnvironmentAdmin' does not exist in the account 'altus'"}
Debugging the error we found that the CRN of resources in the eu-1 CDP tenant are different and a prefixed with crn:altus:iam:eu-1:altus instead of crn:altus:iam:us-west-1:altus. Example from cdpcli is shown below:
# Get list of role
$ cdp --profile prod_eu-1 iam list-roles | grep Environ
cdp --profile prod_eu-1 iam list-roles | grep Environ
"crn": "crn:altus:iam:eu-1:altus:role:EnvironmentAdmin",
"crn": "crn:altus:iam:eu-1:altus:policy:EnvironmentAdminPolicy",
"crn": "crn:altus:iam:eu-1:altus:role:EnvironmentCreator",
"crn": "crn:altus:iam:eu-1:altus:policy:EnvironmentCreatorPolicy",
"environments/createEnvironment",
"crn": "crn:altus:iam:eu-1:altus:role:EnvironmentUser",
"crn": "crn:altus:iam:eu-1:altus:policy:EnvironmentUserPolicy",Tracing the us-west-1 CRN string in the code we found that the CRN namespace is hard-coded in the variables file of the platform role:
cloudera.exe/roles/platform/vars/main.yml
Line 33 in 28b5650
Allow the CDP Control Plane Region to be selected via globals. This can be optional, allowing a default as us-west-1.
Create variables/defaults in common for the CDP Control Plane Region and CRN - these can take the value of the global.cdp_region if specified. Use these variables in any roles where the CDP region or crn is required.
The recent PR #91 introduces a new global variable. This needs to be recorded in configuration.yml.
Would be really useful if you can specify the cdp datalake runtime to deploy as opposed to always getting the latest.
Should be run_cdp_datalake_version, and therefore settable by the user to override the default and pin the runtime version
cloudera.exe/docs/configuration.yml
Line 65 in 28b5650
Many parts of the configuration.yml documentation state that the user may supply tags for individual components and expect them to be applied, except they are not implemented in the Framework. It appears that mostly we generate common infra tags and apply to them to all infra where possible. This should either be implemented or the documentation corrected.
Deployments of experiences like CDW and CDF are likely to fail if infrastructure subnets are not correctly tagged to inform whether the kubernetes elb is public or private
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.