vmware-archive / pcf-pipelines Goto Github PK
View Code? Open in Web Editor NEWPCF Pipelines
License: Apache License 2.0
PCF Pipelines
License: Apache License 2.0
In pcf-pipelines/tasks/config-ert/task.sh
local azs_csv=$1
echo $azs_csv | awk -F "," -v braceopen='{' -v braceclose='}' -v name='"name":' -v quote='"' -v OFS='"},{"name":"' '$1=$1 {print braceopen name quote $0 quote braceclose}'
}```
Only gets the first word and not the full az name as vSphere allows spaces.
I got it to work by changing this value:
https://github.com/pivotal-cf/pcf-pipelines/blob/master/install-pcf/gcp/params.yml#L9
Should that be a CHANGEME
?
When disabling tcp routing, left tcp ports blank. Job failed at config-elastic-runtime-tile stage config-er-tile with error:
staging cf 1.11.0-build.49
finished staging
Status: 200 OK
Cache-Control: max-age=0, private, must-revalidate
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 09 May 2017 20:10:08 GMT
Etag: W/"072de635532b82c376c8c599b299e176"
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: ea690d92-d7f4-483a-ab86-e478f443ad82
X-Runtime: 0.193366
X-Xss-Protection: 1; mode=block
Using self signed certificates generated using Ops Manager...
configuring product...
setting properties
could not execute "configure-product": failed to configure product: request failed: unexpected response:
HTTP/1.1 422 Unprocessable Entity
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 09 May 2017 20:10:10 GMT
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 39a7affc-fb61-4955-8c67-bbaec5abbb0a
X-Runtime: 0.493735
X-Xss-Protection: 1; mode=block
57
{"errors":{".properties.tcp_routing.enable.reservable_ports":["Value can't be blank"]}}
0
download-stemcell tool should be able to handle mix case like vSphere instead of only requiring lowercase. Notes in the params file do state to look up the iaas name from the repo but it would be nice if it is not case sensitive
From @cholick, originally in gcp-concourse:
GCP installation instructions don't restrict traffic flow to tags, see:
http://docs.pivotal.io/pivotalcf/1-9/customizing/gcp-prepare-env.html#firewall_rulesBut the terraformed firewall rule for internal traffic does:
https://github.com/pivotal-cf/gcp-concourse/blob/master/terraform/firewalls.tf#L76-#L77This creates traffic issues on bosh-deployed / odb vms, which don't get tagged in the same way.
See ref: https://github.com/pivotal-cf/aws-concourse/issues/12 (that repository is now deprecated).
This is just a copy of the issue.
@bijukunjummen
I noticed that some of the pipeline steps refer to this docker image virtmerlin/c0-worker, wouldn't hosting and using docker images from a team dockerhub a better idea.
@krishicks
We have chores to replace any non-cflinuxfs2 images with either cflinuxfs2 or one based on
cflinuxfs2 that we maintain.They're here:
https://www.pivotaltracker.com/story/show/144251129
https://www.pivotaltracker.com/story/show/144251109
https://www.pivotaltracker.com/story/show/144251095
Looks like the pipe is downloading something from Rahul Jain's GIT:
Status: Downloaded newer image for rjain/buildbox@sha256:
This should probably only reference materials from C[0] or R&D.
The network name in the params.yml
file for property om_vm_network
does not support space characters in the value (e.g. VLAN 26
). The resulting network name is just VLAN
.
The root cause appears to be this line in tasks/import-opsman/task.sh
:
setNetworkMapping $OM_VM_NETWORK
The $OM_VM_NETWORK
variable should be quoted.
Getting below during opsman install running install-pcf/vsphere/pipeline
./pivnet-opsman-product/pcf-vsphere-1.10.3.ova
jq: error: ip0/0 is not defined at , line 6:
(.PropertyMapping[] | select(.Key == ip0)).Value = $ip0 |
jq: error: netmask0/0 is not defined at , line 7:
(.PropertyMapping[] | select(.Key == netmask0)).Value = $netmask0 |
jq: error: gateway/0 is not defined at , line 8:
(.PropertyMapping[] | select(.Key == gateway)).Value = $gateway |
jq: error: DNS/0 is not defined at , line 9:
(.PropertyMapping[] | select(.Key == DNS)).Value = $dns |
jq: error: ntp_servers/0 is not defined at , line 10:
(.PropertyMapping[] | select(.Key == ntp_servers)).Value = $ntpServers |
jq: error: admin_password/0 is not defined at , line 11:
(.PropertyMapping[] | select(.Key == admin_password)).Value = $adminPassword |
jq: error: custom_hostname/0 is not defined at , line 12:
(.PropertyMapping[] | select(.Key == custom_hostname)).Value = $customHostname
jq: 7 compile errors
It looks like the pcf-pipelines/tasks/import-opsman/task.sh file was refactored for this current v0.7.0 tar release.
It looks like configure-ert
step with 1.9.21
of ERT is breaking as there is a new required field in the Authentication and Enterprise SSO
of configuration. Likely also with the latest patch version of 1.10.
Currently, looking at the parameters available in vsphere/upgrade-ops-manager-params.yml
, the list seems much smaller than what the OpsManager UI exposes. I'd expect it to be much larger (to recreate the original configuration) or much smaller (configuration imported from previously exported installation).
Is the Upgrade Ops Manager pipeline ready for use on vSphere installations?
Hey, thanks for sharing the pipeline setup! We are currently trying to automate a lot of the infrastructure around CF with Concourse. Do you plan to add a step for om configure-product
for the CF ERT? If not, what objections do you see?
I downloaded the Concourse Automation Pipeline from PivNet to update an instance of PCF. I came across a minor error where the downloaded pcf-pipeline-tarball from pivnet did not match the filename present in the run command of the tarball-unpack task within upgrad-tile job. Same issue with upgrade-buildpack pipeline where the tarball-unpack task in for each buildpack has a mismatch in the pcf-pipeline tarball and the product_version define in the resource section.
name: pcf-pipelines-tarball
type: pivnet
source:
api_token: {{pivnet_token}}
product_slug: pcf-automation
product_version: v0.1.0
The last line should be:
run:` { path: tar, args: [ "-xvzf", "pcf-pipelines-tarball/pcf-pipelines-v0.1.0.tgz" ] }
To make it resilient, it would be better if chainging the product_version of pcf-pipeline-tarball does not require changing the run task in the tarball-unpack task.
SSL Configuration:...
Setting CF resources and properties...
configuring product...
setting properties
could not execute "configure-product": failed to configure product: request failed: unexpected response:
HTTP/1.1 422 Unprocessable Entity
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 02 May 2017 01:58:38 GMT
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 5cc38bd6-0372-4057-9854-62725b462763
X-Runtime: 0.574899
X-Xss-Protection: 1; mode=block
cd
{"errors":{".properties.tcp_routing":["You can only change properties that are under the selected option"],".**properties.mysql_backups":["Value 'enable' does not match the select_value of a known option"]**}}
0
See ref: https://github.com/pivotal-cf/aws-concourse/issues/8
This is just a copy of the issue.
@wjjk940
https://github.com/c0-ops/om (0.8.1) https://github.com/pivotal-cf/om/releases (0.22.0) — I don’t know why the script isn’t using om [options] configure-bosh []
@ryanpei
this is a good point. we should refactor the pipelines to use om configure-bosh now that this is supported in om master.
This should fail out, however job shows green.
Configuring networks...
Status: 422 Unprocessable Entity
Cache-Control: no-cache
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Thu, 27 Apr 2017 16:07:57 GMT
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: 043b0400-32af-478a-91cf-569714e40c44
X-Runtime: 0.175094
X-Xss-Protection: 1; mode=block
{
"errors": [
"Networks There is an error on a network.",
"Networks DEPLOYMENT: Subnets : [0] Availability zone references can't be blank",
"Availability zones This infrastructure requires an availability zone",
"Availability zones cannot find availability zone with name PCF"
]
}```
The reference architecture is 3 networks and AZs. Make the scripts smarter to allow for non-reference architectures. Our current need is for 1 network and 1 AZ.
my params.yml
## TCP routing and routing services
tcp_routing: disable # Enable/disable TCP routing (enable|disable)
tcp_routing_ports: # A comma-separated list of ports and hyphen-separated port ranges, e.g. 52135,34000-35000,23478
route_services: disable # Enable/disable route services (enable|disable)
Error during configure-elastic-runtime/config-er-tile:
staging cf 1.10.7-build.1
finished staging
SSL Configuration:...
Setting CF resources and properties...
configuring product...
setting properties
could not execute "configure-product": failed to configure product: request failed: unexpected response:
HTTP/1.1 422 Unprocessable Entity
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 02 May 2017 01:49:08 GMT
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: f46b568e-4b9c-49b6-ae05-2bc98f493b65
X-Runtime: 0.716472
X-Xss-Protection: 1; mode=block
132
{"errors":{".properties.tcp_routing.enable.reservable_ports":["Value can't be blank"],".properties.security_acknowledgement":["Value can't be blank"],".properties.mysql_backups":["Value 'enable' does not match the select_value of a known option"],".mysql_monitor.recipient_email":["Value can't be blank"]}}
0
access to bosh.io is a required firewall rule
https://github.com/pivotal-cf/pcf-pipelines/blob/master/FIREWALL.md
Wrong glob
reference being used in install-pcf/vsphere/pipeline.yml
.
- name: upload-elastic-runtime-tile
plan:
...
- get: pivnet-cli
params: {globs: ["*linux_amd64*"]}
Should be:
- name: upload-elastic-runtime-tile
plan:
...
- get: pivnet-cli
params: {globs: ["*linux-amd64*"]}
Note the dash usage instead of underscore in globs
.
In params.yml
file's trusted_certs
property (and ssl_cert
property), multi-line values are allowed by YAML. Example:
trusted_certs: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
This doesn't translate correctly into JSON during the configure-ops-director/config-opsdir
task. JSON does not allow multiline values. Using a multiline yaml value for a certificate causes a 400 Bad Request
when configuring security.
The workaround is to make the certificate value a single line value in the yaml file. We used \r\n
in place of hard return characters. However, it would be a nice fix to be able to specify a certificate as a multiline value as yaml syntax allows.
Please take this as a suggestion / constructive feedback after using these for a few months on AWS, Azure, and vSphere
The most easy to debug and customize IaaS for pcf-upgrades and installs is vSphere because of the use of govc. Whereas the other clouds Ops Man pipelines have not once done what I'd expect, and have required many rounds of debugging to tweak my setup to the expectations of cliaas.
While golang as a CLI is great, using golang effectively as a scripting language (as it is here, since there are so many variations in the task) has major maintainability problems in the wild.
These pipelines need to be customized for specific customer environments, and unless cliaas is going to handle almost every permutation of variation, it's a black box to system administrators who are never going to learn golang , so people will wind up writing their own scripted CLI , or IaaS Template (Azure ARM template, AWS CF, etc.) or terraform-based version of the task. I'd much rather have something easily tinker-able like this.
that said if the goal is to have cliaas "just handle it all", kudos, it's just not there for a while.
Examples include
Getting following error on upload-elastic-runtime-tile
chmod: cannot access 'tool-om/om-linux': No such file or directory
Believe this is due to the folder being changed to om-cli instead of tool-om.
Hijacking the build container shows these directories:
root@25c4a38e-76d9-4945-658e-8916f27ca8cf:/tmp/build/8ac8d33d# ll
total 0
drwxr-xr-x 1 root root 86 Apr 20 02:22 ./
drwxr-xr-x 1 root root 16 Apr 20 02:08 ../
drwxr-xr-x 1 root root 44 Apr 18 18:27 om-cli/
drwxr-xr-x 1 root root 218 Apr 19 23:58 pcf-pipelines/
drwxr-xr-x 1 root root 20 Apr 20 01:25 pivnet-cli/
drwxr-xr-x 1 root root 116 Apr 20 01:22 pivnet-product/
Doing a search of the repo several tasks still call out the old name it appears.
If you leave ssl cert blank in the params file it attempts to use a nonexistent parameter $OPS_MGR_GENERATE_SSL_ENDPOINT to generate self signed certs.
This fails as it's not defined in the parameters.
Last night while trying to upgrade to Ops Manager 1.10.8, my pipelines (0.14.1, da072c7) failed in the download-stemcells
task with the following error:
Logged-in successfully
pcf-pipelines/tasks/download-pivnet-stemcells/task.sh: line 55: pivnet: unbound variable
pcf-pipelines/tasks/download-pivnet-stemcells/task.sh: line 60: pivnet: unbound variable
Could not find stemcell 3363.24 for vsphere. Did you specify a supported IaaS type for this stemcell version?
With commit 3da8645 the upstream aws-concourse repository was deprecated. The majority of the tasks were brought in but the README references pcfaws_terraform_pipeline.yml
which does not exist.
Either the README.md is wrong or the file was missed.
Since the ops mgr settings are not persisted/backed up as part of the pipeline, if the concourse worker container goes away during the deploy for any reason, then there is the risk that you will not be able to put the settings back into ops mgr.
Can we maybe have a backup done as part of the pipeline to s3? or an option for something like that?
When using the deploy-director task on GCP it fails with the following:
####################################################################
GETTING JSON FOR: DIRECTOR -> networks <- file ...
####################################################################
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}[]
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}[]
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}[]
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}[]
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}[]
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}[]
jq: 2 compile errors
jq: error: syntax error, unexpected '$' (Unix shell quoting issues?) at <top-level>, line 1:
.${key}
jq: error: try .["field"] instead of .field for unusually named fields at <top-level>, line 1:
.${key}
jq: 2 compile errors
thus the networking and the az tab is not filled in correctly. It fails as well for the following:
Applying changes on Ops Manager @ gcp.domain.com
could not execute "apply-changes": could not check for any already running installation: could not make api request to installations endpoint: token could not be retrieved from target url: Post https://gcp.domain.com/uaa/oauth/token: dial tcp: lookup gcp.domain.com on 8.8.8.8:53: no such host
The opsman Url should be opsman.gcp.domain.com
, not gcp.domain.com
.
(for example if the network failed to be updated in the previous step)
install-ert pipelines should support mysql s3
backup configuration
I'm unsure if this is specific to the pipeline, om, or Ops Mgr. But I set haproxy static ip, and got the message "errors":["IP '192.168.144.11' is already taken, cannot be taken for 'cf-f7fc90f1f948b5ffffb9_ha_proxy-f7a7297e6fc566c528ea-partition-21a67ea37bbec67c3833'"]}
during apply.
deployment_network_name: "DEPLOYMENT"
deployment_vsphere_network: "VM Network" # vCenter Deployment network name
deployment_nw_cidr: 192.168.128.0/17 # Deployment network CIDR, ex: 10.0.0.0/22
deployment_excluded_range: "192.168.128.1-192.168.144.9, 192.168.159.250-192.168.255.255" # Deployment network exclusion range
deployment_nw_dns: 192.168.3.14 # Deployment network DNS
deployment_nw_gateway: 192.168.128.1 # Deployment network Gateway
deployment_nw_azs: PCF Cluster # Comma seperated list of AZ's to be associated with this network
ha_proxy_ips: 192.168.144.111, 192.168.144.112 # Comma-separated list of static IPs
install-ert pipelines should support mysql scp
backup configuration
This problem happened in two distinct PCF 1.9.2 environments of a customer that deployed the pcf-pipelines (tested with v0.8, v0.11 and v0.13.2) :
the tasks for Apply-Change
and Wait-for-Opsmgr
of the Upgrade-Tile
pipeline both return that a task is already running in OpsMgr even though there is no one started in the Ops Mgr UI. That return code prevents the pipeline from proceeding to the Apply-Changes phase of the upgrade, requiring the customer to use the OpsMgr UI to continue.
The root cause:
We found out that in the OpsMgr's API "installations" output contained a task a couple of months old that was still in "running" state and with no finished_at
date (see example below).
That entry caused the tasks mentioned above to incorrectly return that a task is already running (even the apply-changes command of the om
tool v0.23 fails because of it) because their methods simply check for the existence of an entry with "running" state.
According to the customer, what seems to have caused that situation was the reboot of the OpsMgr VM after the corresponding running action got stuck. Apparently OpsMgr left that entry unchanged in its installs
table after the reboot and never updated it to failed
state.
{
"user_name": "admin",
"finished_at": null,
"started_at": "2017-02-01T17:38:42.941Z",
"status": "running",
"additions": [],
"deletions": [],
"updates": [
{
"identifier": "p-rabbitmq",
...
},
...
],
"id": 17
},
Potential solutions:
A) Update both wait-for-opsmgr
task.sh and the om
tool to parse the recent OpsMgr events json file instead of just searching for an entry with "running" status; OR
B) Provide a Known-Issues readme in the pcf-pipelines package describing the issue above and the workaround below to fix those event entries in the OpsMgr DB:
1) Make a backup copy of OpsMgr settings (add link to docs)
2) SSH to the OpsMgr VM and become root (sudo su -
)
3) Switch to postgres user (sudo su postgres
)
4) Execute command psql
(no password required)
5) Connect to the DB:
> \connect tempest_production
6) Find the id
of the task in "running" state
SELECT from installs WHERE status='running';
7) Change the status of the corresponding entry:
UPDATE installs SET status='failed', finished_at='2017-05-04T13:58:39.620Z', finished=true WHERE id='<ID-NUMBER-FROM-PREVIOUS-STEP>';
Ran the build pack pipeline and it only updates to 3.16, not to 4.0, even though 4.0 is out.
https://network.pivotal.io/products/buildpacks#/releases/5169
Not sure if this is intentional or not?
When running the create-infrastructure
part of the pipeline, this fails with
The module root could not be found. There is nothing to output.
Changing the terraform output command to
output_json=$(/opt/terraform/terraform output -json -state=terraform-$version.tfstate)
it works.
govc import.ova is failing with below message
`Running` govc import.ova -options=opsman_settings.json -name=OpsManager-1.9.3-201702240222 -k=true -u=VCENTER -ds=DS -dc=DC -pool=POOL -folder=FOLDER /tmp/build/39dbd1bf/pivnet-opsmgr/pcf-vsphere-1.9.3.ova
./govc/govc_linux_amd64: no file specified
I think that something is conflicting. As when I remove all but the below options deploy begins to work correctly.
Running govc import.ova -options=opsman_settings.json -name=OpsManager-1.9.3-201702240222 -k=true -folder=FOLDER /tmp/build/39dbd1bf/pivnet-opsmgr/pcf-vsphere-1.9.3.ova
[24-02-17 02:56:48] Uploading pivotal-ops-manager-disk1.vmdk... (28%, 16.5MiB/s)
See ref: https://github.com/pivotal-cf/aws-concourse/issues/13
This is just a copy of that issue.
@bijukunjummen
TheOPSMAN_URI
parameter is likely to be redundant - in some of the pipeline steps it is derived ashttps://opsman.$ERT_DOMAIN
, if by any chance a differentOPSMAN_URI
is provided the import-stemcell step fails as the url's don't match up. We should either useOPSMAN_URI
at all places or removeOPSMAN_URI
and derive the url.
@ryanpei
@bijukunjummen so we would replaceOPSMAN_URI
withERT_DOMAIN
in all places where it is > currently used?
@bijukunjummen
Yes, I think so @ryanpei, OR make use of theOPSMAN_URI
everywhere a opsman's url is required instead of deriving it ashttps://opsman.$ERT_DOMAIN
. Former may be a good start though.
The OM upload-product command has a a default timeout setting of 1800 seconds. This timeout was not long enough for one of my test environments to upload the ERT causing the pipeline to fail. I hit this issue for the upgrade to 1.9.13 and 1.9.14. I had to manually restart the upgrade-tile job which completed successfully.
https://github.com/pivotal-cf/om/blob/master/docs/upload-product/README.md
Using 0.14.1 release of pcf-pipelines. Running on GCP, attempting to upgrade from Ops Manager 1.10.3 to 1.10.8
The replace-opsman-vm task executes the following command:
cliaas-linux replace-vm --config cliaas-config/config.yml --identifier "ops-manager-1-10-3"
After a couple of minutes, the Ops Manager VM shuts down:
However, the cliaas-linux remains unresponsive until the end of the 10-minute timeout period, and then errors out with:
error: waitforstatus after stopvm failed: polling for status timed out
The pipeline should first check which ERT version is in use with PCF prior to downloading the latest from PivNet and doing any work. If the in-use ERT version is equal to latest on PivNet, do nothing.
Install pipeline failing with following message.
missing input concourse-vsphere
Looking through the repo this folder path is called out several times but doesn't appear to be present in the repo.
This is due to the pivnet cli not being present in the fetch directory.
upload-er-tile:
chmod: missing operand after '+x'
Try 'chmod --help' for more information.
I'm currently investigating this too as it's a blocker for me.
From @bijukunjummen:
The terraform scripts for provisioning an opsman vm should associate an elastic
ip address with the opsman vm, otherwise the upgrade scripts for opsman tends
to fail as it tries to associate the non-elastic ip to the new instance and
fails.
Moved from https://github.com/pivotal-cf/aws-concourse/issues/14.
When running the azure opsman upgrade (v0.8.0) pipeline we get the following error:
line 2: cannot unmarshal !!map into string
line 3: cannot unmarshal !!map into string
line 4: cannot unmarshal !!map into string
line 5: cannot unmarshal !!map into string
line 6: cannot unmarshal !!map into string
line 7: cannot unmarshal !!map into string
line 8: cannot unmarshal !!map into string
line 9: cannot unmarshal !!map into string
line 10: cannot unmarshal !!map into string
When hijacking into the replace-vm container we can see the config.yml
seems to contain extraneous curly brackets like this:
azure:
vhd_image_url: {{}}
subscription_id: {{longid}}
...
When tcp_routing
value is set to disable
, the tcp_routing_ports
field is still required and configuration of the ER tile fails. See screenshot. Ports should not be required when tcp routing is disabled.
## TCP routing and routing services
tcp_routing: disable # Enable/disable TCP routing (enable|disable)
tcp_routing_ports: # A comma-separated list of ports and hyphen-separated port ranges, e.g. 52135,34000-35000,23478
https://github.com/pivotal-cf/pcf-pipelines/tree/master/tasks/upload-product-and-stemcell/task.yml
Still points to :
run:
path: pcf-pipelines/tasks/upload-product/task.sh
needs to be updated to point to :
run:
path: pcf-pipelines/tasks/upload-product-and-stemcell/task.sh
looking at manual tasks once could track and quantify the time/effort it takes to upgrade services, application containers (tomcat for instance) and upgrade the OS. are there any capture details as part of these pipelines that could be used to quantify across tile, platform and stemcell upgrades? possibly output in each report as a stepping stone to storing externally to capture.
The Configure-ert job for AWS CF install(tasks/install-ert/configure-ert.sh
) is very different from a more generic one here (tasks/config-ert/task.sh
). Ideally the latter one should be used. This will also as a side effect fix up #53 as there is a PR in works for the VSphere one.
Tried out this project as you deprecated the older gcp-concourse project. However when using the pipeline running the create-infrastructure
bit, this fails not finding the terraform command.
pcf-pipelines/tasks/install-pcf-gcp/upload-opsman.sh: line 30: terraform: command not found
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.