Fail on finding control plane 60 retries after install about installcentos HOT 12 CLOSED

Hello again! Just following up to let you know I had no problems at all with this install!! Weird!!

So, in summary, when I was using multiple VM's (3 x masters, 3 x infras and 2 x computes) it failed every single time, regardless of how many times I reran the install and no matter what I tried, same issue every time!!

With Grant's install, worked first time on a single VM with 32 GB RAM and 4 CPU.

Go figure!!

Huge thanks to Grant for this setup though, works a treat!

I will try creating another set of VM's though and trying the multi VM install again at some point.

from installcentos.

from installcentos.

After running 'systemctl status origin-node.service' I see 'unable to update cni config: no networks found in /etc/cni/net.d'. So the networkready=false from that. That is my issue. Looking for how to fix that from this setup.

from installcentos.

Are any of you guys on the OpenShift Commons Slack channel? There are a few people having (what seems like...) this same issue with 3.10 installs, both OKD and OpenShift. When they do a 3.9 install it doesn't seem to happen, so it appears to be a new issue in 3.10...

It also doesn't seem tied to Grant's install repo method only, since the folks on the Commons Slack are doing a straight ansible install...

from installcentos.

Also created this below so now the status is ready but nothing deployed.

/etc/cni/net.d/80-openshift-network.conf
{
"cniVersion": "0.2.0",
"name": "openshift-sdn",
"type": "openshift-sdn"
}

from installcentos.

@DaleBingham This is also happening for me on VMWare Fusion, even when the guest has 16 GBs of RAM and a speedy SSD. Since @gshipley also noted he received the error when using VMWare Workstation, it sounds like the failure is VMWare specific.

Interestingly, when I tried using Grant's repo using VirtualBox on a different system, the issue didn't occur, and my single-node cluster built just fine.

from installcentos.

Not sure that it's specific to VMware either... One of the folks on Commons who's having the issue is using proxmox (KVM based). I'm using VMware and didn't hit it during my install...

Just an idea... Can you try to use '-c paramiko' on your command line for your ansible playbook runs?

I had to do that to make my 3.9 install run reliably, and I never removed it when I moved on to 3.10... Not sure if it has anything to do with this issue.

from installcentos.

Hi guys, yep, just to confirm all the above:

• I have exact same problem
• I'm trying 3.10 OKD
• I'm using Proxmox VE as my hypervisor
• Using the same VM's I've got a 3.9 origin up and running fine including Gluster.
• I'm about to try Grant's install so will see how that goes.

If anyone finds a resolution I'd love to hear about it. I'm on Slack so don't mind if here or there.

If it works for me I'll report back.

from installcentos.

Hey Guys,
I am not sure whether this will be useful or not but I have successfully deployed OKD 3.10 using install-openshift.sh (modified not to create inventory file and not to modify /etc/hosts file) from this repo and below inventory file on 3 (1 master + 2 nodes + glusterfs) vms running on VM Workstation 12.5.9:

[OSEv3:children]
masters
nodes
etcd
glusterfs
glusterfs_registry

[OSEv3:vars]
ansible_ssh_user=root
enable_excluders=False
enable_docker_excluder=Flase
ansible_service_broker_install=False

containerized=True
os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability
deployment_type=origin
openshift_deployment_type=origin

template_service_broker_selector='node-role.kubernetes.io/infra=true'
osm_use_cockpit=true
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'file': '/etc/origin/master/htpasswd'}]
openshift_master_api_port=8443
openshift_master_console_port=8443
openshift_public_hostname=console.192.168.217.128.nip.io
openshift_master_default_subdomain=apps.192.168.217.128.nip.io
openshift_master_dynamic_provisioning_enabled=True
openshift_node_groups=[{"name":"node-config-all-in-one","labels":["node-role.kubernetes.io/master=true","node-role.kubernetes.io/infra=true","node-role.kubernetes.io/compute=true"],"edits":[{"key":"kubeletArguments.pods-per-core","value":["10"]},{"key":"kubeletArguments.max-pods","value":["50"]}]},{"name":"node-config-master-infra","labels":["node-role.kubernetes.io/master=true","node-role.kubernetes.io/infra=true"],"edits":[{"key":"kubeletArguments.pods-per-core","value":["10"]},{"key":"kubeletArguments.max-pods","value":["50"]}]},{"name":"node-config-compute","labels":["node-role.kubernetes.io/compute=true"],"edits":[{"key":"kubeletArguments.pods-per-core","value":["10"]},{"key":"kubeletArguments.max-pods","value":["50"]}]}]
openshift_hosted_registry_storage_kind=glusterfs
openshift_hosted_registry_storage_volume_size=50Gi
openshift_hosted_registry_selector='node-role.kubernetes.io/infra=true'

openshift_metrics_install_metrics=true
; https://github.com/openshift/origin-metrics/issues/429
openshift_metrics_cassandra_image="docker.io/openshift/origin-metrics-cassandra:v3.11.0"
openshift_metrics_hawkular_metrics_image="docker.io/openshift/origin-metrics-hawkular-metrics:v3.11.0"
openshift_metrics_heapster_image="docker.io/openshift/origin-metrics-heapster:v3.11.0"
; https://github.com/openshift/origin-metrics/issues/429#issuecomment-417124646
openshift_metrics_schema_installer_image="docker.io/alv91/origin-metrics-schema-installer:v3.10.0"
openshift_metrics_duration=2
openshift_metrics_resolution=1m
openshift_metrics_hawkular_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_cassandra_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_heapster_nodeselector={"node-role.kubernetes.io/infra": "true"}
openshift_metrics_storage_kind=dynamic
openshift_metrics_storage_volume_size=5Gi
openshift_metrics_cassandra_pvc_storage_class_name="glusterfs-storage-block"

; openshift_logging_install_logging=true
; openshift_logging_image_version="v3.9"
; openshift_logging_elasticsearch_proxy_image_version="v1.0.0"
; logging_elasticsearch_rollout_override=false
; openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}
; openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"}
; openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}
; openshift_logging_storage_kind=dynamic
; openshift_logging_es_pvc_size=5Gi
; openshift_logging_es_pvc_storage_class_name="glusterfs-storage-block"

openshift_storage_glusterfs_namespace=app-storage
openshift_storage_glusterfs_storageclass=true
openshift_storage_glusterfs_storageclass_default=false
openshift_storage_glusterfs_block_deploy=true
openshift_storage_glusterfs_block_host_vol_size=100
openshift_storage_glusterfs_block_storageclass=true
openshift_storage_glusterfs_block_storageclass_default=false

openshift_storage_glusterfs_registry_namespace=infra-storage
openshift_storage_glusterfs_registry_block_deploy=true
openshift_storage_glusterfs_registry_block_host_vol_size=100
openshift_storage_glusterfs_registry_block_storageclass=true
openshift_storage_glusterfs_registry_block_storageclass_default=false

[masters]
master openshift_schedulable=True ansible_connection=local

[etcd]
master ansible_connection=local

[nodes]
master openshift_schedulable=True ansible_connection=local openshift_node_group_name='node-config-master-infra'
node0 openshift_node_group_name='node-config-compute'
node1 openshift_node_group_name='node-config-compute'

[glusterfs]
master ansible_connection=local glusterfs_devices='[ "/dev/sda3", "/dev/sdb" ]'
node0 glusterfs_devices='[ "/dev/sda2" ]'
node1 glusterfs_devices='[ "/dev/sda2" ]'

[glusterfs_registry]
master ansible_connection=local glusterfs_devices='[ "/dev/sda3", "/dev/sdb" ]'
node0 glusterfs_devices='[ "/dev/sda2" ]'
node1 glusterfs_devices='[ "/dev/sda2" ]'

from installcentos.

My instance is a single node and master. It deploys just fine with Virtualbox as the hypervisor, only with VMWare Fusion. 3.9 deployed runs fine on Fusion, but not 3.10 for some strange reason. Like @gshipley I get the same error during deployment, i.e., timeout of on the registry.

from installcentos.

Just reporting... the same issue there with 3.10 on bare metal 24 cores and 96 ram.
So it is not related to VMs.
After waiting 3 cycles of those 60 TASK [openshift_control_plane : Wait for all control plane pods to become ready] attempts i got kinda half successful setup (not working)
.... 60 attempts...
1st error:
failed: [192.168.1.99] (item=etcd) => {"attempts": 60, "changed": false, "item": "etcd", "results": {"cmd": "/usr/bin/oc get pod master-etcd-192.168.1.99 -o json -n kube-system", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): pods \"master-etcd-192.168.1.99\" not found\n", "stdout": ""}, "state": "list"}
... 60 attempts...
2nd error:
failed: [192.168.1.99] (item=api) => {"attempts": 60, "changed": false, "item": "api", "results": {"cmd": "/usr/bin/oc get pod master-api-192.168.1.99 -o json -n kube-system", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): pods \"master-api-192.168.1.99\" not found\n", "stdout": ""}, "state": "list"}
...60 attempts...
failed: [192.168.1.99] (item=controllers) => {"attempts": 60, "changed": false, "item": "controllers", "results": {"cmd": "/usr/bin/oc get pod master-controllers-192.168.1.99 -o json -n kube-system", "results": [{}], "returncode": 0, "stderr": "Error from server (NotFound): pods \"master-controllers-192.168.1.99\" not found\n", "stdout": ""}, "state": "list"}
Then all 200 volumes looks like was created..
At the end i have success message but only those projects are installed:

• default
  kube-public
  kube-system
  openshift
  openshift-infra
  openshift-node

I saved a installation log in case...

Will update this message if will figure out solution.

UPDATE: I just run ./install-openshift.sh one more time and got a perfect setup. Everything works as expected so far. It is safe to run script again because Ansible will skip already done things.

from installcentos.

Most probably caused by Ansible 2.7 as in the other issue. Downgrade incorporated in #97.

from installcentos.