Comments (9)
HuzefaMandvi
commented on August 23, 2024
1
So I'm pretty sure the 500 error is unrelated to not getting traces from every service (EDIT: you had already noticed this, sorry for my redundancy). I found this bug, which looks relevant to the output from your journalctl output:
https://bugs.launchpad.net/osprofiler/+bug/1743586
The bug results from Nova executing some parallelized code. At the end of the thread, we see that this is patched in osprofiler 1.15.2 - however, ORE was using stable/pike version of osprofiler which was 1.11.2.
I upgraded osprofiler to stable/queens (1.15.2) and did a pip reinstall. No longer getting the 500 error message.
That said, trace is still only getting keystone and glance.
EDIT: Just kidding - I had uninstalled one of the patches during my testing. With the 2 patches installed and the stable/queens version of osprofiler, I have no errors and a trace with neutron, nova, glance, and keystone!!!
from ore.
HuzefaMandvi
commented on August 23, 2024
Is the VM creation issue fixed now? i.e. is the VM in active state or nostate?
from ore.
EmreAtes
commented on August 23, 2024
Applying these patches enables getting traces from neuron and nova:
https://review.openstack.org/#/c/273951/16/etc/api-paste.ini
https://review.openstack.org/#/c/519664/3/nova/api/openstack/wsgi_app.py
from ore.
EmreAtes
commented on August 23, 2024
I applied the patches to the corresponding services under /opt/stack, and then ran sudo pip install --no-deps --force-reinstall -U . inside /opt/stack/nova and /opt/stack/neutron. After this I ran sudo systemctl restart devstack@*, and then I could get traces, but I also get a 500 error when I try to create VMs.
from ore.
EmreAtes
commented on August 23, 2024
There's an error when creating VMs after applying the patches. This is the debug output:
START with options: [u'--debug', u'--os-profile', u'Devstack1', u'server', u'create', u'--image', u'cirros-0.3.5-x86_64-disk', u'--flavor', u'm1.tiny', u'test_server_3']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', auth_type='password', auth_url='http://192.168.0.10/identity', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', glare_url='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, murano_packages_service='murano', murano_url='', openid_scope='', os_application_catalog_api_version='1', os_baremetal_api_version='1.9', os_beta_command=False, os_clustering_api_version='1', os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_infra_optim_api_version='1', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_volume_api_version='2', os_workflow_api_version='2', passcode='', password='***', profile=u'Devstack1', project_domain_id='default', project_domain_name='', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='RegionOne', roles='', service_provider_endpoint='', service_provider_entity_id='', timing=False, token='***', trust_id='', url='', user='', user_domain_id='default', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://192.168.0.10/identity', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'queues_api_version': '2', 'clustering_api_version': '1', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': '1.9', u'image_api_version': u'2', 'auth': {'project_name': 'admin', 'user_domain_id': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, 'profile': u'Devstack1', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': '1', 'murano_packages_service': 'murano', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '2', 'infra_optim_api_version': '1', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'floating_ip_source': u'neutron', u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'container_infra_api_version': u'1', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'application_catalog_api_version': u'1', u'key_manager_api_version': u'v1', u'workflow_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://192.168.0.10/identity', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'queues_api_version': '2', 'clustering_api_version': '1', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': '1.9', u'image_api_version': u'2', 'auth': {'project_name': 'admin', 'user_domain_id': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, 'profile': u'Devstack1', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': '1', 'murano_packages_service': 'murano', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '2', 'infra_optim_api_version': '1', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'floating_ip_source': u'neutron', u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 2, cmd group openstack.image.v2
volume API version 2, cmd group openstack.volume.v2
identity API version 3, cmd group openstack.identity.v3
object_store API version 1, cmd group openstack.object_store.v1
messaging API version 2, cmd group openstack.messaging.v2
infra-optim API version 1, cmd group openstack.infra_optim.v1
database API version 1, cmd group openstack.database.v1
clustering API version 1, cmd group openstack.clustering.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
neutronclient API version 2, cmd group openstack.neutronclient.v2
application_catalog API version 1, cmd group openstack.application_catalog.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
baremetal API version 1.9, cmd group openstack.baremetal.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
event API version 2, cmd group openstack.event.v2
metric API version 1, cmd group openstack.metric.v1
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://192.168.0.10/identity', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'queues_api_version': '2', 'clustering_api_version': '1', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': '1.9', u'image_api_version': u'2', 'auth': {'project_name': 'admin', 'user_domain_id': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, 'profile': u'Devstack1', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': '1', 'murano_packages_service': 'murano', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '2', 'infra_optim_api_version': '1', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'floating_ip_source': u'neutron', u'interface': None, u'disable_vendor_agent': {}}
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://192.168.0.10/identity', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'queues_api_version': '2', 'clustering_api_version': '1', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': '1.9', u'image_api_version': u'2', 'auth': {'project_name': 'admin', 'user_domain_id': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, 'profile': u'Devstack1', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': '1', 'murano_packages_service': 'murano', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '2', 'infra_optim_api_version': '1', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'floating_ip_source': u'neutron', u'interface': None, u'disable_vendor_agent': {}}
command: server create -> openstackclient.compute.v2.server.CreateServer (auth=True)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': '1', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'auth_url': 'http://192.168.0.10/identity', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], 'queues_api_version': '2', 'clustering_api_version': '1', 'verify': True, 'event_api_version': '2', u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', u'container_infra_api_version': '1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': '1.9', u'image_api_version': u'2', 'auth': {'project_name': 'admin', 'user_domain_id': 'default', 'project_domain_id': 'default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, 'profile': u'Devstack1', u'orchestration_api_version': '1', 'timing': False, 'password': '***', u'application_catalog_api_version': '1', 'murano_packages_service': 'murano', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': '3', u'workflow_api_version': '2', u'volume_api_version': '2', 'infra_optim_api_version': '1', 'cert': None, u'secgroup_source': u'neutron', u'status': u'active', 'debug': True, u'floating_ip_source': u'neutron', u'interface': None, u'disable_vendor_agent': {}}
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_id': 'default', 'auth_url': 'http://192.168.0.10/identity', 'password': '***', 'project_domain_id': 'default'}
Get auth_ref
REQ: curl -g -i -X GET http://192.168.0.10/identity -H "X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5" -H "X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=" -H "Accept: application/json" -H "User-Agent: osc-lib/1.7.0 keystoneauth1/3.1.0 python-requests/2.18.2 CPython/2.7.5"
Starting new HTTP connection (1): 192.168.0.10
http://192.168.0.10:80 "GET /identity HTTP/1.1" 300 606
RESP: [300] Date: Thu, 28 Jun 2018 19:48:10 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token Content-Type: application/json Content-Length: 606 Connection: close
RESP BODY: {"versions": {"values": [{"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://192.168.0.10/identity/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.0.10/identity/v2.0/", "rel": "self"}, {"href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}
Making authentication request to http://192.168.0.10/identity/v3/auth/tokens
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "POST /identity/v3/auth/tokens HTTP/1.1" 201 3655
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "15831ea5b53d46d2b513539230fe7714", "name": "admin"}], "expires_at": "2018-06-28T20:48:10.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "d8dedd2a11d5447aa69f2cc2461de6d8", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.0.10/volume/v1/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "7c2e90686f644d1d8731a6a940c85c35"}], "type": "volume", "id": "4b9218689c22475d87ab298f2676d39a", "name": "cinder"}, {"endpoints": [{"url": "http://192.168.0.10/compute/v2/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2aa2aea7d0ae45ce8f3399aabd67f1c5"}], "type": "compute_legacy", "id": "884cea0d9a2b428794c0641cac1dff71", "name": "nova_legacy"}, {"endpoints": [{"url": "http://192.168.0.10/volume/v2/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f0cf7df6b18646a0bf395e518f8e756e"}], "type": "volumev2", "id": "89fff24856b749da9a74d72ed825d802", "name": "cinderv2"}, {"endpoints": [{"url": "http://192.168.0.10:8977", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0720c175408144438cc380297b65778f"}, {"url": "http://192.168.0.10:8977", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "81d01ce087244362b11f9153fce8ff96"}, {"url": "http://192.168.0.10:8977", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "beb290ce49bf48b18358ea8b7cec54ce"}], "type": "event", "id": "94dae295102546a5acf23ef4bb9a0874", "name": "panko"}, {"endpoints": [{"url": "http://192.168.0.10:9696/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d2332705f9c540c29c2f8319924735c3"}], "type": "network", "id": "9b7be4ae6d464369939c1fa4675f16ec", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.0.10/placement", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ba304f668dbd40bab5de0d31c7ae29ee"}], "type": "placement", "id": "ae34c57217504790aaf012a7ed26ee95", "name": "placement"}, {"endpoints": [{"url": "http://192.168.0.10/image", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "924ac061e2cb4707b7af2185d98a53c7"}], "type": "image", "id": "bcc875169958449f839858c97a5a1201", "name": "glance"}, {"endpoints": [{"url": "http://192.168.0.10/identity", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "0c9b1c08cc10428a952ba0534f5eb3f6"}, {"url": "http://192.168.0.10/identity", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "8502da6061b94809a2fc4b168cbb8309"}], "type": "identity", "id": "d3632663dfc843ee96768a54c5a2f70b", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.0.10/compute/v2.1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2d4bc5abe8134801b01d041c117cd7f3"}], "type": "compute", "id": "e285fed8789540fc900133efa232c5c5", "name": "nova"}, {"endpoints": [{"url": "http://192.168.0.10/volume/v3/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9abf7f3a72434112989f888b0684fd05"}], "type": "volumev3", "id": "ff17b52aafa146019d7b0157b19f9881", "name": "cinderv3"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "32fdf588e1be4fe5b59bc9ca10e2a3d3"}, "audit_ids": ["O9On1ay6RPKhvy4oDEA0dw"], "issued_at": "2018-06-28T19:48:10.000000Z"}}
run(Namespace(availability_zone=None, block_device_mapping={}, columns=[], config_drive=False, file=[], fit_width=False, flavor=u'm1.tiny', formatter='table', hint=[], image=u'cirros-0.3.5-x86_64-disk', key_name=None, max=1, max_width=0, min=1, nic=None, noindent=False, prefix='', print_empty=False, property=None, security_group=[], server_name=u'test_server_3', user_data=None, variables=[], volume=None, wait=False))
Instantiating compute client for API Version Major: 2, Minor: 0
Instantiating compute api: <class 'openstackclient.api.compute_v2.APIv2'>
Instantiating volume client: <class 'cinderclient.v2.client.Client'>
Instantiating image client: <class 'glanceclient.v2.client.Client'>
Making authentication request to http://192.168.0.10/identity/v3/auth/tokens
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "POST /identity/v3/auth/tokens HTTP/1.1" 201 3655
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "15831ea5b53d46d2b513539230fe7714", "name": "admin"}], "expires_at": "2018-06-28T20:48:11.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "d8dedd2a11d5447aa69f2cc2461de6d8", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.0.10/volume/v1/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "7c2e90686f644d1d8731a6a940c85c35"}], "type": "volume", "id": "4b9218689c22475d87ab298f2676d39a", "name": "cinder"}, {"endpoints": [{"url": "http://192.168.0.10/compute/v2/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2aa2aea7d0ae45ce8f3399aabd67f1c5"}], "type": "compute_legacy", "id": "884cea0d9a2b428794c0641cac1dff71", "name": "nova_legacy"}, {"endpoints": [{"url": "http://192.168.0.10/volume/v2/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f0cf7df6b18646a0bf395e518f8e756e"}], "type": "volumev2", "id": "89fff24856b749da9a74d72ed825d802", "name": "cinderv2"}, {"endpoints": [{"url": "http://192.168.0.10:8977", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "0720c175408144438cc380297b65778f"}, {"url": "http://192.168.0.10:8977", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "81d01ce087244362b11f9153fce8ff96"}, {"url": "http://192.168.0.10:8977", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "beb290ce49bf48b18358ea8b7cec54ce"}], "type": "event", "id": "94dae295102546a5acf23ef4bb9a0874", "name": "panko"}, {"endpoints": [{"url": "http://192.168.0.10:9696/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d2332705f9c540c29c2f8319924735c3"}], "type": "network", "id": "9b7be4ae6d464369939c1fa4675f16ec", "name": "neutron"}, {"endpoints": [{"url": "http://192.168.0.10/placement", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ba304f668dbd40bab5de0d31c7ae29ee"}], "type": "placement", "id": "ae34c57217504790aaf012a7ed26ee95", "name": "placement"}, {"endpoints": [{"url": "http://192.168.0.10/image", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "924ac061e2cb4707b7af2185d98a53c7"}], "type": "image", "id": "bcc875169958449f839858c97a5a1201", "name": "glance"}, {"endpoints": [{"url": "http://192.168.0.10/identity", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "0c9b1c08cc10428a952ba0534f5eb3f6"}, {"url": "http://192.168.0.10/identity", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "8502da6061b94809a2fc4b168cbb8309"}], "type": "identity", "id": "d3632663dfc843ee96768a54c5a2f70b", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.0.10/compute/v2.1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2d4bc5abe8134801b01d041c117cd7f3"}], "type": "compute", "id": "e285fed8789540fc900133efa232c5c5", "name": "nova"}, {"endpoints": [{"url": "http://192.168.0.10/volume/v3/d8dedd2a11d5447aa69f2cc2461de6d8", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "9abf7f3a72434112989f888b0684fd05"}], "type": "volumev3", "id": "ff17b52aafa146019d7b0157b19f9881", "name": "cinderv3"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "32fdf588e1be4fe5b59bc9ca10e2a3d3"}, "audit_ids": ["92dHxXS3QzOklAAb4ApLXw"], "issued_at": "2018-06-28T19:48:11.000000Z"}}
Instantiating image api: <class 'openstackclient.api.image_v2.APIv2'>
curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521' -H 'X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5' -H 'X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=' -H 'Connection: keep-alive' -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' http://192.168.0.10/image/v2/images/cirros-0.3.5-x86_64-disk
Starting new HTTP connection (1): 192.168.0.10
http://192.168.0.10:80 "GET /image/v2/images/cirros-0.3.5-x86_64-disk HTTP/1.1" 404 169
GET call to image for http://192.168.0.10/image/v2/images/cirros-0.3.5-x86_64-disk used request id req-36d96fda-6361-4907-93e6-9322c6d3234b
Request returned failure status 404.
curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521' -H 'X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5' -H 'X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=' -H 'Connection: keep-alive' -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' http://192.168.0.10/image/v2/images?limit=20
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "GET /image/v2/images?limit=20 HTTP/1.1" 200 680
GET call to image for http://192.168.0.10/image/v2/images?limit=20 used request id req-dcbf0cc6-7291-4734-9aec-6024b8d1fcd7
HTTP/1.1 200 OK
Date: Thu, 28 Jun 2018 19:48:11 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
Content-Length: 680
Content-Type: application/json
x-openstack-request-id: req-dcbf0cc6-7291-4734-9aec-6024b8d1fcd7
Connection: close
{"images": [{"status": "active", "name": "cirros-0.3.5-x86_64-disk", "tags": [], "container_format": "bare", "created_at": "2018-06-28T19:13:38Z", "size": 13267968, "disk_format": "qcow2", "updated_at": "2018-06-28T19:13:38Z", "visibility": "public", "self": "/v2/images/35d14d2b-629b-43dd-8329-176bf4e462f3", "min_disk": 0, "protected": false, "id": "35d14d2b-629b-43dd-8329-176bf4e462f3", "file": "/v2/images/35d14d2b-629b-43dd-8329-176bf4e462f3/file", "checksum": "f8ab98ff5e73ebab884d80c9dc9c7290", "owner": "d8dedd2a11d5447aa69f2cc2461de6d8", "virtual_size": null, "min_ram": 0, "schema": "/v2/schemas/image"}], "schema": "/v2/schemas/images", "first": "/v2/images?limit=20"}
curl -g -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521' -H 'X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5' -H 'X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=' -H 'Connection: keep-alive' -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' http://192.168.0.10/image/v2/schemas/image
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "GET /image/v2/schemas/image HTTP/1.1" 200 4157
GET call to image for http://192.168.0.10/image/v2/schemas/image used request id req-9ac10f72-a8e3-4a96-8f35-c6c011188b4a
HTTP/1.1 200 OK
Date: Thu, 28 Jun 2018 19:48:11 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
Content-Type: application/json
Content-Length: 4157
x-openstack-request-id: req-9ac10f72-a8e3-4a96-8f35-c6c011188b4a
Connection: close
{"additionalProperties": {"type": "string"}, "name": "image", "links": [{"href": "{self}", "rel": "self"}, {"href": "{file}", "rel": "enclosure"}, {"href": "{schema}", "rel": "describedby"}], "properties": {"status": {"readOnly": true, "enum": ["queued", "saving", "active", "killed", "deleted", "pending_delete", "deactivated"], "type": "string", "description": "Status of the image"}, "tags": {"items": {"type": "string", "maxLength": 255}, "type": "array", "description": "List of strings related to the image"}, "kernel_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the kernel when booting an AMI-style image.", "is_base": false}, "container_format": {"enum": [null, "ami", "ari", "aki", "bare", "ovf", "ova", "docker"], "type": ["null", "string"], "description": "Format of the container"}, "min_ram": {"type": "integer", "description": "Amount of ram (in MB) required to boot image."}, "ramdisk_id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": ["null", "string"], "description": "ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.", "is_base": false}, "locations": {"items": {"required": ["url", "metadata"], "type": "object", "properties": {"url": {"type": "string", "maxLength": 255}, "metadata": {"type": "object"}}}, "type": "array", "description": "A set of URLs to access the image file kept in external store"}, "visibility": {"enum": ["community", "public", "private", "shared"], "type": "string", "description": "Scope of image accessibility"}, "updated_at": {"readOnly": true, "type": "string", "description": "Date and time of the last image modification"}, "owner": {"type": ["null", "string"], "description": "Owner of the image", "maxLength": 255}, "file": {"readOnly": true, "type": "string", "description": "An image file url"}, "min_disk": {"type": "integer", "description": "Amount of disk space (in GB) required to boot image."}, "virtual_size": {"readOnly": true, "type": ["null", "integer"], "description": "Virtual size of image in bytes"}, "id": {"pattern": "^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$", "type": "string", "description": "An identifier for the image"}, "size": {"readOnly": true, "type": ["null", "integer"], "description": "Size of image file in bytes"}, "instance_uuid": {"type": "string", "description": "Metadata which can be used to record which instance this image is associated with. (Informational only, does not create an instance snapshot.)", "is_base": false}, "os_distro": {"type": "string", "description": "Common name of operating system distribution as specified in https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html", "is_base": false}, "name": {"type": ["null", "string"], "description": "Descriptive name for the image", "maxLength": 255}, "checksum": {"readOnly": true, "type": ["null", "string"], "description": "md5 hash of image contents.", "maxLength": 32}, "created_at": {"readOnly": true, "type": "string", "description": "Date and time of image registration"}, "disk_format": {"enum": [null, "ami", "ari", "aki", "vhd", "vhdx", "vmdk", "raw", "qcow2", "vdi", "iso", "ploop"], "type": ["null", "string"], "description": "Format of the disk"}, "os_version": {"type": "string", "description": "Operating system version as specified by the distributor", "is_base": false}, "protected": {"type": "boolean", "description": "If true, image will not be deletable."}, "architecture": {"type": "string", "description": "Operating system architecture as specified in https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html", "is_base": false}, "direct_url": {"readOnly": true, "type": "string", "description": "URL to access the image file kept in external store"}, "self": {"readOnly": true, "type": "string", "description": "An image self url"}, "schema": {"readOnly": true, "type": "string", "description": "An image schema url"}}}
REQ: curl -g -i -X GET http://192.168.0.10/compute/v2.1/flavors/m1.tiny -H "X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5" -H "X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521"
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "GET /compute/v2.1/flavors/m1.tiny HTTP/1.1" 404 80
RESP: [404] Date: Thu, 28 Jun 2018 19:48:11 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 80 x-openstack-request-id: req-3de0e24f-9e60-4f8a-a0e2-ac12ce63540e x-compute-request-id: req-3de0e24f-9e60-4f8a-a0e2-ac12ce63540e Connection: close
RESP BODY: {"itemNotFound": {"message": "Flavor m1.tiny could not be found.", "code": 404}}
GET call to compute for http://192.168.0.10/compute/v2.1/flavors/m1.tiny used request id req-3de0e24f-9e60-4f8a-a0e2-ac12ce63540e
REQ: curl -g -i -X GET http://192.168.0.10/compute/v2.1/flavors -H "X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5" -H "X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521"
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "GET /compute/v2.1/flavors HTTP/1.1" 200 2241
RESP: [200] Date: Thu, 28 Jun 2018 19:48:11 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Content-Length: 2241 Content-Type: application/json OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version x-openstack-request-id: req-59c1cac3-0f34-4cfc-9beb-5fe53511741c x-compute-request-id: req-59c1cac3-0f34-4cfc-9beb-5fe53511741c Connection: close
RESP BODY: {"flavors": [{"id": "1", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/1", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/1", "rel": "bookmark"}], "name": "m1.tiny"}, {"id": "2", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/2", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/2", "rel": "bookmark"}], "name": "m1.small"}, {"id": "3", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/3", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/3", "rel": "bookmark"}], "name": "m1.medium"}, {"id": "4", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/4", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/4", "rel": "bookmark"}], "name": "m1.large"}, {"id": "42", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/42", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/42", "rel": "bookmark"}], "name": "m1.nano"}, {"id": "5", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/5", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/5", "rel": "bookmark"}], "name": "m1.xlarge"}, {"id": "84", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/84", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/84", "rel": "bookmark"}], "name": "m1.micro"}, {"id": "c1", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/c1", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/c1", "rel": "bookmark"}], "name": "cirros256"}, {"id": "d1", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/d1", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/d1", "rel": "bookmark"}], "name": "ds512M"}, {"id": "d2", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/d2", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/d2", "rel": "bookmark"}], "name": "ds1G"}, {"id": "d3", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/d3", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/d3", "rel": "bookmark"}], "name": "ds2G"}, {"id": "d4", "links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/d4", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/d4", "rel": "bookmark"}], "name": "ds4G"}]}
GET call to compute for http://192.168.0.10/compute/v2.1/flavors used request id req-59c1cac3-0f34-4cfc-9beb-5fe53511741c
REQ: curl -g -i -X GET http://192.168.0.10/compute/v2.1/flavors/1 -H "X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5" -H "X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521"
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "GET /compute/v2.1/flavors/1 HTTP/1.1" 200 364
RESP: [200] Date: Thu, 28 Jun 2018 19:48:11 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Content-Length: 364 Content-Type: application/json OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version x-openstack-request-id: req-ea975d24-77c2-4311-ad0c-bf1c16936f2e x-compute-request-id: req-ea975d24-77c2-4311-ad0c-bf1c16936f2e Connection: close
RESP BODY: {"flavor": {"links": [{"href": "http://192.168.0.10/compute/v2.1/flavors/1", "rel": "self"}, {"href": "http://192.168.0.10/compute/flavors/1", "rel": "bookmark"}], "ram": 512, "OS-FLV-DISABLED:disabled": false, "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "disk": 1, "id": "1", "name": "m1.tiny", "vcpus": 1, "swap": "", "OS-FLV-EXT-DATA:ephemeral": 0}}
GET call to compute for http://192.168.0.10/compute/v2.1/flavors/1 used request id req-ea975d24-77c2-4311-ad0c-bf1c16936f2e
network endpoint in service catalog
Connection: <openstack.connection.Connection object at 0x7effa163ce50>
Network client initialized using OpenStack SDK: <openstack.network.v2._proxy.Proxy object at 0x7effa14f4510>
boot_args: [u'test_server_3', {u'status': u'active', u'tags': [], u'container_format': u'bare', u'min_ram': 0, u'updated_at': u'2018-06-28T19:13:38Z', u'visibility': u'public', u'owner': u'd8dedd2a11d5447aa69f2cc2461de6d8', u'file': u'/v2/images/35d14d2b-629b-43dd-8329-176bf4e462f3/file', u'min_disk': 0, u'virtual_size': None, u'id': u'35d14d2b-629b-43dd-8329-176bf4e462f3', u'size': 13267968, u'name': u'cirros-0.3.5-x86_64-disk', u'checksum': u'f8ab98ff5e73ebab884d80c9dc9c7290', u'created_at': u'2018-06-28T19:13:38Z', u'disk_format': u'qcow2', u'protected': False, u'schema': u'/v2/schemas/image'}, <Flavor: m1.tiny>]
boot_kwargs: {'files': {}, 'userdata': None, 'key_name': None, 'availability_zone': None, 'nics': [], 'max_count': 1, 'meta': None, 'block_device_mapping_v2': [], 'min_count': 1, 'scheduler_hints': {}, 'reservation_id': None, 'security_groups': [], 'config_drive': None}
REQ: curl -g -i -X POST http://192.168.0.10/compute/v2.1/servers -H "X-Trace-HMAC: 0c5c1bd6d4e55876026e98930edff2fa3ee12de5" -H "X-Trace-Info: eyJwYXJlbnRfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIiwgImJhc2VfaWQiOiAiMWJkN2M1M2EtZGIzYy00N2ExLWIyNTAtYjMzNWI1OGYzNjZiIn0=" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}9bebb917a22d694aaf85330befe32758046a4521" -d '{"server": {"min_count": 1, "flavorRef": "1", "name": "test_server_3", "imageRef": "35d14d2b-629b-43dd-8329-176bf4e462f3", "max_count": 1}}'
Resetting dropped connection: 192.168.0.10
http://192.168.0.10:80 "POST /compute/v2.1/servers HTTP/1.1" 500 128
RESP: [500] Date: Thu, 28 Jun 2018 19:48:11 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 128 x-openstack-request-id: req-5a86126d-45a3-42cc-aeae-a7704ccd8162 x-compute-request-id: req-5a86126d-45a3-42cc-aeae-a7704ccd8162 Connection: close
RESP BODY: {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}
POST call to compute for http://192.168.0.10/compute/v2.1/servers used request id req-5a86126d-45a3-42cc-aeae-a7704ccd8162
The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5a86126d-45a3-42cc-aeae-a7704ccd8162)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/display.py", line 113, in run
column_names, data = self.take_action(parsed_args)
File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 769, in take_action
server = compute_client.servers.create(*boot_args, **boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1403, in create
**boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 802, in _boot
return_raw=return_raw, **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create
resp, body = self.api.client.post(url, body=body)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 294, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5a86126d-45a3-42cc-aeae-a7704ccd8162)
clean_up CreateServer: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5a86126d-45a3-42cc-aeae-a7704ccd8162)
Trace ID: 1bd7c53a-db3c-47a1-b250-b335b58f366b
Display trace with command:
osprofiler trace show --html 1bd7c53a-db3c-47a1-b250-b335b58f366b
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/display.py", line 113, in run
column_names, data = self.take_action(parsed_args)
File "/usr/lib/python2.7/site-packages/openstackclient/compute/v2/server.py", line 769, in take_action
server = compute_client.servers.create(*boot_args, **boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1403, in create
**boot_kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 802, in _boot
return_raw=return_raw, **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create
resp, body = self.api.client.post(url, body=body)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 294, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
raise exceptions.from_response(resp, body, url, method)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-5a86126d-45a3-42cc-aeae-a7704ccd8162)
END return value: 1
from ore.
EmreAtes
commented on August 23, 2024
This one looks like it might solve our problem
https://bugzilla.redhat.com/show_bug.cgi?id=1412661
https://ask.openstack.org/en/question/53943/nova-instance-creation-500-error/
However, there is no /var/lib/cinder in our VM
from ore.
HuzefaMandvi
commented on August 23, 2024
It looks like the VM is being created successfully? I wonder if this error message is a false flag that we're getting. Also, interestingly, creating a VM without osprofiler raises no error message.
That said..uninstalling the second patch [1] gives no error but only keystone/glance in the trace. Hmm...
[1] https://review.openstack.org/#/c/519664/3/nova/api/openstack/wsgi_app.py
from ore.
EmreAtes
commented on August 23, 2024
Whenever I create a VM, I get this following in the logs (sudo journalctl --since '22:00:26' -p 5):
Jun 29 22:02:02 ore-1.moclocal nova-api-wsgi[4642]: ERROR nova.api.openstack [None req-e3a35247-ef45-4bd1-952d-1271d75785a6 admin admin] Caught error: pop from an empty deque: IndexError: pop from an empty deque
ERROR nova.api.openstack Traceback (most recent call last):
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 84, in __call__
ERROR nova.api.openstack return req.get_response(self.application)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1316, in send
ERROR nova.api.openstack application, catch_exc_info=False)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1280, in call_application
ERROR nova.api.openstack app_iter = application(self.environ, start_response)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
ERROR nova.api.openstack return self.func(req, *args, **kwargs)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__
ERROR nova.api.openstack self._log_req(req, res, start)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR nova.api.openstack self.force_reraise()
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__
ERROR nova.api.openstack res = req.get_response(self.application)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1316, in send
ERROR nova.api.openstack application, catch_exc_info=False)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1280, in call_application
ERROR nova.api.openstack app_iter = application(self.environ, start_response)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 145, in __call__
ERROR nova.api.openstack return resp(environ, start_response)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
ERROR nova.api.openstack return self.func(req, *args, **kwargs)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 128, in __call__
ERROR nova.api.openstack return request.get_response(self.application)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 327, in __exit__
ERROR nova.api.openstack stop()
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 94, in stop
ERROR nova.api.openstack profiler.stop(info=info)
ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 393, in stop
ERROR nova.api.openstack self._notify("%s-stop" % self._name.pop(), info)
ERROR nova.api.openstack IndexError: pop from an empty deque
ERROR nova.api.openstack
It looks like profiler.stop() is being called multiple times, or the number of start() and stop()s do not match, hence the IndexError. However, most of these methods are just generic methods and I couldn't find which one of these is being traced by osprofiler.
from ore.
EmreAtes
commented on August 23, 2024
Also, during ./stack.sh I get this problem:
Jun 29 20:08:13 ore-1.moclocal neutron-openvswitch-agent[11729]: CRITICAL neutron [None req-808e11b6-d3af-4e88-913b-2af278d8a20b None None] Unhandled error: AsyncProcessException: Process is not running.
ERROR neutron Traceback (most recent call last):
ERROR neutron File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
ERROR neutron sys.exit(main())
ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
ERROR neutron agent_main.main()
ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
ERROR neutron mod.main()
ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
ERROR neutron 'neutron.plugins.ml2.drivers.openvswitch.agent.'
ERROR neutron File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 375, in run_apps
ERROR neutron hub.joinall(services)
ERROR neutron File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 103, in joinall
ERROR neutron t.wait()
ERROR neutron File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
ERROR neutron return self._exit_event.wait()
ERROR neutron File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
ERROR neutron current.throw(*self._exc)
ERROR neutron File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
ERROR neutron result = function(*args, **kwargs)
ERROR neutron File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 65, in _launch
ERROR neutron raise e
ERROR neutron AsyncProcessException: Process is not running.
ERROR neutron
from ore.
Related Issues (18)
- Specific additions to top-level file to list changes made to work with CentOS?
- Clean uninstalls things, but we are in offline mode
- README to specify "flavor" must be "large" or "medium"
- Timeout during ./stack.sh HOT 3
- No "neutron" shows up in network-related traces HOT 2
- stack.sh not using branches specified in local.conf for plugins HOT 2
- Have to repeatedly source openrc HOT 6
- Migration to docc-lab? HOT 4
- OSProfiler patch doesn't work HOT 1
- Neutron.conf settings enabled trace_sqlalchemy HOT 9
- Commits are not working HOT 5
- osprofiler update HOT 9
- Port binding issue with fresh install HOT 1
- SSH key configuration HOT 17
- Issue with libvirt breaks ORE HOT 10
- git complains about trailing whitespace HOT 1
- Have to manually rebuild patch HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ore.