Code Monkey home page Code Monkey logo

manageiq-providers-azure's Introduction

ManageIQ::Providers::Azure

CI Maintainability Test Coverage

Chat

Build history for master branch

ManageIQ plugin for the Azure provider.

Development

See the section on plugins in the ManageIQ Developer Setup

For quick local setup run bin/setup, which will clone the core ManageIQ repository under the spec directory and setup necessary config files. If you have already cloned it, you can run bin/update to bring the core ManageIQ code up to date.

License

The gem is available as open source under the terms of the Apache License 2.0.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

manageiq-providers-azure's People

Contributors

01100010011001010110010101110000 avatar agrare avatar alexanderzagaynov avatar bdunne avatar blomquisg avatar bzwei avatar carbonin avatar chessbyte avatar chrisarcand avatar d-m-u avatar djberg96 avatar durandom avatar fryguy avatar gmcculloug avatar gtanzillo avatar jerryk55 avatar jprause avatar jrafanie avatar jvlcek avatar kbrock avatar ladas avatar melshyrule avatar mfeifer avatar mzazrivec avatar nasark avatar renovate[bot] avatar roliveri avatar skateman avatar slemrmartin avatar tumido avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

manageiq-providers-azure's Issues

Filter out events from the Classic providers

We do not support the Azure Classic providers which are part of the older Azure ASM API. However, the event filter does not exclude events from Classic providers, this means we are accepting and storing many unnecessary events.

add Azure Provider over Proxy - does not work

my Proxy-Config

2018-04-24 09_48_23-manageiq_ configuration

Result in the miq-Gui

2018-04-24 09_44_45-manageiq_ cloud providers

Content from /var/log/manageiq/azure.log

RestClient.post "https://login.microsoftonline.com/***REPLACED-ID***/oauth2/token", "grant_type=client_credentials&client_id=***REPLACED-ID***&client_secret=***REPLACED-SECRET***&resource=https%3A%2F%2Fmanagement.azure.com%2F", "Accept"=>"*/*", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"188", "Content-Type"=>"application/x-www-form-urlencoded", "User-Agent"=>"rest-client/2.0.2 (linux-gnu x86_64) ruby/2.3.1p112"

Stack-trace from /var/log/manageiq/evm.log

[----] E, [2018-04-24T08:40:06.617482 #1782:46f140] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager.connection_rescue_block) Error Class=SocketError, Message=Failed to open TCP connection to login.microsoftonline.com:443 (getaddrinfo: Name or service not known), Backtrace=["/usr/local/lib/ruby/2.3.0/net/http.rb:882:in `rescue in block in connect'", "/usr/local/lib/ruby/2.3.0/net/http.rb:879:in `block in connect'", "/usr/local/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'", "/usr/local/lib/ruby/2.3.0/timeout.rb:101:in `timeout'", "/usr/local/lib/ruby/2.3.0/net/http.rb:878:in `connect'", "/usr/local/lib/ruby/2.3.0/net/http.rb:863:in `do_start'", "/usr/local/lib/ruby/2.3.0/net/http.rb:852:in `start'", "/usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'", "/usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'", "/usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/armrest_service.rb:212:in `rest_execute'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/armrest_service.rb:235:in `rest_post'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:292:in `fetch_token'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:246:in `ensure_token'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:170:in `token'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/armrest_service.rb:294:in `rest_execute'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/armrest_service.rb:304:in `rest_get'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/subscription_service.rb:16:in `list'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:217:in `subscriptions'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:233:in `validate_subscription'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:154:in `subscription_id='", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:129:in `block in initialize'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:129:in `each'", "/usr/local/lib/ruby/gems/2.3.0/gems/azure-armrest-0.9.7/lib/azure/armrest/configuration.rb:129:in `initialize'", "/usr/local/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-2401dcc2252d/app/models/manageiq/providers/azure/manager_mixin.rb:32:in `new'", "/usr/local/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-2401dcc2252d/app/models/manageiq/providers/azure/manager_mixin.rb:32:in `block in raw_connect'", "/usr/local/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-2401dcc2252d/app/models/manageiq/providers/azure/manager_mixin.rb:45:in `connection_rescue_block'", "/usr/local/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-2401dcc2252d/app/models/manageiq/providers/azure/manager_mixin.rb:31:in `raw_connect'", "/var/www/miq/vmdb/app/models/ext_management_system.rb:226:in `raw_connect?'", "/var/www/miq/vmdb/app/models/miq_queue.rb:449:in `block in dispatch_method'", "/usr/local/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'", "/usr/local/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'", "/usr/local/lib/ruby/2.3.0/timeout.rb:33:in `catch'", "/usr/local/lib/ruby/2.3.0/timeout.rb:33:in `catch'", "/usr/local/lib/ruby/2.3.0/timeout.rb:106:in `timeout'", "/var/www/miq/vmdb/app/models/miq_queue.rb:448:in `dispatch_method'", "/var/www/miq/vmdb/app/models/miq_queue.rb:425:in `block in deliver'", "/var/www/miq/vmdb/app/models/user.rb:261:in `with_user_group'", "/var/www/miq/vmdb/app/models/miq_queue.rb:425:in `deliver'", "/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message'", "/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'", "/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'", "/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `loop'", "/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'", "/var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop'", "/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop'", "/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop'", "/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'", "/var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start'", "/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'", "/var/www/miq/vmdb/app/models/miq_worker.rb:375:in `block in start_runner_via_fork'", "/usr/local/lib/ruby/gems/2.3.0/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'", "/usr/local/lib/ruby/gems/2.3.0/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'", "/var/www/miq/vmdb/app/models/miq_worker.rb:373:in `start_runner_via_fork'", "/var/www/miq/vmdb/app/models/miq_worker.rb:367:in `start_runner'", "/var/www/miq/vmdb/app/models/miq_worker.rb:414:in `start'", "/var/www/miq/vmdb/app/models/miq_worker.rb:266:in `start_worker'", "/var/www/miq/vmdb/app/models/miq_worker.rb:153:in `block in sync_workers'", "/var/www/miq/vmdb/app/models/miq_worker.rb:153:in `times'", "/var/www/miq/vmdb/app/models/miq_worker.rb:153:in `sync_workers'", "/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'", "/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'", "/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'", "/var/www/miq/vmdb/app/models/miq_server.rb:141:in `start'", "/var/www/miq/vmdb/app/models/miq_server.rb:233:in `start'", "/var/www/miq/vmdb/lib/workers/evm_server.rb:27:in `start'", "/var/www/miq/vmdb/lib/workers/evm_server.rb:48:in `start'", "/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'"]

RestClient-Test in irb without a Proxy

it starts with the same stacktrace

# irb
irb(main):001:0> require "rest-client"
=> true
irb(main):002:0> RestClient.post "https://login.microsoftonline.com/***REPLACED-ID***/oauth2/token", "grant_type=client_credentials&client_id=***REPLACED-ID***&client_secret=***REPLACED-SECRET***&resource=https%3A%2F%2Fmanagement.azure.com%2F", "Accept"=>"*/*", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"188", "Content-Type"=>"application/x-www-form-urlencoded", "User-Agent"=>"rest-client/2.0.2 (linux-gnu x86_64) ruby/2.3.1p112"
SocketError: Failed to open TCP connection to login.microsoftonline.com:443 (getaddrinfo: Name or service not known)
        from /usr/local/lib/ruby/2.3.0/net/http.rb:882:in `rescue in block in connect'
        from /usr/local/lib/ruby/2.3.0/net/http.rb:879:in `block in connect'
        from /usr/local/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
        from /usr/local/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
        from /usr/local/lib/ruby/2.3.0/net/http.rb:878:in `connect'
        from /usr/local/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
        from /usr/local/lib/ruby/2.3.0/net/http.rb:852:in `start'
        from /usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
        from /usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
        from /usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
        from /usr/local/lib/ruby/gems/2.3.0/gems/rest-client-2.0.2/lib/restclient.rb:71:in `post'
        from (irb):2
        from /usr/local/bin/irb:11:in `<main>'

RestClient-Test in irb with Proxy

it works

# irb
irb(main):001:0>  require "rest-client"
=> true
irb(main):002:0> RestClient.proxy = "http://19***replaced***:8080"
=> "http://19***replaced***:8080"
irb(main):003:0> RestClient.post "https://login.microsoftonline.com/***REPLACED-ID***/oauth2/token", "grant_type=client_credentials&client_id=***REPLACED-ID***&client_secret=***REPLACED-SECRET***&resource=https%3A%2F%2Fmanagement.azure.com%2F", "Accept"=>"*/*", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"188", "Content-Type"=>"application/x-www-form-urlencoded", "User-Agent"=>"rest-client/2.0.2 (linux-gnu x86_64) ruby/2.3.1p112"
=> <RestClient::Response 200 "{\"token_typ...">

Timeouts during refresh cause refresh to crash

If the refresh process experiences a timeout opening a connection to the Azure API, then the refresh process will crash. I think editing this logic in the base armrest service class to retry in the case of an OpenTimeoutException would help. Also the persistent connections that come with Excon would likely help as well, given that they were all timeouts while opening a socket, but I don't know how far out that may be.

Error on metrics capture - Disk Read Bytes/sec and Per Disk Write Bytes/sec

Was not able to get Azure metrics working on jansa-3 docker image.
After debugging and finding url call and manually running, received error:
GET /subscriptions/7cde102a-5170-451c-846b-354f7b677b95/resourceGroups/myapp-rg/providers/Microsoft.Compute/virtualMachines/ansible-vm3/providers/microsoft.insights/metrics?timespan=2021-03-18T10%3A48%3A26Z%2F2021-03-18T11%3A42%3A12Z&interval=PT1M&metricnames=Percentage+CPU%2CPer+Disk+Write+Bytes%2Fsec%2CNetwork+In+Total%2CNetwork+Out+Total&aggregation=Average&api-version=2018-01-01
Failed to find metric configuration for provider: Microsoft.Compute, resource Type: virtualMachines, metric: Per Disk Write Bytes/sec

Valid metrics: Percentage CPU,Network In,Network Out,Disk Read Bytes,Disk Write Bytes,Disk Read Operations/Sec,Disk Write Operations/Sec,CPU Credits Remaining,CPU Credits Consumed,Data Disk Read Bytes/sec,Data Disk Write Bytes/sec,Data Disk Read Operations/Sec,Data Disk Write Operations/Sec,Data Disk Queue Depth,Data Disk Bandwidth Consumed Percentage,Data Disk IOPS Consumed Percentage,Data Disk Target Bandwidth,Data Disk Target IOPS,Data Disk Max Burst Bandwidth,Data Disk Max Burst IOPS,Data Disk Used Burst BPS Credits Percentage,Data Disk Used Burst IO Credits Percentage,OS Disk Read Bytes/sec,OS Disk Write Bytes/sec,OS Disk Read Operations/Sec,OS Disk Write Operations/Sec,OS Disk Queue Depth,OS Disk Bandwidth Consumed Percentage,OS Disk IOPS Consumed Percentage,OS Disk Target Bandwidth,OS Disk Target IOPS,OS Disk Max Burst Bandwidth,OS Disk M* Connection #1 to host management.azure.com left intact
ax Burst IOPS,OS Disk Used Burst BPS Credits Percentage,OS Disk Used Burst IO Credits Percentage,Inbound Flows,Outbound Flows,Inbound Flows Maximum Creation Rate,Outbound Flows Maximum Creation Rate,Premium Data Disk Cache Read Hit,Premium Data Disk Cache Read Miss,Premium OS Disk Cache Read Hit,Premium OS Disk Cache Read Miss,VM Cached Bandwidth Consumed Percentage,VM Cached IOPS Consumed Percentage,VM Uncached Bandwidth Consumed Percentage,VM Uncached IOPS Consumed Percentage,Network In Total,Network Out Total"}

Ended up changing: /opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-e7914f85e845/app/models/manageiq/providers/azure/cloud_manager/metrics_capture.rb
changing both:
Per Disk Read Bytes/sec
Per Disk Write Bytes/sec
To:
OS Disk Read Bytes/sec
OS Disk Write Bytes/sec

Image name contains leading "./" characters

I have a private image under a "manageiq" container in one of my storage accounts. The original blob name is "win2k12img.vhd". It should just be parsed out as "win2k12img" but it ends up being "./win2k12img". You can see it in the westus2 region in our dev environment.

I suspect it's caused by the build_image_name method.

Targeted refresh fails to save with `[TypeError]: can't cast Hash` failure

[----] I, [2020-06-22T05:41:01.202632 #4738:2ae52a6d3960]  INFO -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) Refreshing all targets...
[----] I, [2020-06-22T05:41:01.202787 #4738:2ae52a6d3960]  INFO -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [MSAzure], id: [255] Refreshing target
s for EMS...
[----] I, [2020-06-22T05:41:01.202937 #4738:2ae52a6d3960]  INFO -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [MSAzure], id: [255]   InventoryRefres
h::TargetCollection [Collection of 1 targets] id [Collection of targets with id: [{:ems_ref=>{:subscription_id=>"0e0a4287-8719-4849-bb0b-5242e4507709", :vm_resource_group=>"infra-mgmt-auto-test", :type=>"microso
ft.compute/virtualmachines", :vm_name=>"test-ci-t1vp"}}]]
[----] I, [2020-06-22T05:41:13.493839 #4738:2ae52a6d3960]  INFO -- : Q-task_id([r16_miq_provision_16]) Exception in realtime_block :ems_refresh - Timings: {:collect_inventory_for_targets=>12.290756225585938, :em
s_refresh=>12.290783166885376}
[----] E, [2020-06-22T05:41:13.494053 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [MSAzure], id: [255] Refresh failed
[----] E, [2020-06-22T05:41:13.494187 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) [TypeError]: can't cast Hash  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2020-06-22T05:41:13.494260 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) /opt/IBM/infrastructure-management-gemset/gems/activerecord-5.2.4.3/lib/active_record/connection_adapters/ab
stract/quoting.rb:34:in `rescue in type_cast'
[----] E, [2020-06-22T05:41:13.494331 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [MSAzure], id: [255] Unable to perform
 refresh for the following targets:
[----] E, [2020-06-22T05:41:13.494414 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh)  --- InventoryRefresh::TargetCollection [Co
llection of 1 targets] id [Collection of targets with id: [{:ems_ref=>{:subscription_id=>"0e0a4287-8719-4849-bb0b-5242e4507709", :vm_resource_group=>"infra-mgmt-auto-test", :type=>"microsoft.compute/virtualmachi
nes", :vm_name=>"test-ci-t1vp"}}]]
[----] I, [2020-06-22T05:41:13.511627 #4738:2ae52a6d3960]  INFO -- : Q-task_id([r16_miq_provision_16]) MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2020-06-22T05:41:13.511874 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) MIQ(MiqQueue#deliver) Message id: [549090], Error: [can't cast Hash]
[----] E, [2020-06-22T05:41:13.511981 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: can't cast Hash  Method:[block (2 levels
) in <class:LogProxy>]
[----] E, [2020-06-22T05:41:13.512083 #4738:2ae52a6d3960] ERROR -- : Q-task_id([r16_miq_provision_16]) /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:67:in `refresh'

No metrics are collected

I'm trying to get Azure metrics captured on latest master, but there seems to be empty intersection between expected metric names vs obtained metric names, so nothing gets collected. It's possible I'm just having something misconfigured, but can't seem to figure what.

It seems like here in this line:

https://github.com/ManageIQ/manageiq-providers-azure/blob/master/app/models/manageiq/providers/azure/cloud_manager/metrics_capture.rb#L210

We filter out too much, because metrics_conn.list(...) in L209 returns a list similar to

["/builtin/processor/percentiowaittime",
 "/builtin/processor/percentusertime",
 "/builtin/processor/percentnicetime",
 "/builtin/processor/percentprocessortime",
 "/builtin/processor/percentinterrupttime",
...
]

while COUNTER_NAMES that we're filtering against in L210 are similar to

set("\\Processor Information(_Total)\\% Processor Time",
 "\\Processor(_Total)\\% Processor Time",
 "\\Processor\\PercentProcessorTime",
..
)

Now I'm not sure what's causing the difference - did Azure start sending different metric names?

Azure Debug logging does not contain timestamps

Looking at the azure.log with Debug logging on, I see:

RestClient.get "https://nzaccazupdstasdiagdrh1.table.core.windows.net/WADMetricsPT1MP10DV2S20171116?$filter=PartitionKey%20eq%20%27:002Fsubscriptions:<redacted>resourceGroups:<redacted>providers:002FMicrosoft:002ECompute:002FvirtualMachines:002FAZUWDRH1CM4001%27%20and%20CounterName%20eq%20%27%5CPhysicalDisk(_Total)%5CDisk%20Read%20Bytes/sec%27%20and%20Timestamp%20ge%20datetime%272017-11-21T03:15:00Z%27%20and%20Timestamp%20le%20datetime%272017-11-21T04:12:48Z%27&$select=Timestamp,TIMESTAMP,Average", "Accept"=>"application/json;odata=fullmetadata", "Accept-Encoding"=>"gzip, deflate", "User-Agent"=>"rest-client/2.0.2 (linux-gnu x86_64) ruby/2.3.1p112", "auth-string"=>"true", "auth_string"=>"true", "authorization"=>"SharedKey <redacted>", "content-type"=>"", "x-ms-date"=>"Tue, 21 Nov 2017 04:13:12 GMT", "x-ms-version"=>"2016-05-31"
# => 200 OK | application/json 11558 bytes
RestClient.post "https://management.azure.com/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Storage/storageAccounts/<redacted>/listKeys?api-version=2017-10-01", "", "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"Bearer <redacted>", "Content-Length"=>"0", "Content-Type"=>"application/json", "User-Agent"=>"rest-client/2.0.2 (linux-gnu x86_64) ruby/2.3.1p112"
# => 200 OK | application/json 372 bytes

This is useful since it shows the size of the response in bytes. But, it would be even more helpful if it could also show the timing of each request/response. Or timestamps for each request/response.

I'm not sure if the logging configuration for RestClient can get that. But, I assume it can, because I believe it should be in the header info. It's just a matter of determining if RestClient surfaces that in their logging.

Too many marketplace images collected when none are specified

If a user sets the market place image collection to true, but does not specify any specific images, the app will collect every possible marketplace image.

While thorough, it is not very useful, because it will collect thousands of them. The problem isn't that there are really thousands of available images, there are really just a small number of them, but each one has potentially dozens of versions. These "versions" are really just timestamps. This is very confusing to end users, who will have to sift through them all.

In reality we should only collect the "latest" version, because that's what both the Azure portal and the Azure CLI use when you provision a virtual machine. In fact, if you inspect any virtual machine, it will simply say "latest" for the version.

Azure Discover process is failing with the below error "undefined method `address_prefix' for #<Azure::Armrest::Network::VirtualNetwork::Properties::Subnet::Properties:0x000..."

Azure discover process is failing to retrieve all the resources as the process failing with "undefined method 'address_prefix' for #<Azure::Armrest::Network::VirtualNetwork::Properties::Subnet::Properties:0x000..."

  1. Add a Azure provider
  2. wait for the discovery process to start.
  3. Error appears on the Azure provider summary page.

-- Impacts all versions. Have tested on Najdorf, Oparin and Petrosian.

[----] E, [2024-02-07T19:17:20.223618 #3989007:93a8] ERROR -- evm: MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [CFX-Azure-USEast], id: [7] Unable to perform refresh for the following targets:
[----] E, [2024-02-07T19:17:20.223776 #3989007:93a8] ERROR -- evm: MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh)  --- ManageIQ::Providers::Azure::CloudManager [CFX-Azure-USEast] id [7]
[----] I, [2024-02-07T19:17:20.260304 #3989007:93a8]  INFO -- evm: MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2024-02-07T19:17:20.260637 #3989007:93a8] ERROR -- evm: MIQ(MiqQueue#deliver) Message id: [19030211], Error: [undefined method `address_prefix' for #<Azure::Armrest::Network::VirtualNetwork::Properties::Subnet::Properties:0x0000556bc944e078>]
[----] E, [2024-02-07T19:17:20.260865 #3989007:93a8] ERROR -- evm: [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: undefined method `address_prefix' for #<Azure::Armrest::Network::VirtualNetwork::Properties::Subnet::Properties:0x0000556bc944e078>  Method:[block (2 levels) in <class:LogProxy>]
----] E, [2024-02-07T19:17:20.223510 #3989007:93a8] ERROR -- evm: /opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:87:in `block in cloud_subnets
'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:82:in `each'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:82:in `cloud_subnets'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:77:in `block in cloud_networks'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:66:in `each'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:66:in `cloud_networks'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-622e43171bad/app/models/manageiq/providers/azure/inventory/parser/network_manager.rb:10:in `parse'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:42:in `block in parse'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:39:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:39:in `parse'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:130:in `parse_targeted_inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:95:in `block in refresh_targets_for_ems'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.4.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.4.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:56:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:94:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block (2 levels) in refresh'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.4.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.4.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:62:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block in refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:71:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:70:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:70:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:484:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:95:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:110:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:482:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:382:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `deliver'

Bad request when getting metrics from koreacentral region

[----] W, [2021-01-26T04:29:11.735097 #7:2b13dbeb395c] WARN -- : MIQ(ManageIQ::Providers::Azure::CloudManage r::MetricsCapture#perf_collect_metrics) Problem collecting metrics for [FILTERED]: [400 ] {"code":"BadRequest","message":"Failed to find metric configuration for provider: Microsoft.Compute, resour ce Type: virtualMachines, metric: Per Disk Read Bytes/sec, Valid metrics: Percentage CPU,Network In,Network O ut,Disk Read Bytes,Disk Write Bytes,Disk Read Operations/Sec,Disk Write Operations/Sec,CPU Credits Remaining, CPU Credits Consumed,Data Disk Read Bytes/sec,Data Disk Write Bytes/sec,Data Disk Read Operations/Sec,Data Di sk Write Operations/Sec,Data Disk Queue Depth,Data Disk Bandwidth Consumed Percentage,Data Disk IOPS Consumed Percentage,Data Disk Target Bandwidth,Data Disk Target IOPS,Data Disk Max Burst Bandwidth,Data Disk Max Burs t IOPS,Data Disk Used Burst BPS Credits Percentage,Data Disk Used Burst IO Credits Percentage,OS Disk Read By tes/sec,OS Disk Write Bytes/sec,OS Disk Read Operations/Sec,OS Disk Write Operations/Sec,OS Disk Queue Depth, OS Disk Bandwidth Consumed Percentage,OS Disk IOPS Consumed Percentage,OS Disk Target Bandwidth,OS Disk Targe t IOPS,OS Disk Max Burst Bandwidth,OS Disk Max Burst IOPS,OS Disk Used Burst BPS Credits Percentage,OS Disk U sed Burst IO Credits Percentage,Inbound Flows,Outbound Flows,Inbound Flows Maximum Creation Rate,Outbound Flo ws Maximum Creation Rate,Premium Data Disk Cache Read Hit,Premium Data Disk Cache Read Miss,Premium OS Disk C ache Read Hit,Premium OS Disk Cache Read Miss,VM Cached Bandwidth Consumed Percentage,VM Cached IOPS Consumed Percentage,VM Uncached Bandwidth Consumed Percentage,VM Uncached IOPS Consumed Percentage,Network In Total,N etwork Out Total"} (cause: 400 Bad Request). Region [koreacentral] may not be supported.

In contacting Microsoft, they stated that the request was wrong and recommended to edit from %252C to %2C when calling API to get data from Azure OCP cluster. Unfortunately I don't have more information, nor a stack trace.

Since %25 is the escape code for a literal % char, this seems like a double-escape bug.

Downloading Orchestration Templates during Inventory Refresh can cause worker to exceed memory threshold

Unfortunately, this is actually a guess right now. But based on the log snipped below, it looks like the refresher is attempting to download templates when the refresh worker is killed for exceeding the memory threshold.

For this particular customer case, the customer increased the memory threshold and refreshes started working.

However, if there is a significantly large orchestration template in a customer's Azure inventory, this issue would likely come up again.

The Azure Inventory collector should probably not attempt to download all of the orchestration templates in the same process that's handling the general inventory collection.

One way to do that would be to create a skeleton record for the orchestration template, then queue up separate downloads for each orchestration template. This could allow the downloads to happen in parallel if necessary. And, it would remove the memory footprint from the inventory collection process.

But, there's probably other ways to do this, too.

Log snippet:

[----] I, [2017-08-15T12:05:28.160799 #15777:87d14c]  INFO -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshParser#ems_inv_to_hashes) Collecting data for EMS : [Azure-USEast] id: [10000000000012]...
[----] E, [2017-08-15T12:06:33.464578 #15777:87d14c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshParser#download_template) Failed to download Azure template <REDACTED>. Reason: #<RestClient::NotFound: 404 Not Found>
[----] E, [2017-08-15T12:06:34.477883 #15777:87d14c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshParser#download_template) Failed to download Azure template <REDACTED> Reason: #<RestClient::NotFound: 404 Not Found>
[----] E, [2017-08-15T12:06:47.122865 #15777:87d14c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshParser#download_template) Failed to download Azure template <REDACTED> Reason: #<RestClient::NotFound: 404 Not Found>
[----] E, [2017-08-15T12:07:00.357988 #15777:87d14c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshParser#download_template) Failed to download Azure template <REDACTED> Reason: #<RestClient::NotFound: 404 Not Found>
[----] E, [2017-08-15T12:07:34.617183 #15777:87d14c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshParser#download_template) Failed to download Azure template <REDACTED> Reason: #<RestClient::NotFound: 404 Not Found>
[----] I, [2017-08-15T12:07:49.424380 #6174:87d14c]  INFO -- : Q-task_id([log_status]) MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker#log_status) [Refresh Worker for Providers: Azure-USEast] Worker 
ID [10000001094878], PID [15777], GUID [8359f224-81d3-11e7-b79a-00505690684c], Last Heartbeat [2017-08-15 16:05:25 UTC], Process Info: Memory Usage [536494080], Memory Size [914935808], Proportional Set Size: [4
19259000], Memory % [4.32], CPU Time [1221.0], CPU % [0.0], Priority [27]
[----] I, [2017-08-15T12:12:44.412807 #6174:87d14c]  INFO -- : Q-task_id([log_status]) MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker#log_status) [Refresh Worker for Providers: Azure-USEast] Worker 
ID [10000001094878], PID [15777], GUID [8359f224-81d3-11e7-b79a-00505690684c], Last Heartbeat [2017-08-15 16:05:25 UTC], Process Info: Memory Usage [1851252736], Memory Size [2229260288], Proportional Set Size: 
[1705003000], Memory % [14.89], CPU Time [26348.0], CPU % [0.08], Priority [27]
[----] W, [2017-08-15T12:15:18.408088 #5959:87d14c]  WARN -- : MIQ(MiqServer#validate_worker) Worker [ManageIQ::Providers::Azure::CloudManager::RefreshWorker] with ID: [10000001094878], PID: [15777], GUID: [8359
f224-81d3-11e7-b79a-00505690684c] process memory usage [2312689000] exceeded limit [2147483648], requesting worker to exit

Azure all_valid_ems_in_zone does an Azure API call every sync_workers loop

The Azure Provider only starts event_catchers if the Microsoft.Insights resource provider is registered. This is done by calling insights? which is called by the supports?(:timeline) supports feature block.

The problem arises when the Azure::CloudManager::EventCatcher overrides all_valid_ems_in_zone to check supports?(:timeline).

This looks innocuous but it is actually doing an Azure API call every time Azure::CloudManager::EventCatcher.all_valid_ems_in_zone is called. This is done by the PerEmsWorkerMixin.sync_workers method (by way of desired_queue_names).

Long story short we're doing an Azure API call from the MiqServer to check this feature every 15 seconds.

Unmanaged storage account lookup failure with targeted refresh

When performing a targeted refresh, unmanaged storage accounts are not necessarily going to be found. At the moment the ref is using the attached VM's resource group instead of the storage account's resource group. If those happen to be different then the lookup will fail.

The end result is that not only will the unmanaged storage account not be refreshed, but its mode will get set to null. You can see we don't ensure that storage_acct is defined first here:

https://github.com/ManageIQ/manageiq-providers-azure/blob/master/app/models/manageiq/providers/azure/inventory/parser/cloud_manager.rb#L190

A quick fix would be to make a check for storage_acct first. The better fix would be to set the ref properly.

Refresh parser should set resource group where possible

Currently the refresh parser isn't setting the resource group relationship for any resources (except stacks, which uses a string).

We should be setting the relationship so that we can get that information internally via ActiveRecord when needed.

The api-version for each service should be locked down

Each service class in the azure-armrest gem has its own api-version. This "api-version=" string is attached to every http request to Azure. By default it will use the latest non-preview version string.

Normally this is fine. However, on rare occasion Azure does alter the behavior of the request, typically by changing the output format. In such situations fixing it for a customer in the field would be difficult.

Therefore I think the api-version strings should be locked down to whichever version is being used while in testing and development prior to a major release. The easiest thing to do would be to store the api-version string in the config/settings.yml file. It will require a setting for each service class. Those that aren't set would default to the current api-version or use a default setting.

I'm thinking something along the lines of this:

:api_versions:
  :default: 2016-03-30
  :virtual_machine: 2017-03-30
  :ip_address: 2017-03-30
  :network_interface: 2017-04-01
  :storage: 2016-01-01

Add StorageManager?

Currently Azure supports two types of storage - Managed and Unmanaged, but we have no StorageManager to represent this in any way.

While we store disk information, we don't store overall storage information. This currently makes it difficult to distinguish VM's within the app that are managed versus those that are unmanaged.

Ideally, I should be able to tell by doing something like:

vm.storage.store_type

Is this something we should add?

Marketplace Image Support

As talked about here I'm opening an RFE for marketplace image support. From what I can see this will require:

  • Altering prepare_for_clone_task to parse and accept marketplace URNs
  • Adding a method to pull marketplace images to the refresh parser
  • Maybe adding something akin to the public image settings and filter found in the Amazon provider?

Related to ManageIQ/azure-armrest#288. Opening an issue on this project to reference in forthcoming PR

Raise error if VM is generalized when requesting vm to start

When running a HTTP request for ManageIQ to start a VM in azure will respond with a successful run when checking the status of task. But the VM will not be turned on since the VM is a generalized (image template) and cannot be started.

Error in the evm.log show the error this should be raised to ensure people know the VM is not actually started

[----] E, [2017-08-16T11:10:07.643788 #2065:2af844831140] ERROR -- : MIQ(MiqQueue#m_callback) Message id: [1078]: [OperationNotAllowed] Operation 'start' is not allowed on VM 'USTest' since the VM is generalized. (cause: 409 Conflict)
[----] E, [2017-08-16T11:10:07.644041 #2065:2af844831140] ERROR -- : MIQ(MiqQueue#m_callback) backtrace: /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/azure-armrest-0.7.5/lib/azure/armrest/armrest_service.rb:252:in `raise_api_exception'

Config options for API versions in Azure Stack

Azure stack can have a different set of API versions compared to the Azure public clouds. We'll need a way to either discover or set different versions for an Azure Stack provider and make sure that it does not interfere with the Azure providers.

Thoughts on this from @djberg96

The library will automatically determine the latest api-version for each service. For manageiq, we'll need to decide on the semantics. At the moment it looks like this:

:ems:
  :ems_azure:
    :api_versions:
      :availability_set: "2017-12-01"
      :ip_address: "2017-11-01"

As I mentioned on the call, I'm thinking we change it to something like this:

:ems:
  :ems_azure:
    :default
      :api_versions:
        :availability_set: "2017-12-01"
        :ip_address: "2017-11-01"
    :your_provider
        :availability_set: "2016-11-30" # or whatever
        :ip_address: "2016-07-30" # ditto

If the provider name matches a config entry, we use those api-version strings. Otherwise we use the default settings.

Undefined method private_ip_address

Saw this in the log today:

[----] E, [2017-06-22T10:44:55.429941 #82338:3fcfa103f7f4] ERROR -- : [NoMethodError]: undefined method `private_ip_address' for #<Azure::Armrest::Network::NetworkInterface::Properties::IpConfiguration::Properties:0x007f9f3303e9c0>
Did you mean?  private_ip_address_version  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2017-06-22T10:44:55.430104 #82338:3fcfa103f7f4] ERROR -- : /Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:393:in `parse_cloud_subnet_network_port'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:401:in `block in parse_network_port'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:400:in `map'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:400:in `parse_network_port'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:99:in `block in get_network_ports'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/refresh_helper_methods.rb:10:in `block in process_collection'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/refresh_helper_methods.rb:9:in `each'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/refresh_helper_methods.rb:9:in `process_collection'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:99:in `get_network_ports'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:33:in `ems_inv_to_hashes'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresh_parser.rb:8:in `ems_inv_to_hashes'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-providers-azure-add872d4adfc/app/models/manageiq/providers/azure/network_manager/refresher.rb:6:in `parse_legacy_inventory'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:146:in `block in parse_targeted_inventory'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:146:in `parse_targeted_inventory'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `block in refresh_targets_for_ems'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh'
/Users/dberger/Repositories/manageiq-djberg96/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh.rb:97:in `block in refresh'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh.rb:96:in `each'
/Users/dberger/Repositories/manageiq-djberg96/app/models/ems_refresh.rb:96:in `refresh'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue.rb:328:in `block in deliver'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `catch'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `catch'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue.rb:324:in `deliver'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue_worker_base/runner.rb:107:in `deliver_queue_message'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue_worker_base/runner.rb:135:in `deliver_message'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue_worker_base/runner.rb:153:in `block in do_work'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue_worker_base/runner.rb:147:in `loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_queue_worker_base/runner.rb:147:in `do_work'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker/runner.rb:343:in `block in do_work_loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker/runner.rb:340:in `loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker/runner.rb:340:in `do_work_loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker/runner.rb:161:in `run'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker/runner.rb:135:in `start'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker/runner.rb:22:in `start_worker'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker.rb:345:in `block in start_runner_via_fork'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker.rb:343:in `start_runner_via_fork'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker.rb:338:in `start_runner'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker.rb:371:in `start'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_worker.rb:266:in `start_worker'
/Users/dberger/Repositories/manageiq-djberg96/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/Users/dberger/Repositories/manageiq-djberg96/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/Users/dberger/Repositories/manageiq-djberg96/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/Users/dberger/Repositories/manageiq-djberg96/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:346:in `block in monitor'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:346:in `monitor'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:368:in `block (2 levels) in monitor_loop'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/Users/dberger/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/bundler/gems/manageiq-gems-pending-23fd9dd038d3/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:368:in `block in monitor_loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:367:in `loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:367:in `monitor_loop'
/Users/dberger/Repositories/manageiq-djberg96/app/models/miq_server.rb:250:in `start'
/Users/dberger/Repositories/manageiq-djberg96/lib/workers/evm_server.rb:27:in `start'
/Users/dberger/Repositories/manageiq-djberg96/lib/workers/evm_server.rb:48:in `start'
/Users/dberger/Repositories/manageiq-djberg96/lib/workers/bin/evm_server.rb:4:in `<main>'

Add specs for event_parser.rb

We are currently at 0% coverage for event_parser.rb. The Amazon provider has some specs we can probably use as a baseline.

StandardSSD_LRS managed disks break refresh

Managed disks with the SKU StandardSSD_LRS seem to break Azure refreshes. From what I can tell this affects all MIQ versions from Fine-4 to the Hammer 1.1 beta, this stack trace is from Hammer-1-beta1.1.

I opened this here and not on azure-armrest because I wrote a short script to list all managed disks and their SKUs and it did not throw an error on the StandardSSD_LRS disk. I'm not sure what in the Azure provider specifically is causing this, but I'll keep looking.

Automatic placement can fail with Azure if required resources aren't available

Currently if you try to use automatic placement during provisioning it can fail with:

[----] I, [2020-01-08T12:55:55.783175 #16250:5b8f60]  INFO -- : Q-task_id([r7000000000014_miq_provision_7000000000014]) Starting Phase <provision_error>
[----] E, [2020-01-08T12:55:55.798704 #16250:5b8f60] ERROR -- : Q-task_id([r7000000000014_miq_provision_7000000000014]) MIQ(ManageIQ::Providers::Azure::CloudManager::Provision#provision_error) [[NoMethodError]: undefined method `ems_ref' for nil:NilClass] encountered during phase [prepare_provision]
[----] E, [2020-01-08T12:55:55.798786 #16250:5b8f60] ERROR -- : Q-task_id([r7000000000014_miq_provision_7000000000014]) /opt/rh/cfme-gemset/bundler/gems/cfme-providers-azure-efc7b57d34b3/app/models/manageiq/providers/azure/cloud_manager/provision/cloning.rb:166:in 'build_nic_options'

Where line 166 of cloning.rb is :subnet => {:id => cloud_subnet.ems_ref}, but there is apparently no associated cloud_subnet.

https://gist.github.com/lfu/28a252599b3b7585638f8cc5ed323604

Originally spotted by @lfu

As per @gmcculloug the best placement code is at https://github.com/ManageIQ/manageiq-content/blob/master/content/automate/ManageIQ/Cloud/VM/Provisioning/Placement.class/__methods__/best_fit_azure.rb

For this particular issue we can update cloning.rb to create a cloud subnet if one isn't available.

Must Create RGs per-region

During VM provisioning, the only allowed RGs are those in the Azure provider, but because the Azure providers are region specific, one must create resource groups for every region. When you have resource groups defined per-app, which is MS recommendation, this doesn't really make sense as the app may be multi-regional, and Azure does not intend for you to make RGs region specific.

I think this could be fixed by altering allowed_resource_groups in provision_workflow to fetch all the resource groups from other providers with matching subscription IDs, in addition to the current one. Thoughts?

Missing Tenant ID when creating Provider using API

When creating an Azure provider using the API which includes via Ansible,
the tenant_id doesn't get set. As we're actually storing Azure's tenant_id in uid_ems internally.

So it should be:

miq.providers.create({
         :type => type,
         :provider_region => region,
         :name => name,
         :uid_ems => tenant_id,
         :subscription => subscription,
         :credentials => {
           :userid => userid,
           :password => password
          }
        })

not

miq.providers.create({
         :type => type,
         :provider_region => region,
         :name => name,
         :tenant_id => tenant_id,
         :subscription => subscription,
         :credentials => {
           :userid => userid,
           :password => password
          }
        })

screen shot 2018-01-10 at 17 26 56

Refresh fails with error: "\xEF" from ASCII-8BIT to UTF-8, podified

Hi,

when using the Azure provider from ManageIQ Kasparov-1 the refresh is always failing with this error:
"\xEF" from ASCII-8BIT to UTF-8. We are using the podified version running on Kubernetes.

I also tried the azure provider on a Jansa appliance. Here the refresh works without issues.
Then I also tried the azure provider on a Kasparov-1 appliance. Here the refresh works without issues.

I'm not sure what would be the difference between the podified and appliance, perhaps the logging configuration /var/www/miq/vmdb/lib/vmdb/loggers/container_logger.rb:43?

[----] I, [2021-03-30T21:14:17.610988 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::Inventory::Collector::CloudManager#collect_inventory) Retrieving Managed Images...
[----] I, [2021-03-30T21:14:17.663854 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::Inventory::Collector::CloudManager#collect_inventory) Retrieving Managed Images...Complete - Count [0]
[----] I, [2021-03-30T21:14:17.684629 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::Inventory::Collector::CloudManager#collect_inventory) Retrieving Private Images...
[----] E, [2021-03-30T21:14:17.980883 #7:2b25bdd0597c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [test2], id: [5] Refresh failed
[----] E, [2021-03-30T21:14:17.980991 #7:2b25bdd0597c] ERROR -- : [Encoding::UndefinedConversionError]: "\xEF" from ASCII-8BIT to UTF-8  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2021-03-30T21:14:17.981067 #7:2b25bdd0597c] ERROR -- : /opt/manageiq/manageiq-gemset/gems/json-2.5.1/lib/json/common.rb:312:in `generate'
/opt/manageiq/manageiq-gemset/gems/json-2.5.1/lib/json/common.rb:312:in `generate'
/var/www/miq/vmdb/lib/vmdb/loggers/container_logger.rb:43:in `call'
/usr/share/ruby/logger.rb:582:in `format_message'
/usr/share/ruby/logger.rb:472:in `add'
/opt/manageiq/manageiq-gemset/gems/activerecord-session_store-1.1.3/lib/active_record/session_store/extension/logger_silencer.rb:38:in `add_with_threadsafety'
/opt/manageiq/manageiq-gemset/gems/activesupport-5.2.4.4/lib/active_support/logger.rb:27:in `block (2 levels) in broadcast'
/usr/share/ruby/logger.rb:534:in `warn'
/var/www/miq/vmdb/lib/vmdb/logging.rb:25:in `block (2 levels) in <class:LogProxy>'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-e483ca2543bf/app/models/manageiq/providers/azure/inventory/collector/cloud_manager.rb:38:in `rescue in images'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-e483ca2543bf/app/models/manageiq/providers/azure/inventory/collector/cloud_manager.rb:35:in `images'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-e483ca2543bf/app/models/manageiq/providers/azure/inventory/parser/cloud_manager.rb:402:in `images'
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-e483ca2543bf/app/models/manageiq/providers/azure/inventory/parser/cloud_manager.rb:18:in `parse'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:42:in `block in parse'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:39:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/inventory.rb:39:in `parse'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:125:in `parse_targeted_inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:92:in `block in refresh_targets_for_ems'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:56:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:91:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block (2 levels) in refresh'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:20:in `realtime_store'
/opt/manageiq/manageiq-gemset/gems/more_core_extensions-4.3.0/lib/more_core_extensions/core_ext/benchmark/realtime_store.rb:62:in `realtime_block'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:41:in `block in refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:31:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:71:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:70:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:70:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:484:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:482:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:360:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:80:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:122:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:261:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:261:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:113:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:95:in `start'
/var/www/miq/vmdb/lib/workers/bin/run_single_worker.rb:122:in `<main>'
[----] E, [2021-03-30T21:14:17.981115 #7:2b25bdd0597c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [test2], id: [5] Unable to perform refresh for the following targets:
[----] E, [2021-03-30T21:14:17.981189 #7:2b25bdd0597c] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh)  --- ManageIQ::Providers::Azure::CloudManager [test2] id [5]
[----] I, [2021-03-30T21:14:18.134706 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2021-03-30T21:14:18.134874 #7:2b25bdd0597c] ERROR -- : MIQ(MiqQueue#deliver) Message id: [1199757], Error: ["\xEF" from ASCII-8BIT to UTF-8]
[----] E, [2021-03-30T21:14:18.134951 #7:2b25bdd0597c] ERROR -- : [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: "\xEF" from ASCII-8BIT to UTF-8  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2021-03-30T21:14:18.135051 #7:2b25bdd0597c] ERROR -- : /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:67:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:11:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:71:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:70:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:70:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:484:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:482:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:360:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:459:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:80:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:122:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:118:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:264:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:261:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:261:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:113:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:95:in `start'
/var/www/miq/vmdb/lib/workers/bin/run_single_worker.rb:122:in `<main>'
[----] I, [2021-03-30T21:14:18.135130 #7:2b25bdd0597c]  INFO -- : MIQ(MiqQueue#delivered) Message id: [1199757], State: [error], Delivered in [11.536006695] seconds
[----] I, [2021-03-30T21:43:51.265421 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker::Runner#config_out_of_date?) MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker::Runner) Configuration has changed, New TS: 2021-03-30 21:43:41 UTC, Old TS: 2021-03-30 21:14:06 UTC
[----] I, [2021-03-30T21:43:51.265523 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker::Runner#heartbeat) MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker::Runner) Synchronizing configuration...
[----] I, [2021-03-30T21:43:51.386556 #7:2b25bdd0597c]  INFO -- : MIQ(ManageIQ::Providers::Azure::CloudManager::RefreshWorker::Runner#sync_config) ID [430], PID [7], GUID [6ad909fc-59ce-48c5-b5b4-e31f24c06793], Zone [default], Active Roles [automate,database_operations,ems_inventory,ems_operations,event,remote_console,reporting,scheduler,smartstate,user_interface,web_services], Assigned Roles [automate,database_operations,database_owner,ems_inventory,ems_operations,event,remote_console,reporting,scheduler,smartstate,user_interface,web_services], Configuration:
[----] I, [2021-03-30T21:43:51.386854 #7:2b25bdd0597c]  INFO -- : 

UAE regions are missing

Seems we need to refresh this.

I had a side-thought...I like that we come out of the box with pre-known regions. When we do get credentials, however, if we find they are good, could we query for the regions list and merge it into the pre-known regions list? Or is that not possible because they are on different workers?

Azure Stack flavor names are case sensitive

During Azure Stack testing, I tried to deploy a a private image and hit this error:

[----] E, [2018-05-07T12:27:03.581936 #5157:121f114] ERROR -- : Q-task_id([miq_provision_1]) MIQ(ManageIQ::Providers::Azure::CloudManager::Provision#provision_error) [[Azure::Armrest::BadRequestException]: The requested VM size standard_a1 is not available in the current region. The sizes available in the current region are: Basic_A0,Basic_A1,Basic_A2,Basic_A3,Basic_A4,Standard_A0,Standard_A1,Standard_A2,Standard_A3,Standard_A4,Standard_A5,Standard_A6,Standard_A7,Standard_D1,Standard_D2,Standard_D3,Standard_D4,Standard_D11,Standard_D12,Standard_D13,Standard_D14,Standard_D1_v2,Standard_D2_v2,Standard_D3_v2,Standard_D4_v2,Standard_D5_v2,Standard_D11_v2,Standard_D12_v2,Standard_D13_v2,Standard_D14_v2,Standard_DS1,Standard_DS2,Standard_DS3,Standard_DS4,Standard_DS11,Standard_DS12,Standard_DS13,Standard_DS14,Standard_DS1_v2,Standard_DS2_v2,Standard_DS3_v2,Standard_DS4_v2,Standard_DS5_v2,Standard_DS11_v2,Standard_DS12_v2,Standard_DS13_v2,Standard_DS14_v2. Find out more on the available VM sizes in each region at https://aka.ms/azure-regions.] encountered during phase [start_clone_task]

Requested vm size was 'standard_a1', and Standard_A1 is in the list. It appears that the flavor name is case sensitive for the Azure Stack APIs.

We could get around this by only storing the uid as flavor.name.downcase and keeping the name as-is (see master...bsquizz:flavor_names), however I'm not sure if altering the display name will affect anything existing customers are doing with Azure providers. If that's the case, we may need to store another attribute (i.e. actual_name) alongside the downcase'd name and have prepare_for_clone_task in provision/cloning.rb use the actual_name for vmSize

SSA failure during perform_metadata_scan

Error with SSA running docker versions: manageiq/manageiq:jansa-3, latest-jansa, latest-kasparov.
SSA worked on ivanchuk-8

Logs from testing with latest-jansa:
{"@timestamp":"2021-03-08T19:42:22.571355 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"debug","message":"Q-task_id([job_dispatcher]) MIQ(MiqServer#scan_metadata) ansible-vm3 (ManageIQ::Providers::Azure::CloudManager::Vm)"}
{"@timestamp":"2021-03-08T19:42:23.140106 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"debug","message":"Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Azure::CloudManager::Vm#perform_metadata_scan) jstdebug2: in perform_meta
data_scan"}
{"@timestamp":"2021-03-08T19:42:23.140337 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"debug","message":"Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Azure::CloudManager::Vm#perform_metadata_scan) name: ansible-vm3 (templat
e = false)"}
{"@timestamp":"2021-03-08T19:42:23.140452 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"debug","message":"Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Azure::CloudManager::Vm#perform_metadata_scan) jstdebug2: set vm_args res
ource_group.name"}
{"@timestamp":"2021-03-08T19:42:23.179371 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"debug","message":"Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Azure::CloudManager::Vm#perform_metadata_scan) jstdebug2: set vm_args ost.scanData ost: #\u003cOpenStruct args=["eastus", {"ems"=\u003e{"ems"=\u003e{:hostname=\u003enil, :ipaddress=\u003enil, :username=\u003e"4f7b1e0f-8d60-4c7c-8978-8e4c62fa7d77", :password=\u003e"v2:{5yNsUngLLdXAahqvhCCQa6I/6Bp9s7Ln7iPOJApeyx/FHbBsdEc5K4euUQJJRfBtOSAb6smsj9R62ecNEHmIUOW+BLfXPu/riflLqT25VWI=}", :class_name=\u003e"ManageIQ::Providers::Azure::CloudManager"}, "connect_to"=\u003e"host"}, "vmScanProfiles"=\u003e[]}], method_name="scan_metadata", vm_guid="7ddea001-3b3b-4c10-a8f9-cd60c8d98542", category="vmconfig,accounts,software,services,system", taskid="4edcda4f-b853-47f0-91ce-b7987fab5c0a", target_id=3, target_type="VmOrTemplate", scanData={"ems"=\u003e{"ems"=\u003e{:hostname=\u003enil, :ipaddress=\u003enil, :username=\u003e"4f7b1e0f-8d60-4c7c-8978-8e4c62fa7d77", :password=\u003e"v2:{5yNsUngLLdXAahqvhCCQa6I/6Bp9s7Ln7iPOJApeyx/FHbBsdEc5K4euUQJJRfBtOSAb6smsj9R62ecNEHmIUOW+BLfXPu/riflLqT25VWI=}", :class_name=\u003e"ManageIQ::Providers::Azure::CloudManager"}, "connect_to"=\u003e"host"}, "vmScanProfiles"=\u003e[]}, jobid=9, config=#\u003cOpenStruct vmdb=true, forceFleeceDefault=true, capabilities={:vixDisk=\u003efalse}\u003e\u003e"}
{"@timestamp":"2021-03-08T19:42:23.180329 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"err","message":"Q-task_id([job_dispatcher]) MIQ(MiqServer#scan_metadata) undefined method '[]' for nil:NilClass"}

Added debug statements to:
/opt/manageiq/manageiq-gemset/bundler/gems/manageiq-providers-azure-e7914f85e845/app/models/manageiq/providers/azure/cloud_manager/vm_or_template_shared/scanning.rb

def perform_metadata_scan(ost)
    require 'MiqVm/miq_azure_vm'

    vm_args = { :name => name }
    _log.debug("jstdebug2: in perform_metadata_scan")
    _log.debug("name: #{name} (template = #{template})")
    if template
      if managed_image?
        vm_args[:resource_group] = managed_resource_group
        vm_args[:managed_image]  = managed_image_disk_name
      elsif blob_uri
        vm_args[:image_uri] = blob_uri
      else
        vm_args[:image_uri] = uid_ems
      end
    else
      _log.debug("jstdebug2: set vm_args resource_group.name")
      vm_args[:resource_group] = resource_group.name
      #vm_args[:resource_group] = "myapp-rg"
      _log.debug("jstdebug2: set vm_args ost.scanData ost: #{ost}")
      vm_args[:snapshot] = ost.scanData["snapshot"]["name"]
      _log.debug("jstdebug2: vm_args = #{vm_args}")
    end
    _log.debug("jstdebug2: set ost.scantime")
    ost.scanTime = Time.now.utc unless ost.scanTime
    _log.debug("jstdebug2: set armrest")
    armrest      = ext_management_system.connect

seems to fail on:
vm_args[:snapshot] = ost.scanData["snapshot"]["name"]
because I never see:
_log.debug("jstdebug2: vm_args = #{vm_args}")

Note: the Azure snapshot gets created but fails while scanning.

ost object in the log:
{"@timestamp":"2021-03-08T19:42:23.179371 ","hostname":"a2569443fbda","pid":3970,"tid":"2ac3751af94c","level":"debug","message":"Q-task_id([job_dispatcher]) MIQ(ManageIQ::Providers::Azure::CloudManager::Vm#perform_metadata_scan) jstdebug2: set vm_args ost.scanData ost: #\u003cOpenStruct args=["eastus", {"ems"=\u003e{"ems"=\u003e{:hostname=\u003enil, :ipaddress=\u003enil, :username=\u003e"4f7b1e0f-8d60-4c7c-8978-8e4c62fa7d77", :password=\u003e"v2:{5yNsUngLLdXAahqvhCCQa6I/6Bp9s7Ln7iPOJApeyx/FHbBsdEc5K4euUQJJRfBtOSAb6smsj9R62ecNEHmIUOW+BLfXPu/riflLqT25VWI=}", :class_name=\u003e"ManageIQ::Providers::Azure::CloudManager"}, "connect_to"=\u003e"host"}, "vmScanProfiles"=\u003e[]}], method_name="scan_metadata", vm_guid="7ddea001-3b3b-4c10-a8f9-cd60c8d98542", category="vmconfig,accounts,software,services,system", taskid="4edcda4f-b853-47f0-91ce-b7987fab5c0a", target_id=3, target_type="VmOrTemplate", scanData={"ems"=\u003e{"ems"=\u003e{:hostname=\u003enil, :ipaddress=\u003enil, :username=\u003e"4f7b1e0f-8d60-4c7c-8978-8e4c62fa7d77", :password=\u003e"v2:{5yNsUngLLdXAahqvhCCQa6I/6Bp9s7Ln7iPOJApeyx/FHbBsdEc5K4euUQJJRfBtOSAb6smsj9R62ecNEHmIUOW+BLfXPu/riflLqT25VWI=}", :class_name=\u003e"ManageIQ::Providers::Azure::CloudManager"}, "connect_to"=\u003e"host"}, "vmScanProfiles"=\u003e[]}, jobid=9, config=#\u003cOpenStruct vmdb=true, forceFleeceDefault=true, capabilities={:vixDisk=\u003efalse}\u003e\u003e"}

Azure Stack

Below are the changes required to get the Azure Stack refresh working using the existing Azure Public cloud provider implementation:

  • Azure stacks will pin to specific ARM API versions while the Public Cloud provider will use the latest version. Ensure backward compatibility for stacks on older versions.
  • Remove downcase of the flavor name for provisioning to work
  • Refactor raw connect to support custom endpoint
  • UI change to support custom endpoint

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions
.github/workflows/ci.yaml
  • actions/checkout v4
  • ruby/setup-ruby v1
  • paambaati/codeclimate-action v8
  • manageiq/postgresql 13

  • Check this box to trigger a request for Renovate to run again on this repository

Azure Provider Last Refresh Error

Hi, we are getting an error in the container it failed to encode ASCII-8BIT to UTF-8. These issues occur only in Azure Provider

Failed to encode

"\xEF" from ASCII-8BIT to UTF-8

[----] E, [2021-01-29T08:56:34.850698 #12923:10525fc] ERROR -- : [Encoding::UndefinedConversionError]: "\xEF" from ASCII-8BIT to UTF-8 Method:[block (2 levels) in class:LogProxy]
[----] E, [2021-01-29T08:56:34.989466 #12923:10525fc] ERROR -- : MIQ(MiqQueue#deliver) Message id: [208061], Error: ["\xEF" from ASCII-8BIT to UTF-8]
[----] E, [2021-01-29T08:56:34.990272 #12923:10525fc] ERROR -- : [ManageIQ::Providers::BaseManager::Refresher::PartialRefreshError]: "\xEF" from ASCII-8BIT to UTF-8 Method:[block (2 levels) in <class:LogProxy

Warning - uninitialized constant Azure::Armrest::UnauthorizedException

Somehow I'm seeing this when I try to add an Azure provider:

[----] W, [2017-03-13T10:47:26.740435 #4483:3fda9c03fa14]  WARN -- : MIQ(ManageIQ::Providers::Azure::CloudManager#authentication_check_no_validation) type: [:default] for [5] [azure] Validation failed: error, uninitialized constant Azure::Armrest::UnauthorizedException
[----] W, [2017-03-13T10:47:26.740707 #4483:3fda9c03fa14]  WARN -- : MIQ(AuthUseridPassword#validation_failed) [ExtManagementSystem] [5], previously valid on: , previous status: []

I'm not sure how this can happen since it clearly is defined:

irb(main):001:0> require 'azure-armrest'
=> true
irb(main):002:0> Azure::Armrest::UnauthorizedException
=> Azure::Armrest::UnauthorizedException

Is there some kind of load ordering issue?

503 while exporting templates

I'm experiencing 503 errors during refresh for a couple of my subscriptions, one consistently and another intermittently. I found the stack traces for the errors, and they're identical for both subscriptions; they're posted here

It doesn't appear that there is anything wrong with the deployment templates that the refresher is attempting to export; I am able to export them without error from, e.g. Resource Explorer

If I can help with troubleshooting this, please let me know.

A failing Azure SSA scan doesn't delete the snapshot

When an Azure SSA scan fails the process_abort doesn't cleanup the snapshot, causing future scans to fail with "Failed to create evm snapshot" "Another SSA request for this VM is in progress or a previous one failed to clean up properly."

get_stack_templates failure

From http://talk.manageiq.org/t/azure-refresh-issues/2462/5

[----] E, [2017-06-22T11:25:39.594698 #23941:6f5130] ERROR -- : MIQ(ManageIQ::Providers::Azure::CloudManager::Refresher#refresh) EMS: [Azure Production 1 - North Central US], id: [14] Refresh failed
[----] E, [2017-06-22T11:25:39.595763 #23941:6f5130] ERROR -- : [NoMethodError]: undefined method `each' for nil:NilClass  Method:[rescue in block in refresh]
[----] E, [2017-06-22T11:25:39.595943 #23941:6f5130] ERROR -- : /opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-ba9fc3f9e10c/app/models/manageiq/providers/azure/cloud_manager/refresh_parser.rb:137:in `get_stack_templates'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-ba9fc3f9e10c/app/models/manageiq/providers/azure/cloud_manager/refresh_parser.rb:44:in `ems_inv_to_hashes'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-ba9fc3f9e10c/app/models/manageiq/providers/azure/cloud_manager/refresh_parser.rb:10:in `ems_inv_to_hashes'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-providers-azure-ba9fc3f9e10c/app/models/manageiq/providers/azure/cloud_manager/refresher.rb:6:in `parse_legacy_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:146:in `block in parse_targeted_inventory'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:146:in `parse_targeted_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `block in refresh_targets_for_ems'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:91:in `block in timeout'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `block in catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:33:in `catch'
/opt/rubies/ruby-2.3.1/lib/ruby/2.3.0/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:135:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:153:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:341:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:338:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:338:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:160:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:134:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:339:in `block in start_runner'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:337:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:348:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:266:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:349:in `block in monitor'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:349:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:371:in `block (2 levels) in monitor_loop'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/bundler/gems/manageiq-gems-pending-22f8dd86d6e8/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:371:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:253:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.