hawkular / hawkular-client-ruby Goto Github PK

To include the hawkular metrics client at the moment we use:

require 'metrics/metrics_client'

Since metrics is a generic term, I think this should be scoped instead with the hawkular name, e.g.:

require 'hawkular/metrics_client'

cc @pilhuhn @abonas @Fryguy

Hawkular::Metrics::Client.new(entrypoint, credentials, options).strings.get_data_by_tags(:miq_metric => true)

will work, but i need it to work without the strings section.
cc @pilhuhn @zgalor @simon3z.

There are going to be cross tenant apis for alerting e.g get alerts for all tenants.
We would need to have some support for those in clients, currently created with one tenant in mind.

Websockets are currently not offline-tested as VCR can't do that.

"rspec ./spec/integration/metric_spec.rb:100 # metrics_0_16_0 NonSecure Tenants Should create and return tenant" failure

"The first endpoint to be protected by this mechanism is /tenants since its functionality is cross tenant"

cc @yaacov - this will be a change in h-metrics for ose 3.4

See https://issues.jboss.org/browse/HWKMETRICS-420

This adds 'admin' availability type. Also sub-state may be represented via tags on the data points, the client needs to be able to retrieve them as well.

Also the availability bucket query now returns that new field + 'unknown' See
hawkular/hawkular-metrics#522

This will only work with metrics 0.16+ (needs to be mentioned in RDoc)

HWKMETRICS-24 makes (incompatible) changes to some metrics endpoints.
The old endpoints still exist for a while, but will go away.

We may check if h-metrics has a working version endpoint and
then decide which endpoint to call according to the version.

We have several places in the code where we default to localhost:8080 if no input value is given.
E.g.

OperationsClient:initialize:

      args[:host] ||= 'localhost:8080'

Now if the calling code does not supply the :host, we use the default, which means that the caller does not get an indication that it did something wrong. Later in production (or Demos :-) the code silently fails with obscure error messages.

Inventory has an "official" way of storing operation parameters within operation types - a child of the operation type which is a data entity whose structural data is named "parameterTypes"

The Ruby client needs to get operation parameters using this mechanism, rather than the ad-hoc way we were doing it (via general properties).

The WildFly agent is creating both (the "official" way and the general props way) for backward compatibility, but storing these in general props is deprecated and will go away, see JIRA HWKAGENT-130

The new way will have inventory show you the params like this:

URL (notice "/d;parameterTypes"): http://127.0.0.1:8080/hawkular/inventory/entity/f;feed_id/rt;WildFly%20Server/ot;Shutdown/d;parameterTypes

{
  "path": "/t;hawkular/f;feed_id/rt;WildFly%20Server/ot;Shutdown/d;parameterTypes",
  "name": "parameterTypes",
  "identityHash": "1ad5d5963d6978975e958de5abcb49f778695f",
  "contentHash": "5ff8951bc6a6ab6483f69ec0ab139d5a76e22c",
  "syncHash": "27163827304a2c50e31b9ce2ee74e5e32f396d58",
  "value": {
    "timeout": {
      "type": "int",
      "description": "Timeout in seconds to allow active connections to drain",
      "defaultValue": "0",
      "required": false
    },
    "restart": {
      "type": "bool",
      "description": "Should the server be restarted after shutdown?",
      "defaultValue": "false",
      "required": false
    }
  }
}

@jmazzitelli and I saw this fly by:

Exception in thread "JavaFX Application Thread" org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `[]' for nil:NilClass
    at RUBY.initialize(/Users/hrupp/.rvm/gems/jruby-9.1.2.0/gems/hawkular-client-2.4.0/lib/hawkular/inventory/entities.rb:122)
    at RUBY.block in list_operation_definitions(/Users/hrupp/.rvm/gems/jruby-9.1.2.0/gems/hawkular-client-2.4.0/lib/hawkular/inventory/inventory_api.rb:418)
    at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1593)
    at RUBY.list_operation_definitions(/Users/hrupp/.rvm/gems/jruby-9.1.2.0/gems/hawkular-client-2.4.0/lib/hawkular/inventory/inventory_api.rb:417)
    at RUBY.list_operation_definitions_for_resource(/Users/hrupp/.rvm/gems/jruby-9.1.2.0/gems/hawkular-client-2.4.0/lib/hawkular/inventory/inventory_api.rb:429)
    at RUBY.add_operations(/Users/hrupp/src/hawkfx/lib/on_click_cell_factory.rb:130)
    at RUBY.block in initialize(/Users/hrupp/src/hawkfx/lib/on_click_cell_factory.rb:51)

Looking at the code shows param_list = op_hash['properties']['params'],
so it looks like the received entity sometimes has no properties (or worse: H-inventory sends us some empty hashes).

We now have websocket support in several places.
They should all be set up from the credentials passed to the unified client.

Only a single websocket should be setup between Hawkular server and the client gem and should
be (re)used by all the communication.

See https://github.com/hawkular/hawkular-client-ruby/blob/master/lib/hawkular/hawkular_client.rb#L61

  Operations::OperationsClient.new(entrypoint: @state[:entrypoint].gsub(/^https?/, 'ws'),

Allow to find operations/operation definitions for a resource (type?) in inventory

As discussed in #36 the current impl's set of parameters is overloaded and we should rewrite to pass all optional params in a hash.

The Token API (tokens_api) seems to be gone. We should remove it from the client.

josejulio: Does secret-store/v1/tokens endpoint still exists?
pilhuhn: don't think so
pilhuhn: the whole token thing went away when we moved to hawkular-services with the deployment

// cc @pilhuhn

Availability has a parameter to only return distinct values
See https://github.com/hawkular/hawkular-metrics/blob/0.8.2/api/metrics-api-jaxrs/src/main/java/org/hawkular/metrics/api/jaxrs/handler/AvailabilityHandler.java#L243

this should be reflected in the availability part of metrics-sub-client

Also it has no percentiles.

// cc @simon3z

The agent will provide parameters for operation definitions once hawkular/hawkular-agent#229 of https://issues.jboss.org/browse/HWKAGENT-87 is merged.

We should support this and return them.

There is a branch at https://github.com/pilhuhn/hawkular-client-ruby/tree/complete-ops-definitions that has most of the code, but where the test fails with rspec (but passes from within the IDE)

[16:16:47] that seems a change on the API and the ruby-client has detected it
[16:17:04] ys, we no longer have ackBy on the Alert
[16:17:15] (at least the alerts one, the metrics one no idea)
[16:17:21] it has been replaced with the lifecycle list

cc @jshaughn @lucasponce

If you run

    it 'Should err on bad credentials' do
      @creds = {
          username: '-XX-X-jdoe-X',
          password: 'password'
      }
      @client = Hawkular::Inventory::InventoryClient.create(credentials: @creds)

      @client.list_feeds

    end

Then you will see that the exception on console is some html crap from WF, that does not honor the supplied accept header.

We need to catch that and return something that makes sense.
Similar if we run in a connection exception (wrong port, host unreachable, ... )

See ManageIQ/manageiq#8239

The metric:

get "http://localhost:8080/hawkular/metrics/gauges/MI~R~%5b70c798a0-6985-4f8a-a525-012d8d28e8a3%2fLocal~~%5d~MT~WildFly%20Aggregated%20Web%20Metrics~Aggregated%20Active%20Web%20Sessions", "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Content-Type"=>"application/json", "Hawkular-tenant"=>"hawkular"
# => 200 OK | application/json 238 bytes
{"id":"MI~R~[70c798a0-6985-4f8a-a525-012d8d28e8a3/Local~~]~MT~WildFly Aggregated Web Metrics~Aggregated Active Web Sessions"

Exception when calling MetricClient.update_tags
bad URI(is not URI?): http://localhost:8080/hawkular/metrics/gauges/MI~R~[70c798a0-6985-4f8a-a525-012d8d28e8a3/Local~~]~MT~WildFly Aggregated Web Metrics~Aggregated Active Web Sessions/tags
( see spaces and [ and / characters )

also potentially applies to other methods.

see #158 (comment) for more info

There is a use-case to give users some freedom and flexibility to choose what their hawkular-metric IDs look like, rather than force them to adopt the rule "inventory metric ID is always the same as hawkular metric ID".

The hawkular wildfly agent is introducing an optional "metric-id" property found on the inventory metric instance definition. When this property exists, the hawkular-metric ID is the value of that property (it is not the value of that inventory metric instance ID).

The current behavior will remain the same and will become the default - inventory metric instance ID is the same as the hawkular-metric ID - they will only be different if someone adds that "metric-id" property in inventory.

See the HWKAGENT JIRA issue and associated PR where the agent creates this metric-id when appropriate: https://issues.jboss.org/browse/HWKAGENT-78

When adding a deployment it is always set to be enabled, even if the enabled field is set to false.

[13:56:08] lponce for the MISSING condition, the rest api talks about a field 'interval' - is that right? If so would that need a new field in the condition of the ruby client?
[13:57:21] pilhuhn, yes, let me remember the API
[13:58:00] pilhuhn, http://www.hawkular.org/docs/rest/rest-alerts.html#MissingCondition yes, an interval field is required

https://github.com/hawkular/hawkular-client-ruby/blob/master/lib/hawkular/inventory/inventory_api.rb#L261

the implementation takes a resource hash as a parameter and then escapes it's id, but it doesn't honor the resource ids of parent resources

Suppose I somehow supply a bad host to the WS-client

client = OperationsClient.new(host: 'acme.org:42',

then the connect will hang for around a minute.

I am not sure though if it is possible to specify a shorter timeout for the connection setup.

the "more info" section contains several links to documentation, they seem to lead nowhere.
https://github.com/hawkular/hawkular-client-ruby#more-info

cc @josejulio

A lot of methods takes the feed_id as argument, this can be obtained from the path.

here is the list:

list_resources_for_type(feed_id, type, fetch_properties = false)
->
list_resources_for_type(type_path, fetch_properties = false, filter = {})


get_config_data_for_resource(feed_id, res_ids)
->
get_config_data_for_resource(resource_path)


def list_child_resources(Resource parent_resource, recursive = false)
->
def list_child_resources(parent_resource_path, recursive = false)


list_relationships(Resource resource, named = nil)
->
list_relationships(path, named = nil)


list_relationships_for_feed(feed_id, named = nil)
->
list_relationships_for_feed(path, named = nil)


list_metrics_for_metric_type(feed_id, type)
->
list_metrics_for_metric_type(met_type_path)


list_metrics_for_resource_type(feed, type)
->
list_metrics_for_resource_type(res_type_path)


list_metrics_for_resource(Resource resource, filter = {})
->
list_metrics_for_resource(resource_path, filter = {})


create_resource(feed_id, type_path, resource_id, resource_name = nil, properties = {})
->
create_resource(type_path, resource_id, resource_name = nil, properties = {})


create_resource_under_resource(feed_id, type_path, parent_resource_ids, resource_id, resource_name = nil,
                                  properties = {})
->
create_resource_under_resource(type_path, parent_resource_path, resource_id, resource_name = nil,
                                  properties = {})

get_resource(feed_id, res_ids, fetch_resource_config = true)
->
get_resource(res_path, fetch_resource_config = true)


create_metric_for_resource(feed_id, metric_id, type_path, res_ids, metric_name = nil)
->
create_metric_for_resource(metric_type_path, resource_path, metric_id, metric_name = nil)


get_tenant
list_child_resources(parent_resource, recursive = false)
get_entity(path)
create_feed(feed_id, feed_name = nil)
delete_feed(feed_id)
create_resource_type(feed_id, type_id, type_name)
create_metric_type(feed_id, metric_type_id, type = 'GAUGE', unit = 'NONE', collection_interval = 60)
events(type = 'resource', action = 'created')
..untouched

I'm not sure whether XA datasources are implemented or not since there is no tests for them.

It appears from the tests that we only support non-XA datasources:

Regardless, please add test cases for XA and Non-XA datasources.

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=1383611#c3

https://travis-ci.org/hawkular/hawkular-client-ruby/jobs/247589703#L791
..and the documentation can't be pushed

We need to update the Changelog file with the changes.
Especially api-changes need to be made explicit

curl -i -u jdoe:password -Hhawkular-tenant:hawkular http://localhost:8080/hawkular/metrics/gauges/tags/

See http://www.hawkular.org/docs/rest/rest-metrics.html#GET__metrics_tags__tags_ (but that needs a type filter). There are type-specific queries as shown above

We need to first find out what the query format is though.

key:* works to find all tags with a certain key. : seems not to work though

See also https://twitter.com/tsegismont/status/781587507980935168 and https://twitter.com/burmanm/status/781736785302974464

Hawkular Metrics is in both OpenShift and H-Services way beyond version 0.8
I expect that there are no more installations of H-Metrics 0.8 out there.
We should remove support for it completely from the gem

CC @simon3z

I'm just reiterating @cfcosta question in irc about adding support for activesupportgem. It sounds like a good idea to me as it will make our code more rails idiomatic.

Thoughts?

Heads up, tests fail with upcoming rest-client 2.1.0.rc1.,

• strange error (the strange char is 0x1F "UNIT SEPARATOR").

  HawkularClient and Metrics client Should both return the version
  Failure/Error: res.empty? ? {} : JSON.parse(res)
  
  JSON::ParserError:
    784: unexpected token at '�'
  # ./lib/hawkular/base_client.rb:39:in `http_get'
  # ./lib/hawkular/metrics/metric_api.rb:38:in `fetch_version_and_status'
  # ./spec/integration/hawkular_client_spec.rb:214:in `block (4 levels) in <module:RSpec>'
  

  (I also saw this with in manageiq-providers-kubernetes tests, some hawkular-client related tests gave this.)

• lots of VCR::Errors::UnhandledHTTPRequestError

Suppose I pass in an incorrect parameter hash for invoke_generic_operation, the callback is not called, but the method bails out with a 'fail':

RuntimeError: Hash property resourcePath must be specified
./lib/hawkular/operations/operations_api.rb:216:in `block in check_pre_conditions'
./lib/hawkular/operations/operations_api.rb:215:in `each'
./lib/hawkular/operations/operations_api.rb:215:in `check_pre_conditions'
./lib/hawkular/operations/operations_api.rb:105:in `invoke_generic_operation'
./spec/integration/operations_spec.rb:78:in `block (2 levels) in <module:RSpec>'

I think the user would only expect this if no callback is defined at all.

H-Metrics also provides metrics of type string (e.g. for log entries).
The client needs to subscribe this.

There is some work already done, but this needs tests
pilhuhn@649081f

Hawkular-inventory has a new api
and has moved with 0.17 the endpoint of the old api to /deprecated

We need to either support the /deprecated endpoint for a while
or directly the new one

There is /status to get the version of inventory. If it is 0.17+ it is the /deprecated one

When I run in MiQ an operation on the Hawkular-server, which runs successfully, the error callback is still called for the response. See this debugger output

This needs more investigation, as running the Status operation on an agent of a connected Wfly does the right thing.

Currently we have inventoryclient, metricsclient, alertclient and so on, where the user needs to instantiate each client separately.

We need to have one hawkular-client that the gets the base path of hawkular and/or the metrics server and which then internally creates those client objects so that the user can do something like

client = Hawkular.client.new(creds:..., base_path:..., options:...) 
inv_client = client.inventory
inv_client.list_feeds
alert_client = client.alerts
alert_client.create_trigger 

and so on.

When we added this dependency, the MiQ didn't have the support for websocket communication, they add it with migration to rails 5 (iirc). Since the websocket-client-simple is no longer maintained and it has other issues like failing non-deterministically on segfaults (https://travis-ci.org/hawkular/hawkular-client-ruby/jobs/213159442#L438). Also there is currently no way to process incoming binary websocket frames (also failing with segfaults), so that we can't implement the JDR reports in the middleware provider.

This goes

def create_resource_under_resource
...
      rescue HawkularException => error
        # 409 We already exist -> that is ok
        raise unless error.status_code == 409
      end
      Resource.new(res)

In case of the 409, we still return a Resource, but constructed from the input, which does not contain the path.

The path needs to be obtained by clients like this:

resource = @inv_client.create_resource  type_path, data['r'], data['r']
  if resource.path.nil?
    resources = @inv_client.list_resources_for_type type_path, true
    resource = resources.first
  end

We should perhaps internalise getting the path into create_resource to have a consistent behaviour in both cases.

We need to update the changelog to contain pointers to all the work that went into the release
@pilhuhn
@Jiri-Kremser
@jshaughn
@lucasponce

::HawkularUtilsMixin -> ::Hawkular::HawkularUtilsMixin