manageiq / manageiq-consumption Goto Github PK

As an administrator, I want to define a minimum amount that will be charged as an step.

For instance, I want to define storage as a minimum of 1MB. Anything below 1MB will be considered as 1MB (from 1B to 999B), 1001B will be considered 2MB, etc.

from /app/models/manageiq/consumption/showback_event.rb:85

  def update_event
    generate_data unless self.data.present?
    @metrics = if  resource.methods.include?(:metrics) then metrics_time_range(end_time,start_time.end_of_month) else [] end
    self.data.each do |key,dimensions|
      dimensions.keys.each do |dim|
        self.data[key][dim] = self.send("#{key}_#{dim}", data[key][dim].to_d)
      end
    end
    if @metrics.count>0
      self.end_time = @metrics.last.timestamp  # <--- How do we know that last metrics is really the last metric of the interval ?
    end
    collect_tags
  end

I think that we should ordering on the query.

We need to validate that the JSON used are valid and not nil

we have today implemented this metrics

   (0.3ms)  SELECT COUNT(*) FROM "chargeable_fields"
  ChargeableField Load (0.2ms)  SELECT "chargeable_fields".* FROM "chargeable_fields"
  ChargeableField Inst Including Associations (0.2ms - 13rows)
[
    [ 0] #<ChargeableField:0x007fbfdde70d98> {
                                       :id => 10000000000001,
        :chargeback_rate_detail_measure_id => 10000000000002,
                                   :metric => "cpu_usagemhz_rate_average",
                                    :group => "cpu",
                                   :source => "used",
                              :description => "Used CPU"
    },
    [ 1] #<ChargeableField:0x007fbfdde70a50> {
                                       :id => 10000000000002,
        :chargeback_rate_detail_measure_id => nil,
                                   :metric => "v_derived_cpu_total_cores_used",
                                    :group => "cpu_cores",
                                   :source => "used",
                              :description => "Used CPU Cores"
    },
    [ 2] #<ChargeableField:0x007fbfdde706e0> {
                                       :id => 10000000000003,
        :chargeback_rate_detail_measure_id => nil,
                                   :metric => "derived_vm_numvcpus",
                                    :group => "cpu",
                                   :source => "allocated",
                              :description => "Allocated CPU Count"
    },
    [ 3] #<ChargeableField:0x007fbfdde704d8> {
                                       :id => 10000000000004,
        :chargeback_rate_detail_measure_id => 10000000000001,
                                   :metric => "derived_memory_used",
                                    :group => "memory",
                                   :source => "used",
                              :description => "Used Memory"
    },
    [ 4] #<ChargeableField:0x007fbfdde702f8> {
                                       :id => 10000000000005,
        :chargeback_rate_detail_measure_id => 10000000000001,
                                   :metric => "derived_memory_available",
                                    :group => "memory",
                                   :source => "allocated",
                              :description => "Allocated Memory"
    },
    [ 5] #<ChargeableField:0x007fbfdde70140> {
                                       :id => 10000000000006,
        :chargeback_rate_detail_measure_id => 10000000000003,
                                   :metric => "net_usage_rate_average",
                                    :group => "net_io",
                                   :source => "used",
                              :description => "Used Network I/O"
    },
    [ 6] #<ChargeableField:0x007fbfe284be18> {
                                       :id => 10000000000007,
        :chargeback_rate_detail_measure_id => 10000000000003,
                                   :metric => "disk_usage_rate_average",
                                    :group => "disk_io",
                                   :source => "used",
                              :description => "Used Disk I/O"
    },
    [ 7] #<ChargeableField:0x007fbfe284b9e0> {
                                       :id => 10000000000008,
        :chargeback_rate_detail_measure_id => nil,
                                   :metric => "fixed_compute_1",
                                    :group => "fixed",
                                   :source => "compute_1",
                              :description => "Fixed Compute Cost 1"
    },
    [ 8] #<ChargeableField:0x007fbfe284b3a0> {
                                       :id => 10000000000009,
        :chargeback_rate_detail_measure_id => nil,
                                   :metric => "fixed_compute_2",
                                    :group => "fixed",
                                   :source => "compute_2",
                              :description => "Fixed Compute Cost 2"
    },
    [ 9] #<ChargeableField:0x007fbfe284aec8> {
                                       :id => 10000000000010,
        :chargeback_rate_detail_measure_id => 10000000000001,
                                   :metric => "derived_vm_allocated_disk_storage",
                                    :group => "storage",
                                   :source => "allocated",
                              :description => "Allocated Disk Storag"
    },
    [10] #<ChargeableField:0x007fbfe284a9a0> {
                                       :id => 10000000000011,
        :chargeback_rate_detail_measure_id => 10000000000001,
                                   :metric => "derived_vm_used_disk_storage",
                                    :group => "storage",
                                   :source => "used",
                              :description => "Used Disk Storag"
    },
    [11] #<ChargeableField:0x007fbfe284a4c8> {
                                       :id => 10000000000012,
        :chargeback_rate_detail_measure_id => nil,
                                   :metric => "fixed_storage_1",
                                    :group => "fixed",
                                   :source => "storage_1",
                              :description => "Fixed Storage Cost 1"
    },
    [12] #<ChargeableField:0x007fbfe284a158> {
                                       :id => 10000000000013,
        :chargeback_rate_detail_measure_id => nil,
                                   :metric => "fixed_storage_2",
                                    :group => "fixed",
                                   :source => "storage_2",
                              :description => "Fixed Storage Cost 2"
    }
]

in this system are implemented only:
cpu_usage_rate_average (CPU_average)

let me know if I am wrong or missing something.

@aljesusg @sergio-ocon

I run
ManageIQ::Consumption::ConsumptionManager.update_events

  VimPerformanceState Inst Including Associations (0.0ms - 0rows)
NoMethodError: undefined method `state_data' for nil:NilClass
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event/cpu.rb:25:in `CPU_max_number_of_cpu'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:81:in `block (2 levels) in update_event'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:80:in `each'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:80:in `block in update_event'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:79:in `each'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:79:in `update_event'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/consumption_manager.rb:17:in `block in update_events'
	from /usr/local/opt/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.4/lib/active_record/relation/delegation.rb:38:in `each'
	from /usr/local/opt/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.4/lib/active_record/relation/delegation.rb:38:in `eac

it is because the resource don't have vim_perfomance_statesrecords (not use it is validate state, but we need to log at least it and recover from it) in methods
manageiq/consumption/showback_event/mem.rb: (and in manageiq/consumption/showback_event/cpu.rb)

  def MEM_total_mem(value)
    if resource.class.name.ends_with?("Container")
      tmem = resource.vim_performance_states.last.state_data[:total_mem]
    else
      tmem = resource.try(:memory_reserve) || 0
    end
    [value, tmem].compact.max.to_i
  end
end

cc @sergio-ocon @aljesusg

I am getting error when Service#type - not sure it is possible without value in type column - I am just using any provided db.

I run
ManageIQ::Consumption::ConsumptionManager.generate_events

ManageIQ::Consumption::ShowbackUsageType Inst Including Associations (5.4ms - 4rows)
NoMethodError: undefined method `ends_with?' for nil:NilClass
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:45:in `block in generate_data'
	from /usr/local/opt/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.4/lib/active_record/relation/delegation.rb:38:in `each'
	from /usr/local/opt/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.4/lib/active_record/relation/delegation.rb:38:in `each'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:44:in `generate_data'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/consumption_manager.rb:54:in `generate_event_resource'
	from /Users/liborp

cc @sergio-ocon @aljesusg

Time.zone.beginnign is not runnit we need to changeit to Time.zone.now.begin....

Association with pool and add new tags but not remove.

I point out
https://github.com/miq-consumption/manageiq-consumption/blob/master/app/models/manageiq/consumption/showback_event.rb#L78
https://github.com/miq-consumption/manageiq-consumption/blob/master/app/models/manageiq/consumption/showback_event.rb#L107

Why here is not used MetricRollup?

In MetricRollups are roll-up values(so we don't need to calculate it again) and I think that Metrics can be purged.(not sure if we can rely on it)

cc @sergio-ocon

there is ERD diagram:

I know we discussed it but can you please provide some short description for each model - about what they are exactly for?

thanks!

@miq-bot add_label documentation

cc @sergio-ocon @aljesusg

Events are only generated once, so we need to get a means to differentiate charges from one month to another.

Currently, we are just storing the event data (as initially we were going to create an event per pool).

With the new requirements, we wanted to add a snapshot of the start of the cycle in charge, so we could calculate up to date information for the current cycle (i.e # of bytes downloaded is total - start of cycle).

However, that doesn't take into account that once the pool is closed the charge will not be based on current event, but on the status of the event at that moment, so we should also store the snapshot at the end of the event in the charge.

• Create a pool when change to PROCESING
• Filter pool by OPEN state
• Before Action in Pool

When using ManageIQ::Consumption::ShowbackPricePlan.seed in tests, you get random errors where you get the count as 0 instead

• update
• add
• get
• nullify
• sum
• calculate
• find

rake aborted!
Don't know how to build task 'default' (see --tasks)
/home/travis/.rvm/gems/ruby-2.3.1/gems/rake-11.3.0/exe/rake:27:in <top (required)>' /home/travis/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in eval'
/home/travis/.rvm/gems/ruby-2.3.1/bin/ruby_executable_hooks:15:in `

Define tiers in rate, as defined in the wiki:

Showback Rate Wiki docs

Money can be calculated using infinite precession
And you can use Rational for divisions

That should increase accuracy (altough values are stored with 2 decimals)

We need to add manageiq-consumption to miq_bot. How can we do it @Fryguy ?

I have
Host.count => 12 VmOrTemplate.count => 1233 Container.count => 924 Metric.count => 93738 ...

I did the demo step and I ended with
ManageIQ::Consumption::ConsumptionManager.update_events

:rows_by_class:
  ManageIQ::Consumption::ShowbackEvent: 1290
  VmOrTemplate: 366
  Metric: 0
  Hardware: 366
  Tag: 414
  Container: 924
  VimPerformanceState: 3144
:total_queries: 11893
:total_rows: 6504

I think that 11893 of queries is too much.
Can we reduce it ?
I can provide the db with all data.

@sergio-ocon
@aljesusg

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

for example CPU

module ManageIQ::Consumption::ShowbackEvent::CPU
  #
  #  Return the average acumulated with the new one
  #
  def CPU_average(value)
    if @metrics.count>0
      ((value * event_days + @metrics.average(:cpu_usage_rate_average)) / (event_days + 1))
    else
      value
    end
  end

  #
  # Return Number Ocurrences
  #
  def CPU_number(value)
    return value
  end

  #
  #  Return the max number of cpu for object
  #
  def CPU_max_number_of_cpu(value)
    if resource.class.name.ends_with?("Container")
      numcpus = resource.vim_performance_states.last.state_data[:numvcpus]
    else
      numcpus = if resource.methods.include?(:cpu_total_cores) then resource.cpu_total_cores else 0 end
    end
    [value, numcpus].max.to_i
  end
end

I have couple questions for this.

I understand that these methods calculating metrics and these are stored to the ShowbackEvent and values stored in the ShowbackEvent are used for rating.

1. In which units have to be these values related to time base?
For example (to me it looks like metric value per day)

  def CPU_average(value)
    if @metrics.count>0
      ((value * event_days + @metrics.average(:cpu_usage_rate_average)) / (event_days + 1))
    else
      value
    end
  end

1a) should not be average calculated by time as we have similar issue here ManageIQ/manageiq#12582 ?
1b) Should not it be in hours ?
1c) How I know if these methods represents used or allocated ?

1d)
Let's assume that Vm comes to the system on Aug 1 and today is Aug 2 and I want to do chargeback on Aug 2

• on Aug 1 is called update_event on the event as it comes to the system. so it has calculated value as zero.
  what start_time and end_time will be set ?

• 1e) where is called update_event again to update values ? after which process ?

• 1f) let's assume that method is called on 2 Aug on end of day (where ? 1e) and lets assume that we have metrics to the 2 Aug.

@metrics are from interval end_time to start_time.end_of_month so in this case it will take from Aug 2 to end of Aug.
so I think it will not take into account any metrics ?

1g) why when I want to do average only from 1 to 2 Aug(on event) is calculated average of whole month ?

1h) or maybe you can provide example(only what happess with event not other things) what will explain what should exaclty happen,what time are setup, for example when Vm will comes on 1 Aug at 10:00 and I want to do chargeback on 2 Aug on 17:00.
I mean
Vm comes to system after refresh and event is created (when, where ?) and event is updated(when,where ?, by wich process what times will we set up ?) and then what process will updated event on 2.aug (..)?

2. How this works ?

  def CPU_number(value)
    return value
  end

According to the code the value will be still 0 from generating of events . as I am understanding well.

@sergio-ocon @aljesusg

Right now, calculate_cost is called with an event, the event data is used for rating.

We need a function that caluclates charges based on data, not on the event, so it can be called with fake events

after
irb(main):018:0> ManageIQ::Consumption::ConsumptionManager.generate_events
I am getting error:

  Tag Inst Including Associations (0.1ms - 1rows)
  Classification Load (0.2ms)  SELECT  "classifications".* FROM "classifications" WHERE "classifications"."tag_id" = $1 LIMIT $2  [["tag_id", 10000000000187], ["LIMIT", 1]]
  Classification Inst Including Associations (0.0ms - 0rows)
NoMethodError: undefined method `category' for nil:NilClass
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:97:in `block in collect_tags'
	from /usr/local/opt/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.4/lib/active_record/relation/delegation.rb:38:in `each'
	from /usr/local/opt/rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activerecord-5.0.4/lib/active_record/relation/delegation.rb:38:in `each'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/showback_event.rb:96:in `collect_tags'
	from /Users/liborpichler/manageiq/manageiq-consumption/app/models/manageiq/consumption/consumption_manager.rb:55:in `generate_event_resourc
...

because my resource

[20] pry(#<ManageIQ::Consumption::ShowbackEvent>)> resource.tags
=> [#<Tag:0x007faef2cc8c08 id: 10000000000024, name: "/managed/environment/prod">,
 #<Tag:0x007faef2cc8a78 id: 10000000000063, name: "/managed/network_location/dmz">,
 #<Tag:0x007faef2cc8910 id: 10000000000168, name: "/managed/function/app">,
 #<Tag:0x007faef2cc87d0
  id: 10000000000187,
  name: "/miq_policy/assignment/miq_policy_set/10000000000024">]

has also different type of tags which can be without category
I believe we need only managed tags here (maybe there is method for it):

--- a/app/models/manageiq/consumption/showback_event.rb
+++ b/app/models/manageiq/consumption/showback_event.rb
@@ -75,6 +75,7 @@ class ManageIQ::Consumption::ShowbackEvent < ApplicationRecord
   def update_event
     generate_data unless self.data.present?
     @metrics = if  resource.methods.include?(:metrics) then metrics_time_range(end_time,start_time.end_of_month) else [] end
     self.data.each do |key,dimensions|
       dimensions.keys.each do |dim|
         self.data[key][dim] = self.send("#{key}_#{dim}", data[key][dim].to_d)
@@ -93,6 +94,7 @@ class ManageIQ::Consumption::ShowbackEvent < ApplicationRecord
       self.context["tag"] = {} unless self.context.has_key?("tag")
     end
     resource.tags.each do |tag|
+      next unless tag.name.ends_with?('managed')
       category = tag.classification.category
       self.context["tag"][category] = [] unless self.context["tag"].has_key?(category)
       self.context["tag"][category] << tag.classification.name unless self.context["tag"]
-
 ```ruby

@miq-bot assign @sergio-ocon
cc @aljesusg

When calling some functions in tests, reloading the events change the values.

Specifically:

ev = FactoryGirl.create(:showback_event)
ev.resource.type == 'Vm' # Out of the box
ev.reload.resource == nil

The workaround is to define the resource beforehand

resource = FactoryGirl.create(:vm)
ev = FactoryGirl.create(:showback_event, resource: resource)
ev.resource.type == 'Vm'
ev.reload.resource.type == 'Vm'

There should not be a need to do this

The event stores all the information about the event from the beginning of time until the end of the event. However, we don't use that information, only the data for one specific period.

We need to call get_measure in the charge, not in the event, and thus get the information from the

Generate system events:

• System events will cope with those charges that are not generated as events

List of things to cope with:

• Fixed costs per bill. Costs that are assigned to the bill
• CRUD on resources. Set up feeds for vm or containers
• Charges for other things needed (i.e. smart analysis execution)

When defining a rate, is possible to want to define a minimum step in a rate.

Example:

The minimum step is 1 hour. An event lasts:

• 30 seconds --> 1 hour
• 1 h --> 1 h
• 1 h 20 m --> 2 h

Etc.

8cac846

We need to validate in showbacktier in the method validate_interval that tiers has continuity.

• A rate can't have a tier from 0-5 and other one from 7-Infinity, there is values from 5-7 that are not defined.

Right now, rates are defined as numbers that get multiplied, but we need to take into account that rates should be defined as:

• fixed $ + input (measure) * variable_rate * $ / measure

When you create a new envelope or a new month, the charges need to be updated in the old pool and the inventory remains the same:

• Get last data update, store it as a final snapshot in the old pool. Move old pool to processing
• Copy charges into the new pool. Initial snapshot should be the same data that the old pool charge final snapshot data. Charges copied should be the one where the event is still happening, not copying those events that are closed (VM has been deleted, etc)

Once a pool is in processing, you can't update the data, only the charge

Money only stores information with the number of cents that the currency has.
There must be a way to make calculations and store the currency using the new currency "MIQ" to allow 8-12 digits in the calculations.

Hello,

I am playing with your repo and I think that it will be good to see the example on some more real cases.

Let's assume this situation:

• Hourly rate $1 per 1CPU
• let's take the month of 30 days
• 1 CPU was allocated in first 10 days in a month
• 2 CPUs were allocated in last 20 days

So calculation should looks like:
VM as 1 CPU for 10 days and 2 CPU for 20 days then the cost should be as follow: 10 days x 24 Hours x 1 CPU x 1$/CPU/Hour + 20 days x 24 Hours x 2 CPU x 1 CPU x 1$/CPU/Hour = 240$ + 960$ = 1200 $

Can you demonstrate how it can be calculated with miq-consumption?
How to set ShowbackRate,... and what everything is needed from miq db (Metric, MetricRollups,...)

cc @gtanzillo
@miq-bot assign @sergio-ocon @aljesusg

thanks!

We need to use this methods for other classes

Currently, we use Time.utc and Time.current in the code.

The code should be reviewed to align it with the rest of ManageIQ and make sure that things like beginning_of_month is always referring to the same moment.

I followed a link to this repo from ManageIQ/manageiq and did not know what this is. The descriptions in README.md and https://github.com/miq-consumption/manageiq-consumption/blob/master/manageiq-consumption.gemspec#L14 weren't helpful.
After browsing the code and talking to others it seems to be related to ManageIQ's Chargeback feature, is that correct?
If so, can someone please update the description to reflect that?
Is it a replacement for chargeback or enhancements?