Code Monkey home page Code Monkey logo

azure-log-analytics-data-collector's Introduction

azure-log-analytics-data-collector Ruby Client

Azure Log Analytics Data Collector API Client Libraries for Ruby. The repository was originally created for multiple programming languages, but it was refactored as a dedicated one for Ruby client. Python and PHP client libraries were moved to azure-log-analytics-data-colloector-python and azure-log-analytics-data-colloector-php respectively.

Retry policy

The client internal leverage [rest-client] to send HTTP request to the API. The client library retries request using the rest-client on the following status code (which is recommended action).

  • 429 - Too Many Requests
  • 500 - Internal Server Error
  • 503 - Service Unavailable

By default, the client library retres for a total of 3 times, sleeping 5 sec between retries. The number of retries and sleeping time between retries can be changed with set_retries in the client class.

  def set_retries(max_retries, retry_sleep_period)
    @max_retries = max_retries
    @retry_sleep_period = retry_sleep_period
  end

Installation

gem install azure-loganalytics-datacollector-api

Sample code (Ruby Client)

Sample1 - No time_generated_field option

require "azure/loganalytics/datacollectorapi/client"

customer_id = '<Customer ID aka WorkspaceID String>'
shared_key = '<The primary or the secondary Connected Sources client authentication key>'
log_type = "MyCustomLog"

posting_records = []
record1= {
  :string => "MyText1",
  :boolean => true,
  :number => 100
}
record2= {
  :string => "MyText2",
  :boolean => false,
  :number => 200
}
posting_records.push(record1)
posting_records.push(record2)

client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
res = client.post_data(log_type, posting_records)
puts res
puts "res code=#{res.code}"

if Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
  puts "operation was succeeded!"
else
  puts "operation was failured!"
end

Sample2 - With time_generated_field option

require "azure/loganalytics/datacollectorapi/client"

customer_id = '<Customer ID aka WorkspaceID String>'
shared_key = '<The primary or the secondary Connected Sources client authentication key>'
log_type = "MyCustomLog"

posting_records = []
record1= {
  :string => "MyText1",
  :boolean => true,
  :number => 100,
  :timegen => "2017-11-23T11:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
}
record2= {
  :string => "MyText2",
  :boolean => false,
  :number => 200,
  :timegen => "2017-11-23T12:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
}
posting_records.push(record1)
posting_records.push(record2)

time_generated_field = "timegen"
client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
res = client.post_data(log_type, posting_records, time_generated_field)
puts res
puts "res code=#{res.code}"

if Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
  puts "operation was succeeded!"
else
  puts "operation was failured!"
end

Sample3 - With time_generated_field and azure_resource_id option

Supported setting azure_resource_id option from version 0.2.0

require "azure/loganalytics/datacollectorapi/client"

customer_id = '<Customer ID aka WorkspaceID String>'
shared_key = '<The primary or the secondary Connected Sources client authentication key>'
log_type = "MyCustomLog"

posting_records = []
record1= {
  :string => "MyText1",
  :boolean => true,
  :number => 100,
  :timegen => "2017-11-23T11:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
}
record2= {
  :string => "MyText2",
  :boolean => false,
  :number => 200,
  :timegen => "2017-11-23T12:13:35.576Z" # YYYY-MM-DDThh:mm:ssZ
}
posting_records.push(record1)
posting_records.push(record2)

time_generated_field = "timegen"

# Azure Resource ID
# [Azure Resource ID Format]
# /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/{resourceProviderNamespace}/{resourceType}/{resourceName}
azure_resource_id ="/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/otherResourceGroup/providers/Microsoft.Storage/storageAccounts/examplestorage"

client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
res = client.post_data(log_type, posting_records, time_generated_field, azure_resource_id)
puts res
puts "res code=#{res.code}"

if Azure::Loganalytics::Datacollectorapi::Client.is_success(res)
  puts "operation was succeeded!"
else
  puts "operation was failured!"
end

Sample4 - use proxy to access the API

require "azure/loganalytics/datacollectorapi/client"

...
client=Azure::Loganalytics::Datacollectorapi::Client::new( customer_id, shared_key)
# client.set_proxy() # ENV['http_proxy'] is set by default 
client.set_proxy(your_proxy)
res = client.post_data(log_type, posting_records, time_generated_field)
puts res
...

Change log

Links

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/azure-log-analytics-data-collector. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

azure-log-analytics-data-collector's People

Contributors

catweisun avatar yokawasa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

azure-log-analytics-data-collector's Issues

Fix ruby client: CVE-2020-8130 Moderate severity

moderate severity
Vulnerable versions: <= 12.3.2
Patched version: 12.3.3
here is an OS command injection vulnerability in Ruby Rake before 12.3.3 in Rake::FileList when supplying a filename that begins with the pipe character |.

Retry function

hi, thank you for creating & maintaining useful Plugin!
We are using this product via LogStash plugin.
https://github.com/yokawasa/azure-log-analytics-data-collector

I believe this is not LogStash Plugins issue but this repository issue.
so let me raise retry issue here. Please correct me and this issue if it is wrong.

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api
According to manual, they require client to retry when we following error codes.

Code Status Error code Description
429 Too Many Requests ย  The service is experiencing a high volume of data from your account. Please retry the request later.
500 Internal Server Error UnspecifiedError The service encountered an internal error. Please retry the request.
503 Service Unavailable ServiceUnavailable The service currently is unavailable to receive requests. Please retry your request.

But currently, there is no check and because of rest-client specification, a specific exception class will be thrown when we receive these error code.
https://github.com/yokawasa/azure-log-analytics-data-collector/blob/master/lib/azure/loganalytics/datacollectorapi/client.rb#L43

https://rubydoc.info/gems/rest-client/#Exceptions__see_http___www_w3_org_Protocols_rfc2616_rfc2616-sec10_html_

but as we can imagine easily, retry function may cause not good effect like consuming memory, stacking thread.. etc and that should be configured for each purpose.
so how you think we should implement retry function for this HTTP Data collector?

nil response from RestClient causes 'undefined method `code' for nil:NilClass'

res = RestClient.post( uri, body, headers)
res
rescue => e
c = e.response.code.to_i
if c == 429 || c == 500 || c==503
if retries < @max_retries
retries += 1
sleep(@retry_sleep_period)
retry
end
end
raise e
end
end

res can be nil here, leading to a scenario where the following is logged from the logstash-output-azure_loganalytics plugin when Azure::Loganalytics::Datacollectorapi::Client.is_success(res) is called and is_success attempts to evaluate res.code
from logstash:

[ERROR][logstash.outputs.azureloganalytics][pipeline-common][pipeline_common_output_azure_log_analytics_ingest] Exception occured in posting to DataCollector API as log type ExampleLogType: 'undefined method `code' for nil:NilClass', data=>[{"original_timestamp":"2021-04-15T10:36:21.315Z","event_uuid":"ede25ed2-0d3e-4898-8c5d-3...

There appears to be a related issue logged at rest-client/rest-client#655

Check body size not to exceed data limits of Azure Monitor Data Collection API

Add body size check not to exceed max of 30 MB

Maximum of 30 MB per post to Azure Monitor Data Collector API. This is a size limit for a single post. If the data from a single post that exceeds 30 MB, you should split the data up to smaller sized chunks and send them concurrently.

Relevant issue

Testing for invalid characters in log_type

According to the DataCollector API documentation for the log type can only contain letters, numbers, and the underscore character, and may not exceed 100 characters. However the client.py code is testing for isalpha() which only checks for alphabet characters which excludes underscores. Could you change the test to include '_' and test the length?

Unable to set/use proxy

Hi there,

The collector works great, at least on my computer it does. The problem is that our production environment requires me to use a proxy. I have not yet been able to get this working.

I've tried hacking it into net/http/persistent.rb (by setting proxy=:ENV) but this does not seem to work. I'm not even sure this is used by the collector.

Any information that would point me in the right direction is appreciated!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.