Code Monkey home page Code Monkey logo

azure-log-analytics-data-collector's Issues

nil response from RestClient causes 'undefined method `code' for nil:NilClass'

azure-log-analytics-data-collector/lib/azure/loganalytics/datacollectorapi/client.rb

Lines 58 to 71 in c9a05ca

res = RestClient.post( uri, body, headers)
res
rescue => e
c = e.response.code.to_i
if c == 429 || c == 500 || c==503
if retries < @max_retries
retries += 1
sleep(@retry_sleep_period)
retry
end
end
raise e
end
end

res can be nil here, leading to a scenario where the following is logged from the logstash-output-azure_loganalytics plugin when Azure::Loganalytics::Datacollectorapi::Client.is_success(res) is called and is_success attempts to evaluate res.code
from logstash:

[ERROR][logstash.outputs.azureloganalytics][pipeline-common][pipeline_common_output_azure_log_analytics_ingest] Exception occured in posting to DataCollector API as log type ExampleLogType: 'undefined method `code' for nil:NilClass', data=>[{"original_timestamp":"2021-04-15T10:36:21.315Z","event_uuid":"ede25ed2-0d3e-4898-8c5d-3...

There appears to be a related issue logged at rest-client/rest-client#655

Fix ruby client: CVE-2020-8130 Moderate severity 

moderate severity
Vulnerable versions: <= 12.3.2
Patched version: 12.3.3
here is an OS command injection vulnerability in Ruby Rake before 12.3.3 in Rake::FileList when supplying a filename that begins with the pipe character |.

Testing for invalid characters in log_type

According to the DataCollector API documentation for the log type can only contain letters, numbers, and the underscore character, and may not exceed 100 characters. However the client.py code is testing for isalpha() which only checks for alphabet characters which excludes underscores. Could you change the test to include '_' and test the length?

Check body size not to exceed data limits of Azure Monitor Data Collection API

Add body size check not to exceed max of 30 MB

Maximum of 30 MB per post to Azure Monitor Data Collector API. This is a size limit for a single post. If the data from a single post that exceeds 30 MB, you should split the data up to smaller sized chunks and send them concurrently.

Relevant issue

Unable to set/use proxy

Hi there,

The collector works great, at least on my computer it does. The problem is that our production environment requires me to use a proxy. I have not yet been able to get this working.

I've tried hacking it into net/http/persistent.rb (by setting proxy=:ENV) but this does not seem to work. I'm not even sure this is used by the collector.

Any information that would point me in the right direction is appreciated!

Retry function

hi, thank you for creating & maintaining useful Plugin!
We are using this product via LogStash plugin.
https://github.com/yokawasa/azure-log-analytics-data-collector

I believe this is not LogStash Plugins issue but this repository issue.
so let me raise retry issue here. Please correct me and this issue if it is wrong.

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api
According to manual, they require client to retry when we following error codes.

Code Status Error code Description
429 Too Many Requests   The service is experiencing a high volume of data from your account. Please retry the request later.
500 Internal Server Error UnspecifiedError The service encountered an internal error. Please retry the request.
503 Service Unavailable ServiceUnavailable The service currently is unavailable to receive requests. Please retry your request.

But currently, there is no check and because of rest-client specification, a specific exception class will be thrown when we receive these error code.
https://github.com/yokawasa/azure-log-analytics-data-collector/blob/master/lib/azure/loganalytics/datacollectorapi/client.rb#L43

https://rubydoc.info/gems/rest-client/#Exceptions__see_http___www_w3_org_Protocols_rfc2616_rfc2616-sec10_html_

but as we can imagine easily, retry function may cause not good effect like consuming memory, stacking thread.. etc and that should be configured for each purpose.
so how you think we should implement retry function for this HTTP Data collector?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.