elastic / logstash-devutils Goto Github PK

View Code? Open in Web Editor NEW

17.0 342.0 29.0 105 KB

An assortment of tooling/libraries to make Logstash core and plugin development and releasing a bit easier.

License: Apache License 2.0

Ruby 100.00%

logstash-devutils's People

Stargazers

Watchers

logstash-devutils's Issues

Heads up: latest logstash-core will break devutils for some plugins

Hi guys,

Just a heads up; I was testing out devutils using the latest logstash-core bundled with logstash (6.x) rather than the old version (logstash-core-5.6.4) that gets pulled with logstash-core-plugin-api-2.1.28-java. There is a regression that will appear in devutils when the logstash-core-plugin-api gets updated that causes plugins to be registered twice. This causes some plugins to break as this is unexpected behaviour.

4 years ago (https://github.com/elastic/logstash-devutils/blame/master/lib/logstash/devutils/rspec/logstash_helpers.rb#L50)
A commit was made to devutils to register the filter plugins.

1 year ago (https://github.com/elastic/logstash/blame/master/logstash-core/lib/logstash/pipeline.rb#L487)
A commit was made to Logstash (released under 6.x) which registers the filter plugins. The result is that the registration now occurs twice. More complex plugins such as logstash-filter-augment break when registered twice (how/why is irrelevant).

I know that logstash-core-plugin-api is only updated every so often (I'm guessing only when there are relevant changes to the api) but the recent change proves that the api has in fact changed since it causes this issue.

Move `be_a_logstash_timestamp_equivalent_to` into this gem to be reused

With the PR on the integration-jdbc and on filter-date the same utility matcher was copied and pasted:

  RSpec::Matchers.define :be_a_logstash_timestamp_equivalent_to do |expected|
    expected = LogStash::Timestamp.new(expected) unless expected.kind_of?(LogStash::Timestamp)
    description { "be a LogStash::Timestamp equivalent to #{expected}" }

    match do |actual|
      actual.kind_of?(LogStash::Timestamp) && actual == expected
    end
  end

This matcher is used in timestamp expectations to avoid string equality and be portable across different versions of Logstash.
With Logstash PR elastic/logstash#12797 was introduced nanosecond precision and this made fails some plugins test that was based on string compare on timestamps.

Tasks:

move the matcher in logstash-devutils
publish the new version of the gem
update JDBC integration to use the new logstash-devutils
update date filter to use the new logstash-devutils

How to test nested fields

Hello.

I'm trying to use logstash-devutils to test my logstash configurations. I manage to successfully do it for simple scenarios, so that's good 👍

Now one of my logstash config files has an if like there:

if ( [@type] == "LogMessage" and [@source][type] == "ssh" ) {
        mutate {
            add_field => { "x" => "a" }
        }
} else {
        mutate {
            add_field => { "x" => "b" }
        }
}

I'm trying to write an unit test for it and I wrote

  sample("@message" => message, "@type" => "LogMessage", "[@source][type]" => "ssh") do
    # Check the ouput event/message properties
    insist { subject.get("x") } == "a"

This fails because value of x is b.

I tried changing the sample from "[@source][type]" => "ssh" to "@source.type" => "ssh" but it's the same result.

I had a look at the code and I guess that [@source][type] will be interpreted as an array here

  def sample(sample_event, &block)
    name = sample_event.is_a?(String) ? sample_event : LogStash::Json.dump(sample_event)
    name = name[0..50] + "..." if name.length > 50

    describe "\"#{name}\"" do
      let(:pipeline) { new_pipeline_from_string(config) }
      let(:event) do
        sample_event = [sample_event] unless sample_event.is_a?(Array)
        next sample_event.collect do |e|
          e = { "message" => e } if e.is_a?(String)
          next LogStash::Event.new(e)
        end
      end

Therefore it will be ignored.

My knowledge of ruby is pretty much 0, so I can't help much here. My question is: am I doing something wrong? Is there a way to make that if to be true, other than changing the config file itself?

I don't know if this project is still actively monitored, but thanks in advance anyway! 🤞

Should devutils become a placeholder gem ?

Hi,
I start thinking this gem should become a placeholder gem that we can require to load all devutils, but also have specific gems that get more to the specific business of each utility we aim to create. As a concrete example I created https://github.com/elastic/logstash-test-helpers with the intention to provide a complete lib of helpers and matchers for logstash, for example also fixing #31as in here.

I see this gem being required and loaded throw here most probably, then the other important part left here are the rake tasks, are there more common parts we can have here? should be change the name ?

What do you think?

Remove Event monkey patch

Remove the broken => true filter to void broken test

Except from to exclude known broken test by using a broken => true filter we should go there and fix them.

Critical vulnerability CVE-2020-14001 in Kramdown runtime dependency

logstash-devutils has a runtime dependency to the outdated Kramdown version 1.14.0. This causes Trivy to report the critical vulnerability CVE-2020-14001 for logstash as a whole:

$ trivy --exit-code 1 --severity CRITICAL docker.elastic.co/logstash/logstash:7.9.3
2021-01-06T17:10:53.942Z        INFO    Need to update DB
2021-01-06T17:10:53.942Z        INFO    Downloading DB...
19.57 MiB / 19.57 MiB [----------------------------------------------------------------------------------------------------------------------------] 100.00% 6.69 MiB p/s 3s
2021-01-06T17:11:14.850Z        WARN    No OS package is detected. Make sure you haven't deleted any files that contain information about the installed packages.
2021-01-06T17:11:14.850Z        WARN    e.g. files under "/lib/apk/db/", "/var/lib/dpkg/" and "/var/lib/rpm"
2021-01-06T17:11:14.850Z        INFO    Detecting RHEL/CentOS vulnerabilities...
2021-01-06T17:11:14.850Z        INFO    Detecting ruby vulnerabilities...

docker.elastic.co/logstash/logstash:7.9.3 (centos 7.8.2003)
===========================================================
Total: 0 (CRITICAL: 0)


usr/share/logstash/Gemfile.lock
===============================
Total: 1 (CRITICAL: 1)

+----------+------------------+----------+-------------------+---------------+--------------------------------+
| LIBRARY  | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |             TITLE              |
+----------+------------------+----------+-------------------+---------------+--------------------------------+
| kramdown | CVE-2020-14001   | CRITICAL | 1.14.0            | 2.3.0         | rubygem-kramdown: processing   |
|          |                  |          |                   |               | template options inside        |
|          |                  |          |                   |               | documents allows unintended    |
|          |                  |          |                   |               | read access or embedded...     |
+----------+------------------+----------+-------------------+---------------+--------------------------------+

Trivy deduces this by reading logstash's Gemfile.lock file, which contains the following:

    logstash-devutils (1.3.6-java)
      fivemat
      gem_publisher
      insist (= 1.0.0)
      kramdown (= 1.14.0)
      logstash-core-plugin-api (>= 2.0, <= 2.99)
      minitar
      rake
      rspec (~> 3.0)
      rspec-wait
      stud (>= 0.0.20)

An update of kramdown to version 2.3.0 would fix the issue.

Can't extract the ASN database from its tar.gz file

The gem can't extract the ASN database defined in this vendor.json (I hope I got the semantic of files right)

[
    {
        "url": "http://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz",
        "files": "GeoLite2-ASN.mmdb",
        "sha1": "ce70f043bf71602a211cef3565560e17d3543609"
     }
 ]

It's because of these lines in lib/logstash/devutils/rake/vendor.rake.

    161    if download =~ /.tar.gz/
    162      prefix = download.gsub('.tar.gz', '').gsub('vendor/', '')
    163      # prefix GeoLite2-ASN
    164      untar(download) do |entry|
    165        if !file['files'].nil?
    166          next unless file['files'].include?(entry.full_name.gsub(prefix, ''))
    167          out = entry.full_name.split("/").last
    168        end
    169        File.join('vendor', out)
    170      end
        ...

Two problems here:

the gsub at 166 turns GeoLite2-ASN_20170620/GeoLite2-ASN.mmdb into _20170620/.mmdb which won't match anything. A sub would do but it's not a general solution.
the order or the include? is wrong and should be reversed (the entry includes the files)

Fix: replace lines 166 and 167 with

              candidate = entry.full_name.split("/").last
              next unless candidate.include?(file['files'])
              out = candidate

2.1 Has Invalid Dependencies

Using jruby -S bundle install with logstash-devutils at 2.1

Bundler could not find compatible versions for gem "logstash-core":
  In Gemfile:
    logstash-devutils (~> 2.1) was resolved to 2.1.0, which depends on
      logstash-codec-plain was resolved to 1.0.0, which depends on
        logstash-core (>= 1.4.0, < 2.0.0)

    logstash-devutils (~> 2.1) was resolved to 2.1.0, which depends on
      logstash-core (>= 6.3)```

Downgrading to `2.0` installs correctly.

Broken gem install logstash-devutils

I'm developing my first Logstash filter and I can't install logstash-devutils because logstash-core-java-event 5.0.0.pre.beta1 depends on fileutils and RMagick which requires C extensions and jruby/lib/native/include/ruby/ruby.h

There is my Dockerfile:

FROM logstash:latest

RUN DEBIAN_FRONTEND=noninteractive \
 && apt-get update \
 && apt-get --yes --no-install-suggests --no-install-recommends install git less vim

ENV JRUBY_HOME=/opt/logstash/vendor/jruby

ENV PATH=$JRUBY_HOME/bin:$PATH

RUN gem install bundler

There is console log:

# bundle install
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies.........
Installing rake 11.3.0
Installing numerizer 0.1.1
Installing clamp 0.6.5
Installing coderay 1.1.1
Installing concurrent-ruby 1.0.0
Installing diff-lcs 1.2.5
Installing ffi 1.9.14
Installing filesize 0.0.4
Installing five mat 1.3.2
Installing gem_publisher 1.5.0
Installing gems 0.8.3
Installing i18n 0.6.9
Installing insist 1.0.0
Installing jar-dependencies 0.3.5
Installing jrjackson 0.4.0
Installing jrmonitor 0.4.2
Installing jruby-openssl 0.9.16
Installing kramdown 1.12.0
Installing ruby-maven-libs 3.3.9
Installing minitar 0.5.4
Installing method_source 0.8.2
Installing slop 3.6.0
Installing puma 2.16.0
Installing rubyzip 1.1.7
Installing rack 1.6.4
Installing tilt 2.0.5
Installing stud 0.0.22
Installing thread_safe 0.3.5
Installing polyglot 0.3.5
Installing rspec-support 3.5.0
Using bundler 1.13.1
Installing chronic_duration 0.10.6
Installing spoon 0.0.6
Installing ruby-maven 3.3.12
Installing rack-protection 1.5.3
Installing treetop 1.4.15
Installing rspec-core 3.5.3
Installing spec-expectations 3.5.0
Installing rspec-mocks 3.5.0
Installing pry 0.10.4
Installing logstash-core-event-java 5.0.0.pre.beta1
  jar dependencies for logstash-core-event-java-5.0.0.pre.beta1-java.gemspec . . .
LoadError: no such file to load -- fileutils
  require at org/jruby/RubyKernel.java:944
    <top> at uri:classloader:/maven/tools/dsl.rb:1
  require at org/jruby/RubyKernel.java:944
    <top> at classpath:/parser.rb:1
[ERROR] Internal error: org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- fileutils -> [Help 1]
org.apache.maven.InternalErrorException: Internal error: org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- fileutils
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:121)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- fileutils
    at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:131)
    at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1307)
    at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1352)
    at org.sonatype.maven.polyglot.ruby.RubyParser.<init>(RubyParser.java:42)
    at org.sonatype.maven.polyglot.ruby.RubyModelReader.read(RubyModelReader.java:56)
    at org.sonatype.maven.polyglot.TeslaModelProcessor.read(TeslaModelProcessor.java:143)
    at org.sonatype.maven.polyglot.TeslaModelProcessor.read(TeslaModelProcessor.java:95)
    at org.apache.maven.model.building.DefaultModelBuilder.readModel(DefaultModelBuilder.java:533)
    at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:273)
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:469)
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:438)
    at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:401)
    at org.apache.maven.graph.DefaultGraphBuilder.collectProjects(DefaultGraphBuilder.java:419)
    at org.apache.maven.graph.DefaultGraphBuilder.getProjectsForMavenReactor(DefaultGraphBuilder.java:410)
    at org.apache.maven.graph.DefaultGraphBuilder.build(DefaultGraphBuilder.java:83)
    at org.apache.maven.DefaultMaven.buildGraph(DefaultMaven.java:491)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:219)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    ... 11 more
Caused by: org.jruby.exceptions.RaiseException: (LoadError) no such file to load -- fileutils
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:944)
    at RUBY.<top>(uri:classloader:/maven/tools/dsl.rb:1)
    at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:944)
    at RUBY.<top>(classpath:/parser.rb:1)
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/InternalErrorException
Installing sinatra 1.4.7
Installing rspec 3.5.0
Errno::ENOENT: No such file or directory -
/opt/logstash/vendor/jruby/lib/ruby/gems/shared/gems/logstash-core-event-java-5.0.0.pre.beta1-java/deps.lst
An error occurred while installing logstash-core-event-java
(5.0.0.pre.beta1), and Bundler cannot continue.
Make sure that `gem install logstash-core-event-java -v '5.0.0.pre.beta1'`
succeeds before bundling.
#
#
# gem install fileutils
Fetching: rmagick-2.16.0.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fileutils:
    ERROR: Failed to build gem native extension.

    /opt/logstash/vendor/jruby/bin/jruby -r ./siteconf20160928-258-2o3fk5.rb extconf.rb
NotImplementedError: C extension support is not enabled. Pass -Xcext.enabled=true to JRuby or set JRUBY_OPTS.

   (root) at /opt/logstash/vendor/jruby/lib/ruby/shared/mkmf.rb:8
  require at org/jruby/RubyKernel.java:1040
   (root) at /opt/logstash/vendor/jruby/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1
   (root) at extconf.rb:4

extconf failed, uncaught signal 1

Gem files will remain installed in /opt/logstash/vendor/jruby/lib/ruby/gems/shared/gems/rmagick-2.16.0 for inspection.
Results logged to /opt/logstash/vendor/jruby/lib/ruby/gems/shared/extensions/universal-java-1.8/1.9/rmagick-2.16.0/gem_make.out
#
#
# JRUBY_OPTS="-Xcext.enabled=true" gem install fileutils
Building native extensions.  This could take a while...
ERROR:  Error installing fileutils:
    ERROR: Failed to build gem native extension.

    /opt/logstash/vendor/jruby/bin/jruby -r ./siteconf20160928-340-1dw56tf.rb extconf.rb
/opt/logstash/vendor/jruby/lib/ruby/shared/mkmf.rb:14: Use RbConfig instead of obsolete and deprecated Config.
mkmf.rb can't find header files for ruby at /opt/logstash/vendor/jruby/lib/native/include/ruby/ruby.h

extconf failed, uncaught signal 1

Gem files will remain installed in /opt/logstash/vendor/jruby/lib/ruby/gems/shared/gems/rmagick-2.16.0 for inspection.
Results logged to /opt/logstash/vendor/jruby/lib/ruby/gems/shared/extensions/universal-java-1.8/1.9/rmagick-2.16.0/gem_make.out

rspec tests of clone and drop filters are broken on logstash-devutils 2.0.4

Hello!
I'm using logstash-devutils to test my pipelines with rspec.
I've just tried to update to version 2.0.4 (my current version 1.3.6 works too slow) and it stoped to correctly work with filters that delete or create new events (clone and drop).
Here is how I reproduce it:

git clone [email protected]:elastic/logstash.git
cd logstash
sed -i 's/"logstash-devutils", "~> 1"/"logstash-devutils", "~> 2"/g' Gemfile.template
rake bootstrap
rake plugin:install-default
bin/logstash-plugin install --development
bin/rspec logstash-core/spec/logstash/custom_pipeline_spec.rb

where custom_pipeline_spec.rb is cutted copy of pipeline_spec.rb:

require "spec_helper"
require "logstash/inputs/generator"
require "logstash/filters/drop"
require_relative "../support/mocks_classes"
require_relative "../support/helpers"
require "stud/try"
require 'timeout'
require 'logstash/config/pipeline_config'

describe LogStash::Pipeline do
  let(:worker_thread_count)     { 5 }
  let(:safe_thread_count)       { 1 }
  let(:override_thread_count)   { 42 }
  let(:dead_letter_queue_enabled) { false }
  let(:dead_letter_queue_path) { }
  let(:pipeline_settings_obj) { LogStash::SETTINGS.clone }
  let(:pipeline_settings) { {} }
  let(:max_retry) {10} #times
  let(:timeout) {120} #seconds

  before :each do
    pipeline_workers_setting = LogStash::SETTINGS.get_setting("pipeline.workers")
    allow(pipeline_workers_setting).to receive(:default).and_return(worker_thread_count)
    dlq_enabled_setting = LogStash::SETTINGS.get_setting("dead_letter_queue.enable")
    allow(dlq_enabled_setting).to receive(:value).and_return(dead_letter_queue_enabled)
    dlq_path_setting = LogStash::SETTINGS.get_setting("path.dead_letter_queue")
    allow(dlq_path_setting).to receive(:value).and_return(dead_letter_queue_path)

    pipeline_settings.each {|k, v| pipeline_settings_obj.set(k, v) }
  end

  context "compiled flush function" do

    context "cancelled events should not propagate down the filters" do
      config <<-CONFIG
        filter {
          drop {}
        }
      CONFIG

      sample("hello") do
        expect(subject).to eq(nil)
      end
    end

    context "new events should propagate down the filters" do
      config <<-CONFIG
        filter {
          clone {
            clones => ["clone1"]
          }
        }
      CONFIG
      sample(["foo", "bar"]) do
        expect(subject.size).to eq(4)
      end
    end
  end

end

And result is:

Sending Logstash logs to null which is now configured via log4j2.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/IIIEII/Documents/Workspace/elk/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/IIIEII/Documents/Workspace/elk/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.2.0-java/lib/ext/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Run options: exclude {:integration=>true, :redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :windows=>true}

Randomized with seed 293
FF

Failures:

  1) LogStash::Pipeline compiled flush function new events should propagate down the filters "["foo","bar"]" when processed
     Failure/Error: expect(subject.size).to eq(4)
     
       expected: 4
            got: 2
     
       (compared using ==)
     # ./logstash-core/spec/logstash/pipeline_spec_2.rb:91:in `block in <main>'
     # ./spec/spec_helper.rb:66:in `block in <main>'
     # ./logstash-core/lib/logstash/util.rb:43:in `set_thread_name'
     # ./spec/spec_helper.rb:65:in `block in <main>'
     # ./spec/spec_helper.rb:58:in `block in <main>'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-2.0.4-java/lib/logstash/devutils/rspec/spec_helper.rb:51:in `block in <main>'
     # ./lib/bootstrap/rspec.rb:31:in `<main>'

  2) LogStash::Pipeline compiled flush function cancelled events should not propagate down the filters "hello" when processed
     Failure/Error: expect(subject).to eq(nil)
     
       expected: nil
            got: #<LogStash::Event:0x38eb2140>
     
       (compared using ==)
     # ./logstash-core/spec/logstash/pipeline_spec_2.rb:78:in `block in <main>'
     # ./spec/spec_helper.rb:66:in `block in <main>'
     # ./logstash-core/lib/logstash/util.rb:43:in `set_thread_name'
     # ./spec/spec_helper.rb:65:in `block in <main>'
     # ./spec/spec_helper.rb:58:in `block in <main>'
     # ./vendor/bundle/jruby/2.5.0/gems/logstash-devutils-2.0.4-java/lib/logstash/devutils/rspec/spec_helper.rb:51:in `block in <main>'
     # ./lib/bootstrap/rspec.rb:31:in `<main>'

Finished in 2.02 seconds (files took 2.82 seconds to load)
2 examples, 2 failures

Failed examples:

rspec ./logstash-core/spec/logstash/pipeline_spec_2.rb:90 # LogStash::Pipeline compiled flush function new events should propagate down the filters "["foo","bar"]" when processed
rspec ./logstash-core/spec/logstash/pipeline_spec_2.rb:77 # LogStash::Pipeline compiled flush function cancelled events should not propagate down the filters "hello" when processed

Randomized with seed 293

As you can see, drop and clone filters just do nothing. You have same number of event on output as on input.

Issue with "bundle exec rake publish_gem".

Hello. I was following the instructions on how to write a Logstash output plugin, and received the following error when executing "bundle exec rake publish_gem".

I'm building on Windows, and actually successfully built a couple plugins a few weeks ago. Could you please help me out? Thank you.

C:\Projects\GitHub\elastic\logstash-output-https>bundle exec rake publish_gem --trace
io/console not supported; tty will not be manipulated
io/console not supported; tty will not be manipulated
** Invoke publish_gem (first_time)
** Execute publish_gem
rake aborted!
IOError: Cannot run program "gem" (in directory "C:\Projects\GitHub\elastic\logstash-output-https"): CreateProcess error=2, The system cannot find the file specified
org/jruby/RubyIO.java:4386:in `popen3'
C:/jruby/lib/ruby/1.9/open3.rb:74:in `popen3'
C:/jruby/lib/ruby/gems/shared/gems/logstash-devutils-0.0.13-java/lib/logstash/devutils/rake/publish.rake:11:in `execute'
C:/jruby/lib/ruby/gems/shared/gems/gem_publisher-1.5.0/lib/gem_publisher/builder.rb:10:in `build'
C:/jruby/lib/ruby/gems/shared/gems/gem_publisher-1.5.0/lib/gem_publisher/publisher.rb:30:in `publish_if_updated'
C:/jruby/lib/ruby/gems/shared/gems/gem_publisher-1.5.0/lib/gem_publisher.rb:26:in `publish_if_updated'
C:/jruby/lib/ruby/gems/shared/gems/logstash-devutils-0.0.13-java/lib/logstash/devutils/rake/publish.rake:27:in `(root)'
org/jruby/RubyProc.java:271:in `call'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/task.rb:238:in `execute'
org/jruby/RubyArray.java:1613:in `each'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/task.rb:179:in `invoke_with_call_chain'
C:/jruby/lib/ruby/1.9/monitor.rb:211:in `mon_synchronize'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:106:in `top_level'
org/jruby/RubyArray.java:1613:in `each'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:106:in `top_level'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:78:in `run'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
C:/jruby/lib/ruby/gems/shared/gems/rake-10.4.2/bin/rake:33:in `(root)'
org/jruby/RubyKernel.java:1087:in `load'
C:\jruby\bin\rake:23:in `(root)'
Tasks: TOP => publish_gem

Broken Builds after 0.0.20

As title says:

Using Accessor#strict_set for specs
NameError: undefined method 'set' for class 'LogStash::Event'
     alias_method at org/jruby/RubyModule.java:2333
            Event at /home/ubuntu/.rbenv/versions/jruby-1.7.22/lib/ruby/gems/shared/gems/logstash-devutils-0.0.20-java/lib/logstash/devutils/rspec/spec_helper.rb:38
           (root) at /home/ubuntu/.rbenv/versions/jruby-1.7.22/lib/ruby/gems/shared/gems/logstash-devutils-0.0.20-java/lib/logstash/devutils/rspec/spec_helper.rb:37
          require at org/jruby/RubyKernel.java:1040
           (root) at /home/ubuntu/workspace/<path_to_project>/spec/spec_helper.rb:1
          require at org/jruby/RubyKernel.java:1040
           (root) at /home/ubuntu/workspace/<path_to_project>/spec/spec_helper.rb:2
             load at org/jruby/RubyKernel.java:1059

ERROR: Could not find a valid gem 'logstash-devutils' (= 0.0.15) in any repository

Hi guys, I'm newbie to ruby as you will noticed but I wanted to install the logstash-devutils gem (https://rubygems.org/gems/logstash-devutils/versions/0.0.15-java) on the Mac following this command :

gem install --local logstash-devutils -v 0.0.15

And i get below error message:
:
ERROR: Could not find a valid gem 'logstash-devutils' (= 0.0.15) in any repository

Any thoughts?

I also tried latest version of the gem
gem install logstash-devutils
ERROR: Could not find a valid gem 'logstash-devutils' (>= 0), here is why:
Found logstash-devutils (1.3.1), but was for platform java

Download of artifacts from URLs doesn't follow redirects.

Create a rakefile that invokes the download method on an url that has redirect, test_download_file.rake in this project root, as:

import "lib/logstash/devutils/rake/vendor.rake"

task "test_download" do
  output = "vendor/collectd-5.4.2.tar.gz"
  url = "https://github.com/collectd/collectd/archive/refs/tags/collectd-5.4.2.tar.gz"
  actual_sha1 = download(url, output)
end

and execute the task as:

rake --rakefile test_download_file.rake test_download

Checking the content of vendor/collectd-5.4.2.tar.gz would result in empty file

Decompression of downloaded tar.gz doesn't determine the right extraction folder

If a tar.gz archive decompresses in a folder which is not the initial part of the archive name, that it can't put the files attribute into the vendor folder.

Suppose the file name is collectd-5.4.2.tar.gz (here is the file) which contains the folder collectd-collectd-5.4.2.
The prefix extraction

logstash-devutils/lib/logstash/devutils/rake/vendor.rake

Lines 174 to 177 in b695f47

    
           prefix = download.gsub('.tar.gz', '').gsub('vendor/', '') 
        
           untar(download) do |entry| 
        
             if !file['files'].nil? 
        
               next unless file['files'].include?(entry.full_name.gsub(prefix, ''))

in this case doesn't work, because the prefix from the file name is collectd-5.4.2 while the files of the archive are contained in folder collectd-collectd-5.4.2

LogStash configuration is compiled for every sample

We have a quite complex configuration file (approximately 1500 lines) and I believe that this line is costly: https://github.com/elastic/logstash/blob/master/lib/logstash/pipeline.rb#L26

If I understand correctly this code, a new pipeline is created for every sample() method, meaning that the configuration is compiled every time.

Meaning that the following test will load and compile 3 times the same configuration:

require "test_utils"

describe "apache common log format" do
  extend LogStash::RSpec

  config %q{
    # Your logstash filter config here
  }

  sample "apache log in debug" do
    insist { subject["level"] } == "DEBUG"
  end
  sample "apache log in info" do
    insist { subject["level"] } == "INFO"
  end
  sample "apache log with stack trace" do
    insist { subject["stacktrace"] } == "oups!"
  end
end

Would it be possible to configure this behavior ? We have 300+ samples and running our tests suite is really slow.
Having said that I think that this behavior should be the default. If I do want to use a different configuration I can call config again with the new configuration, no ?

Redirected download doesn't validate SHA of artifact

After #105 if an URL is redirected then the SHA validation breaks because try to compare with empty string.

% bundle exec rake vendor
Downloading https://github.com/collectd/collectd/archive/refs/tags/collectd-5.4.2.tar.gz
Follow redirect to: https://codeload.github.com/collectd/collectd/tar.gz/refs/tags/collectd-5.4.2
rake aborted!
SHA1 does not match (expected '69d00bf0589adc53c65e48ec18fe636b3efe2306' but got '')
/Users/andrea/.rvm/gems/jruby-9.4.2.0/gems/logstash-devutils-2.6.1-java/lib/logstash/devutils/rake/vendor.rake:20:in `fetch'
/Users/andrea/.rvm/gems/jruby-9.4.2.0/gems/logstash-devutils-2.6.1-java/lib/logstash/devutils/rake/vendor.rake:31:in `block in file_fetch'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
/Users/andrea/.rvm/gems/jruby-9.4.2.0/gems/logstash-devutils-2.6.1-java/lib/logstash/devutils/rake/vendor.rake:36:in `file_fetch'
/Users/andrea/.rvm/gems/jruby-9.4.2.0/gems/logstash-devutils-2.6.1-java/lib/logstash/devutils/rake/vendor.rake:172:in `block in <main>'
/Users/andrea/.rvm/gems/jruby-9.4.2.0/gems/logstash-devutils-2.6.1-java/lib/logstash/devutils/rake/vendor.rake:171:in `block in <main>'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
org/jruby/ext/monitor/Monitor.java:82:in `synchronize'
/Users/andrea/.rvm/gems/jruby-9.4.2.0/gems/rake-13.1.0/exe/rake:27:in `<main>'
/Users/andrea/.rvm/gems/jruby-9.4.2.0/bin/jruby_executable_hooks:22:in `<main>'
Tasks: TOP => vendor/collectd-5.4.2.tar.gz
(See full trace by running task with --trace)

The problem is

SHA1 does not match (expected '69d00bf0589adc53c65e48ec18fe636b3efe2306' but got '')

wrong number of arguments (given 0, expected 1)

Hello,

I have unit tests that are working perfectly with logstash 7.17. When upgrading to 8.10, all the tests fail with the following error:

An error occurred while loading ./spec/05_logstash_squid_service_prd_spec.rb.
Failure/Error:
  sample('message' => message, 'type' => 'logs-systems', 'logStream' => 'logs-squid-service-prd/service/b99b3b857c8148b19bc73bc8b85aaba4', '@timestamp' => LogStash::Timestamp.coerce('2020-01-14T11:35:54.795Z')) do # <2>
    # Check the ouput event/message properties
    insist { subject.class } == LogStash::Event # <3>
    insist { subject.get('type') } == 'logs-systems'
    insist { subject.get('@timestamp').to_iso8601 } == '2020-02-26T11:47:08.467Z'
    insist { subject.get('system') } == 'squid'
    insist { subject.get('subcomponent') } == 'service'
    insist { subject.get('env') } == 'prd'
    insist { subject.get('url') } == 'http://app.abc.com/api/abc'
    insist { subject.get('cache_result') } == 'TCP_MISS'

ArgumentError:
  wrong number of arguments (given 0, expected 1)
# ./vendor/bundle/jruby/3.1.0/gems/logstash-devutils-2.5.0-java/lib/logstash/devutils/rspec/logstash_helpers.rb:67:in `sample'
# ./spec/05_logstash_squid_service_prd_spec.rb:15:in `block in <main>'
# ./spec/05_logstash_squid_service_prd_spec.rb:10:in `<main>'

Here is the content of the spec:

# encoding: utf-8

require 'logstash/devutils/rspec/spec_helper'

# Load the configuration file
@@configuration = String.new
@@configuration << File.read('/etc/logstash/conf.d/kinesis-filter.conf')

describe 'Kinesis filter' do
  config(@@configuration) # <1>

  # Inject input event/message into the pipeline
  message = '1582717628.467   4724 10.0.4.126 TCP_MISS/200 76725 GET http://app.abc.com/api/abc - - - ORIGINAL_DST/104.24.116.191 application/json'
  sample('message' => message, 'type' => 'logs-systems', 'logStream' => 'logs-squid-service-prd/service/b99b3b857c8148b19bc73bc8b85aaba4', '@timestamp' => LogStash::Timestamp.coerce('2020-01-14T11:35:54.795Z')) do # <2>
    # Check the ouput event/message properties
    insist { subject.class } == LogStash::Event # <3>
    insist { subject.get('type') } == 'logs-systems'
    insist { subject.get('@timestamp').to_iso8601 } == '2020-02-26T11:47:08.467Z'
    insist { subject.get('system') } == 'squid'
    insist { subject.get('subcomponent') } == 'service'
    insist { subject.get('env') } == 'prd'
    insist { subject.get('url') } == 'http://app.abc.com/api/abc'
    insist { subject.get('cache_result') } == 'TCP_MISS'
    insist { subject.get('server_ip') } == '104.24.116.191'
    insist { subject.get('client_address') } == '10.0.4.126'
    insist { subject.get('bytes') } == 76725
    insist { subject.get('duration') } == 4724
    expect(subject.get('tags').include?('_grokparsefailure')).to be_falsey
    expect(subject.get('tags').include?('_dateparsefailure')).to be_falsey
  end
end

What can be the reason for this?

Thanks

When testing aggregate plugin get a ConfigurationError for a valid configuration

Filed a bug against logstash-filter-aggregate plugin but perhaps I should have filed it here. Have been struggling to test my logstash configurations using logstash-devutils. Testing based on this post https://gquintana.github.io/2016/09/07/Testing-Logstash-configuration.html . I define a valid logstash configuration, test it's validity via the logstash tool, run it, all good, etc.

When testing with logstash-devutils get a ConfigurationError but I am only defining one timeout. Even when slimming down my example shown in 119 to have only a single aggregate block in the filter section get the same error. It's really a shame because I was looking forward to learning more about the logstash development ecosystem. But I've lost many days trying to write simple tests.

Aggregate plugin: For task_id pattern '%{eventId}', there are more than one filter which defines timeout options. All timeout options have to be defined in only one aggregate filter per task_id pattern. Timeout options are : timeout, inactivity_timeout, timeout_code, push_map_as_event_on_timeout, push_previous_map_as_event, timeout_timestamp_field, timeout_task_id_field, timeout_tags

logstash-plugins/logstash-filter-aggregate#119

Update version to be inline with to LS 2.0.0 shutdown specifics

When backporting plugin code to 1.5, specs will not run if gemspec is not tied to a version before the one that introduced support for LS 2.0.0

pipeline mocking incompatible with ng-pipeline

For example the input method at

logstash-devutils/lib/logstash/devutils/rspec/logstash_helpers.rb

Lines 68 to 76 in 15c59bf

    
           pipeline.instance_eval do 
        
             # create closure to capture queue 
        
             @output_func = lambda { |event| queue << event } 
        
             # output_func is now a method, call closure 
        
             def output_func(event) 
        
               @output_func.call(event) 
        
             end 
        
           end

is very implementation specific and does not work in ng-pipeline.

/cc @andrewvc

Use new method name for LogStash::Logging::Logger::initialize

lib/logstash/devutils/rspec/spec_helper.rb line 27
new method is LogStash::Logging::Logger::reconfigure
rspec complains about stubbing any method called initialize

Add test coverage for the rspec helper

Even rspec helpers need love and we should take care of writing tests for them.

	prefix = download.gsub('.tar.gz', '').gsub('vendor/', '')
	untar(download) do \|entry\|
	if !file['files'].nil?
	next unless file['files'].include?(entry.full_name.gsub(prefix, ''))

	pipeline.instance_eval do
	# create closure to capture queue
	@output_func = lambda { \|event\| queue << event }

	# output_func is now a method, call closure
	def output_func(event)
	@output_func.call(event)
	end
	end

elastic / logstash-devutils Goto Github PK

logstash-devutils's People

Stargazers

Watchers

Forkers

logstash-devutils's Issues

Recommend Projects

Recommend Topics

Recommend Org