logstash-plugins / logstash-output-google_cloud_storage Goto Github PK

License: Apache License 2.0

Ruby 100.00%

logstash-output-google_cloud_storage's Introduction

Logstash Plugin

This is a plugin for Logstash.

It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

Documentation

Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one central location.

For formatting code or config example, you can use the asciidoc [source,ruby] directive
For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide

Need Help?

Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.

Developing

1. Plugin Development and Testing

Code

To get started, you'll need JRuby with the Bundler gem installed.
You'll also need a Logstash installation to build the plugin against.
Create a new plugin or clone and existing from the GitHub logstash-plugins organization. We also provide example plugins.
export LOGSTASH_SOURCE=1 and point LOGSTASH_PATH to a local Logstash e.g. export LOGSTASH_PATH=/opt/local/logstash-8.7.0
Install Ruby dependencies

bundle install

Install Java dependencies - regenerates the lib/logstash-output-google_cloud_storage_jars.rb script used to load the .jar dependencies when the plugin starts.

./gradlew vendor

NOTE: This step is necessary whenever build.gradle is updated.

Test

Update your dependencies

bundle install

Run Ruby tests

bundle exec rspec

2. Running your unpublished Plugin in Logstash

2.1 Run in a local Logstash clone

Edit Logstash Gemfile and add the local plugin path, for example:

gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"

Install plugin

bin/logstash-plugin install --no-verify

Run Logstash with your plugin

bin/logstash -e 'filter {awesome {}}'

At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.

2.2 Run in an installed Logstash

You can use the same 2.1 method to run your plugin in an installed Logstash by editing its Gemfile and pointing the :path to your local plugin development directory or you can build the gem and install it using:

Build your plugin gem

gem build logstash-filter-awesome.gemspec

Install the plugin from the Logstash home

bin/logstash-plugin install --no-verify

Start Logstash and proceed to test the plugin

Contributing

All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.

Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.

It is more important to the community that you are able to contribute.

For more information about contributing, see the CONTRIBUTING file.

logstash-output-google_cloud_storage's People

Contributors

Stargazers

Watchers

logstash-output-google_cloud_storage's Issues

Errno::ENOENT: No such file or directory

Hi all,

I upgraded my logstash image to 7.2.0 and when the pipeline restarted, it started throwing an Errno::ENOENT: No such file or directory error, where it seems to be trying to access/upload a temporary log file that doesn't exist any longer.

Version: 4.0.1

Output Section:

output {
  if [type] == "log" {
    google_cloud_storage { 
      bucket => "my-bucket"
      log_file_prefix => "logs/%{+YYYY/MM/dd}"
      codec => "json_lines"
      gzip => true
      max_file_size_kbytes => 5242
    }
  }

  if [type] == "raw-log" {
    google_cloud_storage { 
      bucket => "my-bucket"
      log_file_prefix => "logs/raw/%{+YYYY/MM/dd}/"
      codec => "json_lines"
      gzip => true
      max_file_size_kbytes => 5242
    }
  }
}

Error trace

Pipeline aborted due to error 
{
  :pipeline_id=>"main", 
  :exception=>
    #<Errno::ENOENT: No such file or directory - 
    /tmp/logstash-gcs-504f0f1052346b639f01477069a72f1f76be227ff8e41870e5f7859b5be1/logs/%{+YYYY/MM/dd}_logstasher-785c699966-sg4c9_2019-08-05T19:00.part001.log.gz>, 
  :backtrace=>[
    "org/jruby/RubyIO.java:1236:in `sysopen'", 
    "org/jruby/RubyFile.java:367:in `initialize'", 
    "org/jruby/RubyIO.java:876:in `new'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/temp_log_file.rb:29:in `initialize'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/temp_log_file.rb:14:in `create'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/log_rotate.rb:51:in `block in rotate_log!'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.5/lib/concurrent/atomic/reentrant_read_write_lock.rb:147:in `with_write_lock'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/log_rotate.rb:42:in `rotate_log!'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/log_rotate.rb:19:in `initialize'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/google_cloud_storage.rb:266:in `initialize_log_rotater'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/google_cloud_storage.rb:164:in `register'", 
    "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", 
    "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:192:in `block in register_plugins'", 
    "org/jruby/RubyArray.java:1792:in `each'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `register_plugins'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:462:in `maybe_setup_out_plugins'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:204:in `start_workers'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `run'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:105:in `block in start'"
  ], 
  :thread=>"#<Thread:0x7f6c858e run>"
}

Error when reading p12 file

Hi,

I'm getting the following error when trying to use this plugin:

Error: The following config files contains non-ascii characters but are not UTF-8 encoded [\"/etc/path/to/file/gcs.p12\"]

Any idea?

deprecate `max_file_size_kbytes` and add `max_file_size`

We should deprecate max_file_size_kbytes and replace it with max_file_size which would use the :bytes validation type so users can enter human-readable numbers like 10mb and also have more fine-grained control over their desired file sizes.

log_file_prefix and date_pattern don't support the use of forward slashes

Logstash information:

Please include the following information:

Logstash version: 7.13.2
Logstash installation source: Docker
How is Logstash being run: Kubernetes
How was the Logstash Plugin installed: Dockerfile line: RUN ./bin/logstash-plugin install logstash-output-google_cloud_storage

OS version Linux kernel 4.14.174 (I believe the Docker image is based on CentOS)

Description of the problem including expected versus actual behavior:

Putting a / character in log_file_prefix (e.g. log_file_prefix => "my-prefix/") causes i/o to the temp_directory location to fail with errors like the following:

[2021-08-20T10:00:00.000][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Errno::ENOENT: No such file or directory - /tmp/logstash-gcs/my-prefix/logstash_gcs_cw054563.local_2021-08-20T10:00.part001.log.gz>, :backtrace=>["org/jruby/RubyIO.java:1239:in `sysopen'"

Presumably this is because it's trying to write a file into a subdirectory of temp_directory that doesn't exist yet, assuming that all temp files should be written to one flat directory, and uploaded to GCS as such as well. Similar symptoms arise when putting forward slashes in date_pattern.

See #45 and #46 for further examples of this.

Steps to reproduce:

A pipeline to reproduce:

input { generator { } }
output {
  google_cloud_storage {
    bucket => "my-bucket"
    json_key_file => "/secrets/credentials.json"
    log_file_prefix => "my-prefix/"
  }
}

Deleted files remain in use by the system eventually filling up disk space

We're using a slightly modified version of this plugin (3.0.4) in our infrastructure. There are two differences:

just below

message = LogStash::Json.dump(event.to_hash)

on line 149, we have:

message = message.gsub(/\"@(\w+)\"/, '"\1"')

to remove "@" sign from the field names to make Bigquery happy. The second difference is the filename format. Instead of doing this in get_base_path:

  def get_base_path
    return @temp_directory + File::SEPARATOR + @log_file_prefix + "_" +
      Socket.gethostname() + "_" + Time.now.strftime(@date_pattern)
  end

we do this:

  def get_base_path
    return @temp_directory + File::SEPARATOR + @log_file_prefix + "_" +
      Time.now.strftime(@date_pattern) + "_" + Socket.gethostname()
  end

so it is possible to use prefix on Google GCS with a single wildcard to load daily files into Google Bigquery -- i.e., instead of "prefix_hostname_YYYY-MM-DD" file names will have the pattern "prefix_YYYY-MM-DD_hostname" which makes it possible to use a single wildcard to load into BQ when there are multiple logstash instances.

Ex: prefix_YYYY-MM-DD_* which works vs prefix_host*YYYY-MM-DD* which does not)

Version: Logstash-5.6.2-1 and logstash-output-google_cloud_storage-3.0.4
Operating System: CentOS6
Config File (if you have sensitive info, please remove it):

  google_cloud_storage {
    bucket => "mybucket"
    key_path => "/ssl/key"
    service_account =>"service@account"
    temp_directory => "/tmp/logstash_storage"
    date_pattern => "%Y-%m-%d"
    flush_interval_secs => 2
    uploader_interval_secs => 180
    max_file_size_kbytes => 102400
    gzip => true
    output_format => "json"
    codec => "json"
  }

The instances that run logstash with this plugin eventually run out of disk space. When I run lsof I get this:

$ sudo lsof | grep deleted | grep shipper
java 28456 logstash 127r REG 8,1 104859155 6242 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part012.log.gz (deleted)
java 28456 logstash 139r REG 8,1 104862422 8797 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part013.log.gz (deleted)
java 28456 logstash 154r REG 8,1 104860142 13306 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part014.log.gz (deleted)
java 28456 logstash 155r REG 8,1 104865750 22383 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part008.log.gz (deleted)
java 28456 logstash 156r REG 8,1 104864309 13003 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part010.log.gz (deleted)
java 28456 logstash 157r REG 8,1 104864216 12593 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part011.log.gz (deleted)
java 28456 logstash 158r REG 8,1 104860763 4211 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part016.log.gz (deleted)
java 28456 logstash 162r REG 8,1 104859803 13363 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part017.log.gz (deleted)
java 28456 logstash 163r REG 8,1 104865470 3907 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part009.log.gz (deleted)
java 28456 logstash 164r REG 8,1 104858908 22297 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part019.log.gz (deleted)
java 28456 logstash 165r REG 8,1 104860501 8871 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part015.log.gz (deleted)
java 28456 logstash 166r REG 8,1 104861010 22365 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part024.log.gz (deleted)
java 28456 logstash 167r REG 8,1 104860015 12764 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part021.log.gz (deleted)
java 28456 logstash 168r REG 8,1 104859781 11550 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part018.log.gz (deleted)
java 28456 logstash 170r REG 8,1 104861647 13002 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part020.log.gz (deleted)
java 28456 logstash 172r REG 8,1 104860524 22301 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part022.log.gz (deleted)
java 28456 logstash 173r REG 8,1 104857893 5064 /tmp/logstash_storage/logstash_gcs_2018-02-08_logstash_shipper_hostname.part023.log.gz (deleted)
$

Can't install plugin with logstash 2.3.4

Can't install google_cloud_storage plugin with logstash 2.3.4
I get following error:

Error Bundler::InstallError, retrying 1/10
An error occurred while installing google-api-client (0.9.11), and Bundler cannot continue.
Make sure that `gem install google-api-client -v '0.9.11'` succeeds before bundling.
WARNING: SSLSocket#session= is not supported

Version:
Logstash 2.3.4
Operating System:
Ubuntu 16.04
Steps to Reproduce:

On a fresh ubuntu 16.04 installation (in my case ubuntu:16.04 docker image)

export LOGSTASH_MAJOR=2.3
export DEBIAN_FRONTEND=noninteractive


apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4
echo "deb http://packages.elastic.co/logstash/${LOGSTASH_MAJOR}/debian stable main" > /etc/apt/sources.list.d/logstash.list
apt-key adv --keyserver ha.pool.sks-keyservers.net --recv-keys EEA14886
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu vivid main" > /etc/apt/sources.list.d/java.list
echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 boolean true" | debconf-set-selections

apt-get install -y oracle-java8-installer logstash
/opt/logstash/bin/logstash-plugin install logstash-output-google_cloud_storage

Update README to use this plugin in the instructions

As of now, the README install instructions are for the logstash-filter-awesome plugin.

Plugin doesn't start on logstash 7.1.1 - TypeError

Way to reproduce

Install fresh version of logstash 7.1.1 as zip file, Debian 9
Install plugin using bin/logstash-plugin install logstash-output-google_cloud_storage
Start logstash using very simple config

input { stdin { } } 
output { 
    stdout {} 
    google_cloud_storage {
        bucket => "kzalasa-xxxxx-backup"
        json_key_file => "/home/kzalasa/kzalasa-xxxxx-xxxxxx.json"
        temp_directory => "/tmp/logstash"
        log_file_prefix => "xxxx_log"
        flush_interval_secs => 2
        uploader_interval_secs => 5
        max_file_size_kbytes => 10
        output_format => "plain"
        date_pattern => "%Y-%m-%dT%H:00"    
        gzip => false            
        gzip_content_encoding => false
        upload_synchronous => false   
        include_hostname => false     
    }
}

I received following error:

[2019-06-14T13:44:30,388][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<TypeError: can't convert nil into Integer>, :backtrace=>["org/jruby/RubyString.java:1249:in `%'", "/hom
e/kzalasa/logstash-ticket/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/gcs/path_factory.rb:83:in `template_variables'", "/home/kzalasa/logstash-ticket/logstash-
7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/gcs/path_factory.rb:89:in `next_base'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstas
h-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/gcs/path_factory.rb:67:in `starting_part'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java
/lib/logstash/outputs/gcs/path_factory.rb:27:in `initialize'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/gcs/path_factory.rb:1
14:in `build_path_factory'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/gcs/path_factory.rb:102:in `build'", "/home/kzalasa/log
stash-ticket/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/google_cloud_storage.rb:219:in `initialize_path_factory'", "/home/kzalasa/logstash-ticket/logstash-7.1
.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.0-java/lib/logstash/outputs/google_cloud_storage.rb:163:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", "org/lo
gstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins'", "org/jruby/RubyArray.java:
1792:in `each'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:445:i
n `maybe_setup_out_plugins'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:203:in `start_workers'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/logstash-core/lib/logstash/java_pipelin
e.rb:145:in `run'", "/home/kzalasa/logstash-ticket/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in start'"], :thread=>"#<Thread:0x3305e54f run>"}

In file lib/logstash/outputs/gcs/path_factory.rb is a cyclic dependency:

@part_number is defined using method start_part
start_part uses next_base
next_base uses template_variables
template_variables uses @part_number, before it's defined and cause error

codec doesn't seem to do anything.

I'm using this plugin to output transformed log lines to GCS.

My goal is to be able to import those files into BQ at a later time - (streaming to BQ is $$$ right now).

When I use

output_format => "json"

I run into a problem because BQ doesn't like fields starting with @ (i.e. @timestamp)

So now I'm trying to use

output_format => "plain"

But the default format

@timestamp %{host} %{message}

doesn't suit me.
Therefore I am adding a codec.

codec => plain  { format =>  "%{[myfield]}" }

but it doesn't seem to do anything.

Would any one have recommendations on how I could achieve this?

Refactor Job Queue Architecture

This will fix #2, #5, #15, #19.

This plugin currently has some confusing architecture around the way upload jobs are done. The upload queue is used as both a queue and temporary storage, files aren't always uploaded on exit, there are race conditions around file rotation, and there is only one upload thread allowed at a time.

This proposal is to refactor the plugin to use ThreadPoolExecutor so that:

There is exactly one place files are rotated and submitted to the worker pool. Fixes: #2, #19
The job queue ONLY contains files ready to be uploaded. Fixes #2, #19
It can have multiple workers. Fixes #5
We wait for the job queue to upload all files before stopping the plugin. Fixes #15

New config options:

max_concurrent_uploads, the maximum number of concurrent uploads (workers). No effect if upload_synchronous is set. Default: 5
max_pending_uploads, the maximum number of pending uploads. No effect if upload_synchronous is set. Default: 5

Clean up Documentation

The documentation for this plugin could be greatly improved. General steps to achieve it:

Remove documentation in code, this is no longer recommended for new Logstash plugins because it leads to out-of-sync docs.
Make the example config runnable if the user copy/pastes, right now it has extra info (required), (optional) which need to be deleted
Provide example configuration for common use-cases to onboard faster and provide guidelines.
Add a section for extra resources including pricing, different storage types, ADC and service accounts.
Add documentation about minimal service account permissions needed to run the plugin.
Remove the TODOs/feature requests and add them as issues in Github.

better exception handling upon invalid temp log file name

Per #41 if an invalid temp file name is created using, for example, an invalid pattern in log_file_prefix then the resulting exception is very hard to decrypt.

    #<Errno::ENOENT: No such file or directory - 
    /tmp/logstash-gcs-504f0f1052346b639f01477069a72f1f76be227ff8e41870e5f7859b5be1/logs/%{+YYYY/MM/dd}_logstasher-785c699966-sg4c9_2019-08-05T19:00.part001.log.gz>, 
  :backtrace=>[
    "org/jruby/RubyIO.java:1236:in `sysopen'", 
    "org/jruby/RubyFile.java:367:in `initialize'", 
    "org/jruby/RubyIO.java:876:in `new'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/temp_log_file.rb:29:in `initialize'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/temp_log_file.rb:14:in `create'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/log_rotate.rb:51:in `block in rotate_log!'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.5/lib/concurrent/atomic/reentrant_read_write_lock.rb:147:in `with_write_lock'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/log_rotate.rb:42:in `rotate_log!'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/gcs/log_rotate.rb:19:in `initialize'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/google_cloud_storage.rb:266:in `initialize_log_rotater'", 
    "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-google_cloud_storage-4.0.1-java/lib/logstash/outputs/google_cloud_storage.rb:164:in `register'", 
    "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", 
    "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:192:in `block in register_plugins'", 
    "org/jruby/RubyArray.java:1792:in `each'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `register_plugins'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:462:in `maybe_setup_out_plugins'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:204:in `start_workers'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `run'", 
    "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:105:in `block in start'"

We should improve the error reporting for this.

Use `gzip` for `Content-Encoding` instead of `Content-Type`

Before diving into what the enhancement is that I am asking for, let me explain a little context for why I am asking for it.

Currently when turning on gzip in the pipeline configuration, the actual Content-Type is changed from text/plain to application/gzip and the .gz file extension is appended to the filename uploaded to GCS. This is surely fine for many use cases, but assumes that the client can properly deal with decoding gzip files, which isn't always true.

With the current set up, any clients downloading the content must add an extra explicit step of decompressing the data, whereas with simply changing the Content-Encoding, the decompression would be implicit. Changing only the Content-Encoding gives the benefit of reduced storage and transmission costs, while allowing any receiving clients to simply think of the actual content as still just being a plain text .log file (which is what it really is). This works much more inline with our use cases with the data. It also allows decompression to happen server side, if the client indicates it can't handle gzip encoding, further increasing flexibility in serving the content.

Since changing the semantics of the gzip setting at this point would surely break existing deployments that depend on the current behavior, I suggest adding another setting to control this. For instance, something along the lines of gzip_content_encoding with a default value of false should work.

I am happy to add this feature and create a pull request for it, but I thought I should open an issue first to see whether something like this would even be considered for merging or not.

Make filename to upload customizable

Disclaimer: I think this could be a missing feature, however it may be possible to achieve same result in different ways (with different plugin maybe, or by combining more of them). Also, it is more or less the first I open a feature request, I am not exactly what level of details you may need)

For my use case, having the filename fixedly bound to the socket host name is not good (or at least, that would be my understanding from https://github.com/logstash-plugins/logstash-output-google_cloud_storage/blob/master/lib/logstash/outputs/google_cloud_storage.rb#L310, I don't really know Ruby). We have an old system that copies log files to Google Cloud Storage for later consumption and I would like to push logs from logstash to GCS keeping our naming strategy.

For us, it would help to be able to customized the filename based on appender attributes. For example, we should be able to retrieve an attribute GCS_file set to my-old-filename and use my-old-filename to generate instead of Socket.gethostname().

Possibly, a transformer can be configured as well to alter the filename configured, although this would probably make life unnecessarily hard (just specify the chosen attribute with the proper value).

Prefix and suffix should still be added.

I hope it all makes sense 😄

How to output file to bucket like `bucketname\%Y-%m-%d\logstash_gcs*`?

Currently, the logstash-output-google_cloud_storage (4.0.0) is used and the config as below

output {
	google_cloud_storage {
		bucket => "my-bi"
		json_key_file => "/tmp/gcs_key.json"
		output_format => "json"
		gzip => true
		uploader_interval_secs => 60
		temp_directory => "/tmp/logstash-gcs"
	}
}

And the files in gcs buckets are

logstash_gcs_0.0.0.0_2019-03-20T13:00.part005.log.gz
logstash_gcs_0.0.0.0_2019-03-20T14:00.part001.log.gz
logstash_gcs_0.0.0.0_2019-03-20T15:00.part001.log.gz
...
logstash_gcs_0.0.0.0_2019-03-21T13:00.part005.log.gz

I want to split all those files into different folder with date as folder name,

2019-03-20
         logstash_gcs_0.0.0.0_2019-03-20T13:00.part005.log.gz
         logstash_gcs_0.0.0.0_2019-03-20T14:00.part001.log.gz
         logstash_gcs_0.0.0.0_2019-03-20T15:00.part001.log.gz
         ...
2019-03-21
        logstash_gcs_0.0.0.0_2019-03-21T13:00.part005.log.gz
       ...

How to output file to bucket like bucketname\%Y-%m-%d\logstash_gcs* ?

Add (optional) UUID to log name

This proposal is to add the ability for the user to request a UUID in the log name and optionally turn off the hostname e.g. logstash_2018-01-01T01:01:00.part1.{{UUID}}.log. This has the following benefits:

A new log won't clobber the old one after Logstash is restarted on a host with the same name. See @neuromantik33 description in #20 for more info.
Users could turn off using the hostname in their file names if:

They don't want an information leak: @joshdvir opened #9 for this
They needed to be able to use a single glob to match their logs to import into BigQuery @arslanm had to modify the plugin in #19 to do this

Thoughts @neuromantik33, @joshdvir and @arslanm?

Test reliant on timing causing nondeterministic results

It looks like there's a timing issue in this test even with nanoseconds being the timestamp. Maybe replace all instances of the %N with millis and nanos and add a sleep (or mock the calls to Time).

logstash-output-google_cloud_storage/spec/outputs/gcs/path_factory_spec.rb

Line 117 in db32da1

builder.set_date_pattern '%N'

It's causing this error to show up in CI:

https://travis-ci.org/logstash-plugins/logstash-output-google_cloud_storage/jobs/379924151

Adding "prefix" to log_file_prefix causes logstash to fail

When you add a prefix to log_file_prefix field, logstash fails with an error.
Config:

input {
  generator { }
}
output {
  google_cloud_storage {
    bucket => "my-logs"
    temp_directory => "/tmp/logstash-gcs"
    log_file_prefix => "/test/logstash_gcs"
    ...
  }
}

Error:

[2020-02-26T16:00:20,738][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Errno::ENOENT: No such file or directory - /tmp/logstash-gcs/test/logstash_gcs_cw054563.local_2020-02-26T16:00.part001.log.gz>, :backtrace=>["org/jruby/RubyIO.java:1239:in `sysopen'"

Please align features with logstash-s3-output plugin.

Adding "prefix" to bucket silently drops data

We would like to be able to use prefixes (similiar to logstash-s3-output plugin), but when you add a "prefix" to the bucket field, this plugin silently drops data.
Config like:

google_cloud_storage {
  bucket => "my-logs/test"
  ...
}

You see a message like:
[2020-02-26T15:46:26,987][INFO ][logstash.outputs.googlecloudstorage][main] Uploading file to my-logs/test/logstash_gcs_cw534756.local_2020-02-26T15:00.part036.log.gz
but the logs are not ending up in the bucket. Removing /test from the bucket field uploads the logs to the root of the bucket (without a prefix).

Silently dropping data is not good.

Google BigQuery crashes after temp file rotation

Migrated from: elastic/logstash#1599

From the issue:

hi
after investigating an major issue i had with google_bigquery plugin i've found the following:
every time a file is being deleted using the google_bigquery plugin, logstash crushes with the error it can't find the file.

to fix the issue i've added to the File.delete(filename) statement in both google_bigquery and google_cloud_storage ruby files the following change

File.delete(filename) if File.exist?(filename)
it fixes the issue as files can be deleted in couple of statements prior to that roll out

for more info:
https://logstash.jira.com/browse/LOGSTASH-2247

please implement the change in the upcoming versions of the product,
Thanks,
Dave

Doesn't work anymore

Hi,

It seems that the plugin doesn't work.
debugging shows that everything is OK, but the upload doesn't succeed (No error).
https://github.com/logstash-plugins/logstash-output-google_cloud_storage/blob/master/lib/logstash/outputs/google_cloud_storage.rb#L387
this line shows nil for both params

client.execute does not check for upload errors

When you run client.execute, you don't check for errors. The result object is Class: Google::APIClient::Result[https://www.rubydoc.info/gems/google-api-client/0.8.2/Google/APIClient/Result]. This plugin will delete the log and silently fail.

For example, if you give a service account without storage.objects.create

metadata_insert_result.error_message would report:

<service account> does not have storage.objects.create access to <bucket name>.

Cannot load key file

Using any of logstash versions 2.3.4, 2.4.0, or 5.3.2, with this configuration:

input {
file {
path => "/opt/logs/trx.log"
start_position => "beginning"
}
}

output {
google_cloud_storage {
bucket => "xxxxxx-prod-trx-logs"
key_path => "/root/logstash-5.3.1/config/trxlogger.p12"
service_account => "[email protected]"
temp_directory => "/tmp/logstash-gcs"
log_file_prefix => "logstash_gcs"
max_file_size_kbytes => 1024
output_format => "plain"
date_pattern => "%Y-%m-%dT%H:00"
flush_interval_secs => 2
gzip => true
uploader_interval_secs => 60
}
}

Logstash refuses to start as it is unable to initialize the GCS output plugin:

[2017-04-27T20:26:34,289][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x31f1ecc3 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x46124a29 @Metric=#<LogStash::Instrument::Metric:0x20cdae13 @collector=#<LogStash::Instrument::Collector:0x1e621c96 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x204b7c6d @store=#<Concurrent::Map:0x745c294f @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x39acbce1, @fast_lookup=#<Concurrent::Map:0x372481ee @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2"]>, @Metric=#<LogStash::Instrument::NamespacedMetric:0x597465e3 @Metric=#<LogStash::Instrument::Metric:0x20cdae13 @collector=#<LogStash::Instrument::Collector:0x1e621c96 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x204b7c6d @store=#<Concurrent::Map:0x745c294f @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x39acbce1, @fast_lookup=#<Concurrent::Map:0x372481ee @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0x165b5b3b @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x751cca24>, @strategy=#<LogStash::OutputDelegatorStrategies::Single:0x1923fcde @output=<LogStash::Outputs::GoogleCloudStorage bucket=>"xxxxxx-prod-trx-logs", key_path=>"/root/logstash-5.3.1/config/trxlogger.p12", service_account=>"[email protected]", temp_directory=>"/tmp/logstash-gcs", log_file_prefix=>"logstash_gcs", max_file_size_kbytes=>1024, output_format=>"plain", date_pattern=>"%Y-%m-%dT%H:00", flush_interval_secs=>2, gzip=>true, uploader_interval_secs=>60, id=>"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_335e17aa-e6af-4c70-b0d9-de7b5e70d543", enable_metric=>true, charset=>"UTF-8">, workers=>1, key_password=>"notasecret">, @mutex=#Mutex:0xd5c133>, @id="cbd98b05461273c0fdc1975a85706ffcb0e748cc-2", @metric_events=#<LogStash::Instrument::NamespacedMetric:0x4c9bd091 @Metric=#<LogStash::Instrument::Metric:0x20cdae13 @collector=#<LogStash::Instrument::Collector:0x1e621c96 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x204b7c6d @store=#<Concurrent::Map:0x745c294f @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x39acbce1, @fast_lookup=#<Concurrent::Map:0x372481ee @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"cbd98b05461273c0fdc1975a85706ffcb0e748cc-2", :events]>, @output_class=LogStash::Outputs::GoogleCloudStorage>", :error=>"Invalid keyfile or passphrase"}

I've verified with openssl that /root/logstash-5.3.1/config/trxlogger.p12 is a valid key file and that I can use the passphrase notasecret to open it. auditd confirms that the file is being opened and read by logstash.

I've tried to use the workaround in #6 and trying to open the key file directly, with the same result:

irb(main):001:0> require "google/api_client"
=> true
irb(main):002:0> Google::APIClient::KeyUtils.load_from_pkcs12('/root/logstash-5.3.1/config/trxlogger.p12', 'notasecret')
ArgumentError: Invalid keyfile or passphrase
from /root/logstash-2.3.4/vendor/jruby/lib/ruby/gems/shared/gems/google-api-client-0.8.6/lib/google/api_client/auth/key_utils.rb:88:in load_key' from /root/logstash-2.3.4/vendor/jruby/lib/ruby/gems/shared/gems/google-api-client-0.8.6/lib/google/api_client/auth/key_utils.rb:34:in load_from_pkcs12'
from (irb):2:in evaluate' from org/jruby/RubyKernel.java:1079:in eval'
from org/jruby/RubyKernel.java:1479:in loop' from org/jruby/RubyKernel.java:1242:in catch'
from org/jruby/RubyKernel.java:1242:in catch' from /root/logstash-2.3.4/vendor/jruby/bin/irb:13:in (root)'
irb(main):003:0>

Running on Ubuntu 16.04, java version "1.8.0_131", same behavior with logstash versions 2.3.4, 2.4.0, 5.3.1, and 5.3.2.

Add support for json key files

Would be convenient to have support for json keys as well, just to keep up to date!

Unable to connect to GCS (certificate verify failed)

I'm unable to use with plugin within a docker container, using the 6.1.0 and 5.6.8 tags. I'm pretty sure it isn't the key since I've generated 3 different p12 keys for the same service account and they all fail, and the stacktrace fails beforehand.

Here is my configuration (I've changed all bucket names and project ids for obvious reasons) :

Version: v3.0.4
Java version (in container):

bash-4.2$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

Ruby version (in container):

jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.151-b12 on 1.8.0_151-b12 +indy +jit [linux-x86_64]

Operating System: docker.elastic.co/logstash/logstash:6.1.0
logstash.yml:

http.host: 0.0.0.0
log.format: json
path.config: /usr/share/logstash/pipeline
config:
  debug: true
  reload:
    automatic: true
    interval: 5s
queue:
  type: persisted
  drain: true

logstash.conf:

input {
  beats {
    port => 5044
  }
}
output {
  google_cloud_storage {
    bucket => "my-bucket"
    flush_interval_secs => 5
    gzip => true
    key_path => "/shh/key.p12"
    max_file_size_kbytes => 102400
    output_format => "plain"
    service_account => "[email protected]"
    temp_directory => "/usr/share/logstash/data/tmp"
  }
}

Error message:

...
2018/03/22 11:01:25 Setting 'log.format' from environment.
2018/03/22 11:01:25 Setting 'xpack.monitoring.elasticsearch.url' from environment.
Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2018-03-22T11:02:02,331][WARN ][logstash.runner          ] --config.debug was specified, but log.level was not set to 'debug'! No config info will be logged.
[2018-03-22T11:02:02,373][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-03-22T11:02:02,397][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-03-22T11:02:03,460][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.1.0-java/modules/arcsight/configuration"}
[2018-03-22T11:02:04,125][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-03-22T11:02:05,238][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.0"}
[2018-03-22T11:02:05,845][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-03-22T11:02:08,562][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>["http://elasticsearch:9200"], bulk_path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", manage_template=>"false", document_type=>"%{[@metadata][document_type]}", sniffing=>"false", id=>"482e2490d75257e21cd2b7d49268d74674b3b7c32f0cdb0eef4694242d57f5fb">}
[2018-03-22T11:02:09,380][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2018-03-22T11:02:09,411][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
[2018-03-22T11:02:09,731][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2018-03-22T11:02:09,810][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[2018-03-22T11:02:09,874][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]}
[2018-03-22T11:02:09,914][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0xca43c07 run>"}
[2018-03-22T11:02:10,053][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2018-03-22T11:02:10,054][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
[2018-03-22T11:02:10,074][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2018-03-22T11:02:10,112][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>nil}
[2018-03-22T11:02:10,292][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2018-03-22T11:02:16,086][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x270a138 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0xd76dfbf @metric=#<LogStash::Instrument::Metric:0x7be2e6f @collector=#<LogStash::Instrument::Collector:0x7827e615 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7f59f148 @store=#<Concurrent::Map:0x00000000000fb4 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x75828e9e>, @fast_lookup=#<Concurrent::Map:0x00000000000fb8 entries=63 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :\"09d2e504e8de6887836a4879cca23b984f165252a32f136bcf5d24ff1cc04bb1\"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x4b9708da @metric=#<LogStash::Instrument::Metric:0x7be2e6f @collector=#<LogStash::Instrument::Collector:0x7827e615 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7f59f148 @store=#<Concurrent::Map:0x00000000000fb4 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x75828e9e>, @fast_lookup=#<Concurrent::Map:0x00000000000fb8 entries=63 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0xe7a6303 @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0x44474f51>>, @out_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 -  name: out value:0, @strategy=#<LogStash::OutputDelegatorStrategies::Single:0x373893af @mutex=#<Mutex:0x6a078c30>, @output=<LogStash::Outputs::GoogleCloudStorage bucket=>\"my-bucket\", flush_interval_secs=>5, gzip=>true, key_path=>\"/shh/key.p12\", log_file_prefix=>\"wt2\", max_file_size_kbytes=>102400, output_format=>\"plain\", service_account=>\"[email protected]\", temp_directory=>\"/usr/share/logstash/data/tmp\", id=>\"09d2e504e8de6887836a4879cca23b984f165252a32f136bcf5d24ff1cc04bb1\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_8dd97555-2f0c-4eb1-ab7d-e3d8650557d1\", enable_metric=>true, charset=>\"UTF-8\">, workers=>1, key_password=>\"notasecret\", date_pattern=>\"%Y-%m-%dT%H:00\", uploader_interval_secs=>60>>, @in_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 -  name: in value:0, @id=\"09d2e504e8de6887836a4879cca23b984f165252a32f136bcf5d24ff1cc04bb1\", @time_metric=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 -  name: duration_in_millis value:0, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x14c47be6 @metric=#<LogStash::Instrument::Metric:0x7be2e6f @collector=#<LogStash::Instrument::Collector:0x7827e615 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7f59f148 @store=#<Concurrent::Map:0x00000000000fb4 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x75828e9e>, @fast_lookup=#<Concurrent::Map:0x00000000000fb8 entries=63 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :\"09d2e504e8de6887836a4879cca23b984f165252a32f136bcf5d24ff1cc04bb1\", :events]>, @output_class=LogStash::Outputs::GoogleCloudStorage>", :error=>"certificate verify failed", :thread=>"#<Thread:0x7a1c234e run>"}
[2018-03-22T11:02:16,116][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Faraday::SSLError>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:228:in `connect_nonblock'", "/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:938:in `connect'", "/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:868:in `do_start'", "/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:857:in `start'", "/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/net/http.rb:1409:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/request/url_encoded.rb:15:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:177:in `post'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/signet-0.8.1/lib/signet/oauth_2/client.rb:967:in `fetch_access_token'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/signet-0.8.1/lib/signet/oauth_2/client.rb:1005:in `fetch_access_token!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/google-api-client-0.8.7/lib/google/api_client/auth/jwt_asserter.rb:105:in `authorize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-google_cloud_storage-3.0.4/lib/logstash/outputs/google_cloud_storage.rb:374:in `initialize_google_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-google_cloud_storage-3.0.4/lib/logstash/outputs/google_cloud_storage.rb:132:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:10:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:343:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:743:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:364:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:288:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"], :thread=>"#<Thread:0x7a1c234e run>"}
[2018-03-22T11:02:16,146][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: LogStash::PipelineAction::Create/pipeline_id:main, action_result: false", :backtrace=>nil}
[2018-03-22T11:02:16,241][INFO ][logstash.inputs.metrics  ] Monitoring License OK
[2018-03-22T11:02:20,620][ERROR][logstash.inputs.metrics  ] Failed to create monitoring event {:message=>"undefined method `system?' for nil:NilClass", :error=>"NoMethodError"}

As I said it doesn't seem to be a p12 key issue (that line is never reached) but some other odd behviour. Is this plugin still supported with Google Cloud's current storage API, or even logstash 5.x or 6.x for that matter. Any help would be much appreciated (which may include alternatives for shipping filebeat logs to GCS)

Thanks in advance

Fails to upload files when logstash exits

In situations where Logstash exits it is quite likely that the plugin will leave some files on disk, not uploaded to GCS. This is especially likely when large amounts of data is exported over a short execution.

My use case is exporting data from Elasticsearch to GCS using the logstash-input-elasticsearch plugin. This plugin will pull data out of Elasticsearch based on a configurable query. When all data from that query has been processed, Logstash will exit. In this case, it is almost certain that a significant amount of data will not be uploaded to GCS.

A related issue is covered in the Improvements TODO list in the header comment:

There's no recover method, so if logstash/plugin crashes, files may not be uploaded to GCS.

That description trivialities the issue by assuming that it only applies to crash situations, when in fact it applies any time Logstash exits including a user initiated shutdown and a successful completion. The export use case describe above transforms this from a potential minor problem to a fatal flaw.

Major performance issues

Hey,

When using this plugin we see a huge drop of events/s that logstash processes.

We believe its because the upload of the files are happening on the main plugin thread instead of a different one like on S3/ES plugins.

Any chance to get this implemented?