bsm / bfs.rb Goto Github PK

When writing a file for upload to S3 exceptions in the file writing process are ignored, causing incomplete files to be uploaded to S3. Unless specifically requested otherwise, e.g. for a stream writer where getting some data up is better than no data, any exception in the Tempwriter should cause feed upload to be abandoned.

https://github.com/bsm/bfs.rb/blob/master/s3/lib/bfs/bucket/s3.rb#L94

There is also similar logic in FeedX so it may be that this needs to be fixed in 2 places.

https://github.com/bsm/bfs.rb/blob/master/s3/lib/bfs/bucket/s3.rb#L159

BFS.register('s3') do |url|
  params = CGI.parse(url.query.to_s)

  BFS::Bucket::S3.new url.host,
    region: params.key?('region') ? params['region'].first : nil,
    sse: params.key?('sse') ? params['sse'].first : nil,
    access_key_id: params.key?('access_key_id') ? params['access_key_id'].first : nil,
    secret_access_key: params.key?('secret_access_key') ? params['secret_access_key'].first : nil,
    acl: params.key?('acl') ? params['acl'].first : nil,
    storage_class: params.key?('storage_class') ? params['storage_class'].first : nil
end

I'd say, let's do smth like attrs.compact there?

This way, it will be consistent with AWS gem(s), which can accept ENV vars like access/secret key.

I can PR it (but without the .compact method unless activesupport is included here), should I?

When stdout returns a large amount of data it may be split into multiple on_data responses. When that happens a line may be split between 2 on_data calls. We're processing each of these blocks individually, which causes us to have unexpected data at the boundaries. We will likely need to buffer the data and only process based on newline boundaries.