kjohnston / s3_relay Goto Github PK

View Code? Open in Web Editor NEW

33.0 33.0 19.0 133 KB

Direct uploads to S3 and ingestion by your Rails app.

License: MIT License

Ruby 80.31% CoffeeScript 6.94% CSS 1.88% JavaScript 1.85% HTML 9.01%

s3_relay's People

Contributors

Stargazers

Watchers

Forkers

mazharoddin djohnson mfpiccolo hartator g-gs mejibyte jonbuda bousquet hoffmanc ctmaloney norbertszivos a1iashoor owyatt khakob eashman yunixon

s3_relay's Issues

One file error makes all files stop uploading and it's not easy to retry

Proposed fix:

Ability to keep uploading other files that don't have errors
Ability to retry failed files without having to reselect them with the file <input> button

[tutorial] Adding AWS credentials

Tried few methods known to me but none of them worked. Where to put AWS credentials and how Relay will hook them up?

Bundle issue with mimemagic (0.3.5)

Your bundle is locked to mimemagic (0.3.5), but that version could not be found in any of the sources listed in your Gemfile. If you haven't changed sources, that means the author of
mimemagic (0.3.5) has removed it. You'll need to update your bundle to a version other than mimemagic (0.3.5) that hasn't been removed in order to install.

The 0.3.5 version was yanked due to license issues.

Reference: rails/rails#41750

No events API to create own UI elements & JS listeners

Proposed events (and parameters) to implement:
-Batch upload started (list of files)
-Batch progress (progress)
-Batch upload finished (files[errors])
-File upload started (file)
-File upload progress (file[progress])
-File upload finished (file)
-File error (file[error details])

Backwards compatibility
Gem needs backwards compatibility with existing implementations so I was going to leave the existing UI elements in as overwriteable defaults that use the new events API

S3 endpoint template doesn't work with "us-east-1" region

The following AWS S3 endpoint template in S3Relay::Base#endpoint:

"https://#{bucket}.s3-#{region}.amazonaws.com"

Doesn't work with the (most widespread) us-east-1 region, as these are the only valid enpoints for the region:

s3.amazonaws.com
s3.us-east-1.amazonaws.com
s3-external-1.amazonaws.com

I suggest to change the template to "https://#{bucket}.s3.#{region}.amazonaws.com" (dot instead of dash), as it seems to be the only template working with all the regions.

Thanks!

File upload is tied to parent record

Per @bousquet we want the ability to decouple the upload from a particular parent record, and per @kjohnston we want uploads tied to the user. Would it be fine to decouple from both, and then use the javascript API to POST whatever data to whatever endpoint you want?

Questions:
-Is this configured in Rails or in the JS? In the template tag/helper?
-For backwards compatibility it seems like it will need to notify parent as default. Could we have maybe 3 options: parent, user, none?

Directly inheriting from ActiveRecord::Migration is not supported on Rails 5

I'm using Rails 5.1.4 with ruby 2.5.0, when run

rake s3_relay:install:migrations db:migrate

got this error:

Copied migration 20180323143543_create_s3_relay_uploads.s3_relay.rb from s3_relay
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Directly inheriting from ActiveRecord::Migration is not supported. Please specify the Rails release the migration was written for:

  class CreateS3RelayUploads < ActiveRecord::Migration[4.2]
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:525:in `inherited'
.../db/migrate/20180323143543_create_s3_relay_uploads.s3_relay.rb:2:in `<top (required)>'
.../.rvm/gems/ruby-2.5.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
.../.rvm/gems/ruby-2.5.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `block in require'
.../.rvm/gems/ruby-2.5.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:258:in `load_dependency'
.../.rvm/gems/ruby-2.5.0/gems/activesupport-5.1.4/lib/active_support/dependencies.rb:292:in `require'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:962:in `load_migration'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:958:in `migration'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:953:in `disable_ddl_transaction'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1305:in `use_transaction?'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1297:in `ddl_transaction'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1229:in `execute_migration_in_transaction'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1201:in `block in migrate_without_lock'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1200:in `each'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1200:in `migrate_without_lock'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1148:in `block in migrate'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1317:in `with_advisory_lock'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1148:in `migrate'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:1007:in `up'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/migration.rb:985:in `migrate'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/tasks/database_tasks.rb:171:in `migrate'
.../.rvm/gems/ruby-2.5.0/gems/activerecord-5.1.4/lib/active_record/railties/databases.rake:58:in `block (2 levels) in <top (required)>'
.../.rvm/gems/ruby-2.5.0@global/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
.../.rvm/gems/ruby-2.5.0/bin/ruby_executable_hooks:15:in `eval'
.../.rvm/gems/ruby-2.5.0/bin/ruby_executable_hooks:15:in `<main>'

The solution is easy just need to add the missing version as the error message suggested:

class CreateS3RelayUploads < ActiveRecord::Migration[4.2]

but a little bit annoying thing...

Undefined method `helper' for ActionController::API:Class

Ruby 2.4.2 and Rails 5.1.4

s3_relay-0.6.0/lib/s3_relay/engine.rb:7→ block (2 levels) in <class:Engine>
--
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:69→ instance_eval
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:69→ block in execute_hook
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:60→ with_execution_control
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:65→ execute_hook
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:41→ block in on_load
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:40→ each
activesupport-5.1.4/lib/active_support/lazy_load_hooks.rb:40→ on_load
s3_relay-0.6.0/lib/s3_relay/engine.rb:6→ block in <class:Engine>
railties-5.1.4/lib/rails/initializable.rb:30→ instance_exec
railties-5.1.4/lib/rails/initializable.rb:30→ run
railties-5.1.4/lib/rails/initializable.rb:59→ block in run_initializers
/usr/local/lib/ruby/2.4.0/tsort.rb:228→ block in tsort_each
/usr/local/lib/ruby/2.4.0/tsort.rb:350→ block (2 levels) in each_strongly_connected_component
/usr/local/lib/ruby/2.4.0/tsort.rb:431→ each_strongly_connected_component_from
/usr/local/lib/ruby/2.4.0/tsort.rb:349→ block in each_strongly_connected_component
/usr/local/lib/ruby/2.4.0/tsort.rb:347→ each
/usr/local/lib/ruby/2.4.0/tsort.rb:347→ call
/usr/local/lib/ruby/2.4.0/tsort.rb:347→ each_strongly_connected_component

/usr/local/lib/ruby/2.4.0/tsort.rb:226→ tsort_each
/usr/local/lib/ruby/2.4.0/tsort.rb:205→ tsort_each
railties-5.1.4/lib/rails/initializable.rb:58→ run_initializers
railties-5.1.4/lib/rails/application.rb:353→ initialize!
config/environment.rb:5→ <top (required)>
activesupport-5.1.4/lib/active_support/dependencies.rb:292→ require
activesupport-5.1.4/lib/active_support/dependencies.rb:292→ block in require
activesupport-5.1.4/lib/active_support/dependencies.rb:258→ load_dependency
activesupport-5.1.4/lib/active_support/dependencies.rb:292→ require
railties-5.1.4/lib/rails/application.rb:329→ require_environment!
railties-5.1.4/lib/rails/application.rb:445→ block in run_tasks_blocks
rake-12.1.0/lib/rake/task.rb:251→ block in execute
rake-12.1.0/lib/rake/task.rb:251→ each
rake-12.1.0/lib/rake/task.rb:251→ execute
airbrake-5.3.0/lib/airbrake/rake/task_ext.rb:19→ execute
rake-12.1.0/lib/rake/task.rb:195→ block in invoke_with_call_chain
/usr/local/lib/ruby/2.4.0/monitor.rb:214→ mon_synchronize
rake-12.1.0/lib/rake/task.rb:188→ invoke_with_call_chain
rake-12.1.0/lib/rake/task.rb:217→ block in invoke_prerequisites
rake-12.1.0/lib/rake/task.rb:215→ each
rake-12.1.0/lib/rake/task.rb:215→ invoke_prerequisites
rake-12.1.0/lib/rake/task.rb:194→ block in invoke_with_call_chain
/usr/local/lib/ruby/2.4.0/monitor.rb:214→ mon_synchronize
rake-12.1.0/lib/rake/task.rb:188→ invoke_with_call_chain
rake-12.1.0/lib/rake/task.rb:217→ block in invoke_prerequisites
rake-12.1.0/lib/rake/task.rb:215→ each
rake-12.1.0/lib/rake/task.rb:215→ invoke_prerequisites
rake-12.1.0/lib/rake/task.rb:194→ block in invoke_with_call_chain
/usr/local/lib/ruby/2.4.0/monitor.rb:214→ mon_synchronize
rake-12.1.0/lib/rake/task.rb:188→ invoke_with_call_chain
rake-12.1.0/lib/rake/task.rb:181→ invoke
rake-12.1.0/lib/rake/application.rb:153→ invoke_task
rake-12.1.0/lib/rake/application.rb:109→ block (2 levels) in top_level
rake-12.1.0/lib/rake/application.rb:109→ each
rake-12.1.0/lib/rake/application.rb:109→ block in top_level
rake-12.1.0/lib/rake/application.rb:118→ run_with_threads
rake-12.1.0/lib/rake/application.rb:103→ top_level
rake-12.1.0/lib/rake/application.rb:81→ block in run
rake-12.1.0/lib/rake/application.rb:179→ standard_exception_handling
rake-12.1.0/lib/rake/application.rb:78→ run
rake-12.1.0/exe/rake:27→ <top (required)>
[GEM_ROOT]/bin/rake:23→ load
[GEM_ROOT]/bin/rake:23→ <top (required)>
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74→ load
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74→ kernel_load
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:27→ run
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:362→ exec
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/command.rb:27→ run
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:126→ invoke_command
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor.rb:387→ dispatch
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:22→ dispatch
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/base.rb:466→ start
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:13→ start
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/exe/bundle:30→ block in <top (required)>

/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/friendly_errors.rb:121→ with_friendly_errors
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/exe/bundle:22→ <top (required)>
/usr/local/bin/bundle:23→ load
/usr/local/bin/bundle:23→ <main>

uninitialized constant S3Relay::PrivateUrl::Addressable

I'm trying to get s3_relay running on Rails 5.1 and getting an POST /s3_relay/uploads. Full stack trace below:

Started POST "/s3_relay/uploads" for ::1 at 2017-07-29 20:11:48 -0500
Processing by S3Relay::UploadsController#create as */*
  Parameters: {"parent_type"=>"thing", "parent_id"=>"1", "association"=>"photo_uploads", "uuid"=>"36cb2295-0f0c-424f-834a-5b7b229a9697", "filename"=>"GOPR1273.JPG", "content_type"=>"image/jpeg", "public_url"=>"https://[redacted].s3-us-west-2.amazonaws.com/37cb2295-0e0c-414f-834a-5b7b229a9597%2FGOPR1273.JPG"}
  Thing Load (0.1ms)  SELECT  "things".* FROM "things" WHERE "things"."id" = ? LIMIT ?  [["id", 1], ["LIMIT", 1]]
   (0.2ms)  begin transaction
  S3Relay::Upload Exists (0.2ms)  SELECT  1 AS one FROM "s3_relay_uploads" WHERE "s3_relay_uploads"."uuid" = ? LIMIT ?  [["uuid", "<36 bytes of binary data>"], ["LIMIT", 1]]
  SQL (4.0ms)  INSERT INTO "s3_relay_uploads" ("uuid", "parent_type", "parent_id", "upload_type", "filename", "content_type", "state", "pending_at", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)  [["uuid", "<36 bytes of binary data>"], ["parent_type", "Thing"], ["parent_id", 1], ["upload_type", "PhotoUpload"], ["filename", "GOPR1273.JPG"], ["content_type", "image/jpeg"], ["state", "pending"], ["pending_at", "2017-07-30 03:11:48.800299"], ["created_at", "2017-07-30 03:11:48.803264"], ["updated_at", "2017-07-30 03:11:48.803264"]]
   (2.0ms)  commit transaction
Completed 500 Internal Server Error in 44ms (ActiveRecord: 6.8ms)


  
NameError (uninitialized constant S3Relay::PrivateUrl::Addressable):
  
s3_relay (0.6.0) lib/s3_relay/private_url.rb:7:in `initialize'
s3_relay (0.6.0) app/models/s3_relay/upload.rb:44:in `new'
s3_relay (0.6.0) app/models/s3_relay/upload.rb:44:in `private_url'
s3_relay (0.6.0) app/controllers/s3_relay/uploads_controller.rb:15:in `create'
actionpack (5.1.2) lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
actionpack (5.1.2) lib/abstract_controller/base.rb:186:in `process_action'
actionpack (5.1.2) lib/action_controller/metal/rendering.rb:30:in `process_action'
actionpack (5.1.2) lib/abstract_controller/callbacks.rb:20:in `block in process_action'
activesupport (5.1.2) lib/active_support/callbacks.rb:131:in `run_callbacks'
actionpack (5.1.2) lib/abstract_controller/callbacks.rb:19:in `process_action'
actionpack (5.1.2) lib/action_controller/metal/rescue.rb:20:in `process_action'
actionpack (5.1.2) lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
activesupport (5.1.2) lib/active_support/notifications.rb:166:in `block in instrument'
activesupport (5.1.2) lib/active_support/notifications/instrumenter.rb:21:in `instrument'
activesupport (5.1.2) lib/active_support/notifications.rb:166:in `instrument'
actionpack (5.1.2) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (5.1.2) lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
activerecord (5.1.2) lib/active_record/railties/controller_runtime.rb:22:in `process_action'
actionpack (5.1.2) lib/abstract_controller/base.rb:124:in `process'
actionview (5.1.2) lib/action_view/rendering.rb:30:in `process'
actionpack (5.1.2) lib/action_controller/metal.rb:189:in `dispatch'
actionpack (5.1.2) lib/action_controller/metal.rb:253:in `dispatch'
actionpack (5.1.2) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (5.1.2) lib/action_dispatch/routing/route_set.rb:31:in `serve'
actionpack (5.1.2) lib/action_dispatch/journey/router.rb:46:in `block in serve'
actionpack (5.1.2) lib/action_dispatch/journey/router.rb:33:in `each'
actionpack (5.1.2) lib/action_dispatch/journey/router.rb:33:in `serve'
actionpack (5.1.2) lib/action_dispatch/routing/route_set.rb:832:in `call'
railties (5.1.2) lib/rails/engine.rb:522:in `call'
railties (5.1.2) lib/rails/railtie.rb:185:in `public_send'
railties (5.1.2) lib/rails/railtie.rb:185:in `method_missing'
actionpack (5.1.2) lib/action_dispatch/routing/mapper.rb:17:in `block in <class:Constraints>'
actionpack (5.1.2) lib/action_dispatch/routing/mapper.rb:46:in `serve'
actionpack (5.1.2) lib/action_dispatch/journey/router.rb:46:in `block in serve'
actionpack (5.1.2) lib/action_dispatch/journey/router.rb:33:in `each'
actionpack (5.1.2) lib/action_dispatch/journey/router.rb:33:in `serve'
actionpack (5.1.2) lib/action_dispatch/routing/route_set.rb:832:in `call'
rack (2.0.3) lib/rack/etag.rb:25:in `call'
rack (2.0.3) lib/rack/conditional_get.rb:38:in `call'
rack (2.0.3) lib/rack/head.rb:12:in `call'
rack (2.0.3) lib/rack/session/abstract/id.rb:232:in `context'
rack (2.0.3) lib/rack/session/abstract/id.rb:226:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/cookies.rb:613:in `call'
activerecord (5.1.2) lib/active_record/migration.rb:556:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
activesupport (5.1.2) lib/active_support/callbacks.rb:97:in `run_callbacks'
actionpack (5.1.2) lib/action_dispatch/middleware/callbacks.rb:24:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/executor.rb:12:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
railties (5.1.2) lib/rails/rack/logger.rb:36:in `call_app'
railties (5.1.2) lib/rails/rack/logger.rb:24:in `block in call'
activesupport (5.1.2) lib/active_support/tagged_logging.rb:69:in `block in tagged'
activesupport (5.1.2) lib/active_support/tagged_logging.rb:26:in `tagged'
activesupport (5.1.2) lib/active_support/tagged_logging.rb:69:in `tagged'
railties (5.1.2) lib/rails/rack/logger.rb:24:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/request_id.rb:25:in `call'
rack (2.0.3) lib/rack/method_override.rb:22:in `call'
rack (2.0.3) lib/rack/runtime.rb:22:in `call'
activesupport (5.1.2) lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/executor.rb:12:in `call'
actionpack (5.1.2) lib/action_dispatch/middleware/static.rb:125:in `call'
rack (2.0.3) lib/rack/sendfile.rb:111:in `call'
railties (5.1.2) lib/rails/engine.rb:522:in `call'
rack (2.0.3) lib/rack/handler/webrick.rb:86:in `service'

::1 - - [29/Jul/2017:20:11:48 PDT] "POST /s3_relay/uploads HTTP/1.1" 500 11790
http://localhost:3000/things/1/edit -> /s3_relay/uploads

Rails 5 Deprecation warning

I am getting a the following warning:

DEPRECATION WARNING: Passing string to define callback is deprecated and will be removed in Rails 5.1 without replacement. (called from s3_relay at /Users/mfpiccolo/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/bundler/gems/s3_relay-1b7710f12867/lib/s3_relay/model.rb:40)

Reading config from secrets.yml?

As folks move to Rails 5.1 with encrypted secrets it might make sense to allow the config to be set from Rails.application.secrets... or have a way to set configuration in an initializer to be a bit more flexible? Happy to issue to a PR if you're open to it.

S3 Relay for Mongoid

Not really an issue, and thanks for the awesome work done in s3_relay.

I've ported your Gem to support a MongoDB wrapper, Mongoid: https://github.com/hartator/mongoid-direct-s3-upload

Do you want me to make changes concerning the wording referring back to this repository?

I've done a few modifications not directly related to the Mongoid port. Notably the addition of an helper to get the S3 url with the signature part. (ie. logo_upload.public_url) And, I don't know if it's on purpose or an overlook, but when dealing with one attachment, it was returning the least recent one, in opposition of the most recent one (.lastusing DESC on pending_at will return the very first upload).

Refs:

Maybe, we can backport these two changes to the original Gem.

Broken private_url with chars: '#', '?' or '\' in filename

I think we need to sanitize/encode filename or remove some special chars from the file name for private urls

example:

file name: test#4554 [8901].pdf
broken private_url: https://s3.xxxx/xxxx-3cbd-4b3b-b63d-ddacc92907b3/test#4554%20%5B8901%5D.pdf?AWSAccessKeyId=xxxx&Expires=1629181031&Signature=xxxx%2BD2j3Fb8vCBJWbWo%3D