continuum / active_importer Goto Github PK

What do you think about this change? :)
46868c3

The gem is working great in my project so far, but I'm looking at ways to improve workflow for the end user. It would be great if they could import multiple model sets just by having them in separate sheets rather than having to have separate files for each model. The situation I have is that we allow the user to import a List (model) with it's ListItems (each belongs_to List) as rows in a spreadsheet. But if the user wants to import multiple Lists, they currently have to create a separate .xls file for each one and import them one at a time. (It would also be great to be able to read the tab names and assign the List.name from the tab name).

I saw that I can specify a particular sheet to import, but I can't figure out a way to loop through each sheet in a workbook and import each one in turn. Is this something that's possible? Or is there perhaps a better approach to this?

Is there a way to set a CSV column delimiter?
Currently, i can't import anything other than data separated by comma.
Would like to change it to semicolon.

I was using the previous version (0.2.4) with fantastic results. I updated to 0.2.5 today and a number of my tests started failing. Looking at the code, it appears that this is because the initialize is now raising errors in addition to sending them to the import_failed callback. I'm having trouble understanding the reasoning behind doing both. I would think that sending the errors to the import_failed callback so that my app can handle them (which is what I'm doing) would be sufficient. Am I possibly missing some other implication here?

I've set up an importer and the direct rspec tests all work great. However, when I try to implement the controller action for this, I can't get it to import using file information submitted from a file_field_tag. In my controller, I first tried:

importer = MyImporter.new(params[:file])
importer.import

This fails because the tmp file name has no extension so it tries to parse it through Roo::Google (which tells me I need the google_drive gem).

So then I tried

importer = MyImporter.new(params[:file].original_filename)
importer.import

which also failed because that filename doesn't actually exist in the filesystem. I know I can physically save the file and then pass the filename in that way. However, I'd really like to be able to import without going through a file save process. Is there a way to do this?

Spreadsheet files may have more than one sheet. This gem should somehow support specifying which sheet to process.

I realize might not be something that you want to cater for but I thought I'd document it here just in case it trips anyone else up in the future.

If the last column contains data but the header is blank the importer will fail and throw:

Fatal error while importing data: undefined method 'strip' for nil:NilClass

I'm going to look into a way to catch it in a row_* event so that I can deliver a more meaningful error message to the user.

Hello,

Will this gem be able to handle importing of houndres or thousands of records/lines?

What should be the importer behavior when an exception occurs inside an on :row_success block?

The row was supposedly processed successfully, so exceptions on this event are not captured and sent to the on :row_error handler. Actually, if an exception occurs on the success handler, the importer process ends abnormally, and it shouldn't.

The same problem applies to most if not all of the other callback events, including on :row_processed and on :row_error. How should active_importer handle exceptions occurring inside these event callbacks?

I have a local column want to import (string type)

language_locale_name
العربية
简体中文
繁體中文
français
हिन्दी, हिंदी

However, it doesn't import and no error is given

When I tried to import an xlsx file I received the next error:

Ole::Storage::FormatError - OLE2 signature is invalid:
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:378:in `validate!'
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:370:in `initialize'
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:112:in `new'
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:112:in `load'
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:79:in `initialize'
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:85:in `new'
  ruby-ole (1.2.11.8) lib/ole/storage/base.rb:85:in `open'
  spreadsheet (1.0.3) lib/spreadsheet/excel/reader.rb:1282:in `setup'
  spreadsheet (1.0.3) lib/spreadsheet/excel/reader.rb:140:in `read'
  spreadsheet (1.0.3) lib/spreadsheet/excel/workbook.rb:32:in `open'
  spreadsheet (1.0.3) lib/spreadsheet.rb:65:in `open'
  spreadsheet (1.0.3) lib/spreadsheet.rb:71:in `open'
  roo (1.13.2) lib/roo/excel.rb:40:in `block in initialize'
  roo (1.13.2) lib/roo/base.rb:565:in `block in make_tmpdir'
  /home/user/.rvm/rubies/ruby-2.0.0-p247/lib/ruby/2.0.0/tmpdir.rb:88:in `mktmpdir'
  roo (1.13.2) lib/roo/base.rb:564:in `make_tmpdir'
  roo (1.13.2) lib/roo/excel.rb:31:in `initialize'
  roo (1.13.2) lib/roo/spreadsheet.rb:17:in `new'
  roo (1.13.2) lib/roo/spreadsheet.rb:17:in `open'
  active_importer (0.2.5) lib/active_importer/base.rb:175:in `initialize'
  active_importer (0.2.5) lib/active_importer/base.rb:82:in `new'
  active_importer (0.2.5) lib/active_importer/base.rb:82:in `import'
.
.
.

But when I probed with xls files I don't have problem. I use:

• 'active_importer', '0.2.5'.
• 'rails', '~> 4.2.0'
• ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]

Do you know what is the origin of this error, in other projects I never received this with xlsx files. Plus I tried with xlsx files created on linux (Libre Office and WPS office) and windows (Office 2013), greetings!

I need to use Active_importer from a background service (Resque). The problem I got is that I need to let every new model created get the same account_id (account_id will vary from user to user of course). Is there a way to provide the individual account_id to be used for each import job?

Thanks!

Is there any way to import this kind of data?
So no headers present, but comma separated phone numbers for example.

07012345678, 071234567, 00467012345680,

Hi, how can we abort the import, for example, if a date validation fails?

Like that:

 column 'mes', :mes do |mes|
       if Date.valid_date? *mes.to_s.split(/[\-\.\/\|]/).reverse.map{|m|m.to_i}
      Date.parse(mes).strftime('01-%m-%Y')
    else
       abort! #how???
    end
 end

Hi,

I have seen there is one similar question in closed status, but mine is for a bit different scenario.
I have a model with common attributes and additional dynamic attributes. Because for each category of the model there can be different sets of dynamic attributes i am exporting to excel with sheets for each category...

Now, i would also like to import records in same fashion. The problem is category names and number is unknown, so one file could have 5 sheets while the other a different number, but they are named by category.

Is there a way to iterate through all the sheets and make additional dynamic attribute definitions for import programaticaly?

Thanks

Hey guys,

I have a suggestion about using helpers on the importer, for instance it is very common to add a helper to normalize the data received when reading and excel like this:

class ProductImporter < ActiveImporter::Base
  imports Product

  column 'Category', :category do |value|
    normalize(value)
  end

  private

  def normalize(value)
     value.strip.downcase
  end
end

but if we have a lot of column on Product we have to define a lot of similar blocks. It would be nice to have something like this:

class ProductImporter < ActiveImporter::Base
  imports Product

  column 'Category', :category, :normalize
  column 'Section', :section, :normalize

  ...  

  private

  def normalize(value)
     value.strip.downcase
  end
end

So in this case the normalize helper would be used to set each column of the product.
I know we could use also on :row_processing to call the normalize method inside the on row_processing block but I love the idea of having an alternative way ActiveImporter DSL.

what do you think? @gnapse ?

Best,

Luis Porras

During the import process I'm using the row_* callbacks to build a summary message for the user, something like:

156 row(s) processed.
154 row(s) successfully imported.
Error on row 25 - Validation failed: Email address has already been taken.
Error on row 89 - Validation failed: Email address has already been taken.

Is there any method implemented for passing this back to the caller of the importer class?

I tried result = UserImporter.import('/path/to/file.xls') to see what I got back but it was just a Range.

Hello Guys,

I'm wondering if is possible to populate the information without specifying one by one all the column names for example let's say I have a current CSV file with all the valid column names in place but some of the columns need special logic but for all the remaining columns names that appear in the CSV are exactly the same in the model but after to run the importer is only creating the models with the columns that I have specified in other words this is my data:

body; title; votes_count; image_url
some 1, title, 25,              , http://example.com/image.jpg
some 2, title, 25,              , http://example.com/image.jpg
some 3, title, 25,              , http://example.com/image.jpg

# my model


 Book(id: integer, title: string, votes_count: integer, body: text, image_file_name: string, image_content_type: string, image_file_size: integer, image_updated_at: datetime)

# My importer
class BookImporter < ActiveImporter::Base
  imports Book
  column 'image_url', :image do |image_url|
    URI.parse(image_url)
  end
end

After to run my importer the title, body, and votes_count are nil and only the image is populated correctly.

Any ideas about this behavior?, thanks in advanced guys, btw active importer it is a really good gem :)

First use gives this error: Using Roo::Google requires the google_drive gem

I can see that the roo gem was installed on bundle but it seems that this dependency was not installed.

My app user must be able to import data into three different models, say Customer, Prospect and Lead. At the moment I have a MODEL_importer.rb class for each type, containing mostly the same code. The main difference is that they have different imports calls at the top of the file.

I've tried passing in the model name to import to within the params hash but it doesn't work. Maybe the imports call is run before the hash is processed?

Is there any way for me to reduce duplication and have one data_importer.rb class and pass things like class name, columns, etc in the params hash?

Hi, is it possible to have different variations of a name for each column?
Something like:

    column ['unit', 'units', 'u'], :unit
    column ['note', 'notes', 'n'], :note, optional: true

Hey,

not really an issue, but more like a general question.
When something happens on a specific row that is being imported, i can catch row_index.
But I would also like to know what exactly went wrong.
I tried to do exception.message, but it always returns empty.
My next try was exception.inspect and the result is ActiveRecord::RecordInvalid - which is better, but still not very helpful.

Is there any way to get an exact message?
Sorry, i am not really exceptions master :)

I have a column on which you can find the object in the database, such as "handle"
I correctly write the code to see if this object already exists, then update its attributes, if it is not, create a new one?

According to https://github.com/continuum/active_importer/blob/develop/lib/active_importer/base.rb#L293
The model is always save! which will cause a DB transaction (touching updated_at even when no change)
But if the model is not changed, then it should not be saved.

I obtain the error RuntimeError - Duplicate importer column 'column' when I try to rerun an import with different data, after a successful run.

Are there ways to initialize the importer to enable a new run, directly after a successful run.

It would be good to have a switch for case-insensitive column headers.

Additionally, stripping spaces and other separators from the headers would be useful.

So referencing 'First Name' in the class would match FirstName, first name and first_name etc.

Hi,
i'm using active_importer gem but i noticed that if you import an excel with a cell containing a number the importer automatically add .0 at the end of the value. Why? i'm doing something wrong?

thanks

kind regards

I am transforming my importers from normal to transactional.
Before the change the scenario was that "on row_error" the error message was logged and the importer continues with the import. The ActiveRecord::RecordInvalid was not raised.

Now, with transactional import, the "on row_error" is skipped and "import_aborted" is called, but the exception is raised...

How can I deal with that? Thx!

Attempting to import a spreadsheet with incorrect headers is detected in the gem's load_header method but it does a raise error which stops the import process dead unless I wrap the import call in a rescue block with:

begin
  UserImporter.import(file.path, extension: :csv)
rescue => e
  puts e.message
end

This makes every call to import quite verbose and unnecessary when there is already an elegant :import_failed handler.

Maybe it would be better to fire the :import_failed event when load_header fails? This would allow me to handle the error in the same way as I handle other errors like row validations.

I'm using the skip_rows_if feature to look for empty rows, it's very handy but maybe rows that are completely empty could be skipped automatically?

Many people use blank rows to logically separate their data so maybe skip by default rather than have to explicitly check for them or deal with a list of validation failures.

Could this be done when the row is converted to a hash? Check for all-blank values and trigger skip_row? Sorry I don't have time to do a PR right now, I'm on a tight deadline! :-)

Hi y used your gem and I have this problem. When I make the installations steps and trying to import, I recived this;

LoadError in TurnsController#import

Using Roo::Google requires the google_drive gem
Rails.root: /home/ciro/pruebaCalendario

Application Trace | Framework Trace | Full Trace
app/controllers/turns_controller.rb:93:in `import'

Next if include gem 'google_drive' in the Gmefile, the method send send_notification returns this before import:

Fatal error while importing data: undefined method `spreadsheet_by_key' for nil:NilClass

I tried to solve this for a many days ago, but I can't find the problem. The form that I used is this:

<%= form_tag import_turns_path, :class => 'form-horizontal', :multipart => true do %>

And the controller:

def import
TurnImporter.import(params[:file].path)
redirect_to :action => "index"
end

I would apreciate it if somebody could help me. Sorry my english is very bad, for this I leave the same problem in spanish. Thanks

Hola estoy usando su gema y ocurre el siguiente problema. Cuando hago los pasos de instalación que ustedes indican y hago el import me sale el siguiente error:

LoadError in TurnsController#import

Using Roo::Google requires the google_drive gem
Rails.root: /home/ciro/pruebaCalendario

Application Trace | Framework Trace | Full Trace
app/controllers/turns_controller.rb:93:in `import'

Acto seguido, si incluyo en el Gemfile gem 'google_drive', el médoto send_notification me devuelve el siguiente error, antes de comenzar a importar:

Fatal error while importing data: undefined method `spreadsheet_by_key' for nil:NilClass

Llevo varios días tratando de solucionarlo y no logro hallar el error. El formulario que estoy utilizando es el siguiente:

<%= form_tag import_turns_path, :class => 'form-horizontal', :multipart => true do %>

y en el controlador

def import
TurnImporter.import(params[:file].path)
redirect_to :action => "index"
end

si alguien me pudiera dar alguna idea se los agradecería.

Hello,

First off, thank you for this cool gem!

Before I decide to use it I need to know if there is a way of using validation? If for example the Excel file does not contain a title (which is required) what happens then? Can I display errors?

Thanks!