agardiner / csv-diff-report Goto Github PK

Hello, first off great tool, works a lot faster then the previous tool I used.
But I have a question, when comparing two CSV's and exporting it to a HTML file it takes about 1 minute to complete a compare of around 40000 records. However when exporting the same compare to XLSX it takes quite a lot longer when there are a lot off differences (maybe 20 minutes).
Where does this difference come from and is it possible to make the xlsx export faster?

When I try this tool with UTF8 encoded file I get error
"invalid column name as key"
It seems to work fine when the file is not UTF8 encoded.

Hello.
I'm trying to generate csv diff report as html.
But, when csv file path is absolute, it does not work.
it just works by specifying the csv file as a relative path.

$ csvdiff test.csv test_old.csv
/Library/Ruby/Gems/2.0.0/gems/arg-parser-0.3.0/lib/arg-parser/dsl.rb:70: warning: instance variable @arg_opts not initialized
Performing file diff:
  From File:    test.csv
  To File:      test_old.csv
Opening FROM file 'test.csv'...  3 lines read
Opening TO file 'test_old.csv'...  3 lines read
Found 1 differences: 1 Updates
Diff report saved to './Diff_test_to_test_old.html'

$ csvdiff /Users/maeken/csv/test.csv /Users/maeken/csv/test_old.csv
/Library/Ruby/Gems/2.0.0/gems/arg-parser-0.3.0/lib/arg-parser/dsl.rb:70: warning: instance variable @arg_opts not initialized

USAGE: ruby /usr/local/bin/csvdiff FROM TO [PATTERN] [OPTIONS]

Specify the /? or --help option for more detailed help

is there a way to get ALL fields of the row to show in the Excel file for an update? I really can't figure out the pattern to what fields show up by some of my key fields don't show and it makes it nearly impossible to find the row in the original source (this is because some of the files have dupes and the diff ignores dupe rows so the row field in the output isn't accurate.. at least that's what I'm thinking is causing the row number to be wrong).. at any rate.. having the fields there in the diff would be a huge win for me. I spend a ton of time tracking down the data in the source files..

thanks
Gary

It can be useful to show only the fields that have been changed when comparing two CSVs.
Currently the rows are filtered this way, but the columns are not.
Apologies in advance if the tool already has options to support this but I am not seeing it.

When using keyfields of parent-child fields there is a difference in the time it takes to diff two csv's.
With a small number of records this is not a big issue, however, with 20.000+ record using the keyfields function takes a lot more time.
Is there a reason the keyfield function takes more time, and is there a way to speed up this process?

This exception may occur under, for example, Japanese Windows environment.

In the case, the encoding of the string got via ARGV may be not UTF-8 but Windows 31J (a.k.a. Codepage 932).

# demo1.rb

p ARGV.first.encoding

> ruby demo1.rb foo
#<Encoding:Windows-31J>

Generally, two strings which do not contain any non-ASCII character can be joined, even if one is in UTF-8 and the other is in Windows 31J.

p ["foo".encode("UTF-8"), "bar".encode("CP932")].join
# Not raised.

However, if these strings contain a non-ASCII character, Encoding::CompatibilityError occurs.

p ["あ".encode("UTF-8"), "う".encode("CP932")].join
# => `join': incompatible character encodings: UTF-8 and Windows-31J (En
coding::CompatibilityError)

So, if one of the file paths and a cell value of the CSV have any non-ASCII character, the same exception may occur here.

I think that converting the file paths into UTF-8 may solve the problem.

When comparing two UTF-8 CSVs, I use csvdiff one.csv two.csv --encoding utf-8. The outputted HTML does not render German Umlaute properly: e.g. instead of ü you get ü.

Example CSV:

one.csv:

Aufgabe
Änderung übernehmen
Übergang planen

two.csv:

Aufgabe
Camp-Farbkombi überlegen
Übergang planen

I'm running into this error when using csvdiff, and wondering if you could help resolve it?

csvdiff a.csv b.csv
Performing file diff:
From File: a.csv
To File: b.csv
Opening FROM file 'a.csv'...
An error occurred opening file a.csv: wrong number of arguments (given 3, expected 1..2)
/usr/local/Cellar/ruby/3.0.0/lib/ruby/3.0.0/csv.rb:1412:in open': wrong number of arguments (given 3, expected 1..2) (ArgumentError) from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-0.6.1/lib/csv-diff/csv_source.rb:60:in initialize'
from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/lib/csv-diff-report/report.rb:262:in new' from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/lib/csv-diff-report/report.rb:262:in open_source'
from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/lib/csv-diff-report/report.rb:219:in diff_file' from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/lib/csv-diff-report/report.rb:87:in diff'
from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/lib/csv-diff-report/cli.rb:131:in process' from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/lib/csv-diff-report/cli.rb:101:in run'
from /usr/local/lib/ruby/gems/3.0.0/gems/csv-diff-report-0.3.6/bin/csvdiff:7:in <top (required)>' from /usr/local/bin/csvdiff:23:in load'
from /usr/local/bin/csvdiff:23:in `

'

In the README, under the 'Usage' section, it says in order to run the help menu to enter the command 'csv-diff --help'...this produces the error '-bash: csv-diff: command not found' whereas removing the hyphen and running the command 'csvdiff --help' produces the desired result of generating the help menu.

Hello,
I'm trying to save a diff as a text file, but I'm getting a lot of errors.
I've copied my input in het terminal and the corresponding result:

ruby /users/broex/.rvm/gems/ruby-2.2.1/bin/csvdiff prod.csv new.csv --parent-fields data_uri --child-fields data_uri --format text
Performing file diff:
From File: prod.csv
To File: new.csv
Opening FROM file 'prod.csv'... 43907 lines read
Opening TO file 'new.csv'... 43907 lines read
Found 43907 differences: 43907 Updates
/Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/text.rb:31:in text_diff': undefined local variable or methodcsv' for #CSVDiff::Report:0x007f8e71a1e350 (NameError)
from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/text.rb:17:in block (2 levels) in text_output' from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/text.rb:16:ineach'
from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/text.rb:16:in block in text_output' from /users/broex/.rvm/rubies/ruby-2.2.1/lib/ruby/2.2.0/csv.rb:1273:inopen'
from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/text.rb:15:in text_output' from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/report.rb:121:inoutput'
from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/cli.rb:116:in process' from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/lib/csv-diff-report/cli.rb:78:inrun'
from /Users/broex/.rvm/gems/ruby-2.2.1/gems/csv-diff-report-0.3.1/bin/csvdiff:7:in <top (required)>' from /users/broex/.rvm/gems/ruby-2.2.1/bin/csvdiff:23:inload'
from /users/broex/.rvm/gems/ruby-2.2.1/bin/csvdiff:23:in `

When settings --include-matched True, the not changed columns are not added to the HTML file

If I add new = d below this line:

The output is as expected when adding this line:

Without the line: