https://github.com/phusion/traveling-ruby

Would mean CSVlint could be installed via (for example) a package manager, and would run like any other *nix application.

Hi,

Similar to the example below, i would love to be able to know how many rows my csv is without reparsing all the files via csv ruby library ...

# get some information about the CSV file that was validated
validator.encoding
validator.content_type
validator.extension
**validator.rows**

Regards,

https://github.com/densitydesign/raw/

Presumably these are byte positions? It would be useful to have a row/col if possible.

I've made a fix for this bug. Please review & merge. Thanks!
#130

Currently both schema_validation.feature && schema_spec.rb do not test for the full range of errors and warnings that are included in this gems readme
There are some well specified features in csvlint.rb - these will be incorporated into specs for the gem

Currently we're using the default options of the read_line and CSV parser, probably best to be explicit about the defaults.

Internally we should probably use CSV DDF, ready to accept external schemas.

Need to:

• specify a delimiter to read_line (this is what CSV DDF calls lineterminator
• add an opts hash to CSV.parse_line.

See https://github.com/theodi/datapackage.rb/blob/master/lib/datapackage/validator.rb#L203 for example of mapping CSV DDF to Ruby CSV options.

The schema col count should be used if a schema is available, and the current simple ragged row check should not be performed.

This generates an inconsistent values warning:

https://raw.github.com/datasets/cpi-gb/master/data/cpi-uk-annual.csv

Its because some numbers include decimal points and some dont. The former get matched as alphanumeric (because of the full stop) the latter as numeric.

Several date and time formats were blindly imported from Rails' Date/Time formats, but I'd expect several of those are never in a CSV (e.g. the 23 digit timestamp with nanoseconds). Eliminating the unused date formats will speed up #103 even further.

It isn't currently possible to pipe data into csvlint on the command line. As a result it isn't possible to use it to compose workflow where one process in the chain is generating csv. For example - proc_that_generates_csv | csvlint

It should probably default to reading from stdin if no argument are supplied, and have a separate 'usage' trigger (--help, -h)

Does csvlint support enumerations? I did not see an explicit enum type in the documentation. This could probably be simulated with a regex constraint but it would be nice to have native support for enumerable fields.

Looking at this one:

http://csvlint.io/validation/5331acb86373767901030000

I think that the "inconsistent values" that are being highlighted as warnings are because some of the values are blank. Blank values shouldn't count as inconsistencies (ie only look at the non-blank values when working out what type of value the column contains).

Currently schema validation for a line is by-passed if we get a :line_ending error triggered by a CSV::MalformedCSVError.

Ideally if we know is a line ending issue, then we should log the error and then attempt to re-process it with an alternative line ending, so we then get schema validation too.

In XML, it's a lot more usual to use xsd:integer than xsd:int, but we're using xsd:int rather than xsd:integer. Is this deliberate and if so why?

I am supplying a pull request for a feature to fix this bug.

Currently validating local csv files that aren't encoded as utf-8 always fails with encoding error.
Validator currently sets the encoding of a file by using io.charset which doesn't exist for files that's read in.

Suggested implementation:
Use options={} parameter to take in a file encoding and merge it into @csv_options

@pezholio I'd like to implement this feature as I think it'll be helpful. I also will need this feature for my own use. Let me know if this makes sense in turns of the overall design of the gem.

Running CSVLint on large remote CSVs is slow, because it needs to download the file, then validate. It would be possible in Ruby to stream the response body, in which case it can be validated as it downloads, saving time and memory. Also, if :limit_lines is set, CSVLint can stop the download once those lines have been read.

JSON Table Schema defines the following optional fields:

• type
• title
• description
• format

If Csvlint::Schema could expose those, I can show them in the UI.

The fetch_error method can trigger the above if it can't map the CSV error message to a known error type. The code should be defaulting to :unknown but its not doing that, we just get an error.

I've not tracked down the specific errors that aren't being handled, probably need to check over the ruby source.

Extend #5 to allow user to provide defaults. E.g. by providing a Ruby hash that contains a dialect key that conforms to CSV DDF.

(This style would let datapackage.rb delegate CSV validation to csvlint.rb)

That's the behavior. It seems like unusual behavior to me. Shouldn't it check if any row has more fields than the first row?

As per #92, we've published this on RubyGems now, but still need to get this continuously deploying through Travis.

E.g:

http://csvlint.io/validate?url=http%3A%2F%2Fdata.defra.gov.uk%2Fenv%2Faqfg14-13but-201105.csv

Error is reported multiple times, but there's no indication of the column or row that triggered the warning.

specifies how to interpret whitespace which immediately follows a delimiter; if False, it means that whitespace immediately after a delimiter should be treated as part of the following field.

By this definition, the following CSV:

foo,bar, baz,   bzz

should parse as follows if skipInitialSpace is true:

["foo", "bar", "baz", "bzz"]

What csvlint's code does is set the delimiter to ,  (the original delimiter, in this case ,, followed by a space). That parses the CSV as:

["foo,bar", "baz", "  bzz"]

Which is clearly incorrect.

Ideally we want to detect what encoding a file is actually using, as this might differ from what is advertised.

See features in theodi/shared#120

For example, take the NY honours spreadsheet:

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/246500/New_Year_Honours_2013_full_list.csv/preview

This is delivered with a Content-Type of text/csv, with no mime type. US-ASCII might be assumed to be the default, although I think UTF-8 is increasingly common and might be reasonable default assumption

However when trying to open the file, it apparently has invalid characters.

E.g.:

$ curl -v https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/246500/New_Year_Honours_2013_full_list.csv >/tmp/test.csv
$ file -bi /tmp/test.csv
text/plain; charset=unknown-8bit

Using charlock_holmes we get a bit more information:

$ irb
> require "charlock_holmes"
> contents = File.read("/tmp/test.csv")
> detection = CharlockHolmes::EncodingDetector.detect(contents)
 => {:type=>:text, :encoding=>"ISO-8859-1", :confidence=>61, :language=>"en"} 

Confidence level is relatively low, but guessed encoding seems reasonable.

If my schema has, for example:

...
{
  "name": "Date",
  "type": "date",
  "description": "",
  "constraints": {
    "format": "yyyy-mm-dd"
    "required": true
  }
}
...

And I validate against a CSV that has an incorrect date pattern e.g. "23/09/1980", then I don't get any errors.

Also, http://dataprotocols.org/json-table-schema/#field-types says that every field that has type date should match ISO8601, unless format is specified.

What's the syntax of the pattern constraint? Is this using Ruby regexps or something more platform neutral such as those specified in XML Schema?

If a numeric column has numbers with 8 digits (date_number), 14 digits (dateTime_number) or 23 digits (dateTime_nsec) and if those digits happen to be valid dates >10% of the time, then an inconsistent_values validation error is incorrectly reported.

Unless these date/time formats are extant (do we know any software that reads or writes CSV dates in this format?), they should be eliminated.

Note that this happily helps solve #105, and it is in fact a significant speed improvement after #103 (like 25% faster with #103).

From @JeniT:

Specs are at:

http://w3c.github.io/csvw/syntax/
http://w3c.github.io/csvw/metadata/

The examples at:

http://w3c.github.io/csvw/csv2json/#examples

are particularly useful.

There are tests are described at:

http://www.w3.org/2013/csvw/tests/

and are in Github at:

https://github.com/w3c/csvw/tree/gh-pages/tests

There’s a manifest in JSON at:

https://github.com/w3c/csvw/blob/gh-pages/tests/manifest.jsonld

which points to the manifest for the validation tests (which are the only ones I think we have to care about) which is at:

https://github.com/w3c/csvw/blob/gh-pages/tests/manifest-validation.jsonld

Creating something that maps that test manifest into whatever format tests are needed for csvlint would be good.

I suspect that you will want to distinguish between validation based on datapackage and validation based on CSV on the Web metadata: there are many similarities but they’re not identical.

UTF-8 files with a Byte Order Mark have the BOM passed through to the content by default in Ruby, and the result is whitespace errors reported by csvlint.

Here's an example: http://csvlint.io/validation/543526f36373760fc6020000.

The BOM only needs to be filtered from the first line in the file, e.g.:

row.delete!("\xEF\xBB\xBF")

Related to Data-Liberation-Front/csvlint.io#107

Should we support zipped CSV files? Would make it possible to validate, e.g. the data at http://smtm.labs.theodi.org/download/

Two possible things to consider:

• Zip files containing one or more CSV files, we could just take the first
• Files served with a gzip content-encoding. Not sure if the current code will handle that or not.

This http://dataprotocols.org/json-table-schema/ suggests the type descriptor is at the root level of a field descriptor. This project's README file says it should be inside constraints and it uses XMLSchema URLs. Please could you give some info on why the difference?

Regardless of whether the input is Valid or Invalid, csvlint returns 0. This makes it hard to use it in a script to determine whether the input was valid or not.

> csvlint a_bad.csv; echo $?
> 0

> csvlint unicorn.csv; echo $?
> 0

usage, triggered by csvlint does return 1 as an exit code, but I think it is normal for usage/help to return 0.

> ls --help; echo $?
...
GNU coreutils home page: <http://www.gnu.org/software/coreutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
For complete documentation, run: info coreutils 'ls invocation'
0

> ls -?; echo $?
ls: invalid option -- '?'
Try `ls --help' for more information.
2

So then we can tell the user what actually failed, e.g. pattern, required, etc.

Example files:

quoting issues, new-lines in quoted fields
https://raw.github.com/datasets/shell-oil-spills-niger-delta/master/data/data.csv

new-lines in quoted fields
https://raw.github.com/datasets/cofog/master/data/cofog.csv

These generate errors because there are newlines in some long comment fields. This should be valid. Its down to us manually parsing line by line

Could Schema.load_from_json_table throw an error or print a warning if a schema file cannot be parsed correctly? It took a while to realise that the reason my CSV file wasnt showing the errors I expected from schema validation was the schema wasnt loading.

The Schema.from_json_table should be tolerant of missing fields key. Currently it'll blow up if there aren't any.

https://raw.github.com/datasets/cpi/master/data/cpi.csv

Generates two warnings:

• Incorrect content type
• Incorrect file extension

It definely has the wrong content type, but the extension is correct.

The README suggests to construct a Validator and then call validator.validate. This is not necessary as validate is called in initialize. This results in double the number of recorded issues.

Suggest either fix the docs or remove call to validate in initialize?

A few suggestions for detecting errors in headers are in jekyll/jekyll#2761, from @paulfitz. Might be nice to integrate.

I think it'd definitely be reasonable to treat the following cases as errors:

Blank cells in the alleged header.
Repeated cells in the alleged header.
Numeric-looking cells (integer, float) in the alleged header (this one is a bit less reasonable than the first two, but would catch a lot more headerless CSV files).

Anything that tries to be much smarter than that, it'd be great to have a configuration switch to turn off for when predictability is important.

Sometimes 0-based, sometimes 1-based. Failing test in the inconsistent-column-base branch. Not sure where this is happening...

Does the library support skipping the header so that no warnings or errors are outputted?

Given you already have a gemspec, is there any reason you haven't published it as a gem on rubygems.org? Thanks.

I guess it should disregard these.

Add Command Line Support

Similar as https://github.com/Clever/csvlint

We should ensure that header rows are properly handled:

The presence of a header might be indicated in HTTP response, or in the CSV dialect which can now include a header key to indicate if there is a header (defaulting to true)

The draft CSV syntax states:

The first line of a CSV+ file must contain a comma-separated list of names of columns. This is known as the header line and forms the columns of the table. Column names must be unique and must not be blank

We should ensure header rows are properly handled by:

• checking content type for indication of whether header is present
• checking dialect for presence of header
• adding an error if either of the above indicates that there is no header row
• calling Schema.validate_header to check column names on the header
• extend the above to check whether column names are unique and not blank