barry-jones / csv-validator Goto Github PK

Fast CSV file validation.

License: MIT License

C# 61.52% PowerShell 38.48%

csv-text csv-validator netcore dotnet validation csv text validations validator delimited

csv-validator's Introduction

CSV Validator

.NET Core CSV text file validator. Enables the quick verification of column separated data files. Columns can be checked against multiple requirements for correctness.

This application is provided via a CLI and a NuGet package. Details for using both are provided below.

CLI Usage

The application is command line based, and has two arguments:

validate --file "input-datafile.csv" --with "configuration.json"

Configuration

To configure the verification a JSON file is used with the following format:

{
	"rowSeperator": "\r\n",
	"columnSeperator": ",",
	"hasHeaderRow": true,
	"columns": {
		"1": {
			"name": "ID",
			"isRequired": true,
			"unique": true
		},
		"2": {
			"name": "DOB",
			"pattern": "^\\d\\d\\d\\d-\\d\\d-\\d\\d$"
		},
		"3": {
			"name": "NOTES",
			"maxLength": "250"
		}
	}
}

The pattern property uses regular expressions but it is important to escape the characters else the application will fail when reading the configuration file.

rowSeperator can be any number of characters, rows can also be separated by characters and do not need the new line characters to be available in the input file.

columnSeperator can be one or more characters.

The columns require the number, which is the ordinal of the column in the input file, you do not need to specify all columns, only those that are to be validated.

Supported validation

{
    // validates the column has content
    "isRequired": true|false,
    // validates the content is unique in this column across the full file
    "unique": true|false,
    // validates a string against a regular expression
    "pattern": "regular expression string",
    // Maximum allowable length for a column
    "maxLength": "int",
    // Check if content is numerical
    "isNumeric": true|false
}

API Usage

CSV Validator is also available as a NuGet package, to enable in application validation of text files. The API conforms to netstandard 2.0.

Installation

dotnet add package csvvalidator

<ItemGroup>
  <PackageReference Include="csvvalidator" Version="1.0.1" />
</ItemGroup>

Usage

Validator validator = Validator.FromJson(config);
RowValidationError[] errors = validator.Validate(inputStream);

Dealing with validation errors.

Errors are reported heirarchicly, by row and then columns.

foreach(RowValidationError current in errors) 
{
	// row errors provides details of the row number and the content
	Console.WriteLine($"Errors in row[{current.Row}]: {current.Content}");
	foreach(ValidationError error in current.Errors)
	{
		// all errors then that occur on that row are reported in the error collection
		Console.WriteLine($"{error.Message} at {error.AtCharacter}");
	}
}

csv-validator's People

Contributors

Stargazers

Watchers

Forkers

fmms douskaki wtc1968 makeawishamerica rainyclouds alexander-sml darayman aleskey gianlucatofi limking24 gibsonndemanga sw2github

csv-validator's Issues

Improve validator instantiation

Update validator configuration to make constructing validator configurations easier.

Examples:

ValidatorConfiguration config = new ValidatorConfiguration();
config.Set(1, isNumeric: true);
config.Set(2, maxLength: 20, isRequired: true);
Validator validator = Validator.FromConfiguration(config);

provide a summary of the errors

After the validation is complete currently the application only provides information about the overall failure or success of the file. When a failure occurs it should display a summary of the errors.

Feature Request - Header validation

If a header is expected:

Validate the number of columns expected (eg: if we specify 10 columns in the definition check that we have 10 headers)
Validate the header name - if the 'name' property is defined in the schema

For heavy input files, StreamSourceReader is not reading whole files

does not support header rows

Header rows are not validated and when a header row is checked against the row requirements it can produce invalid failures

Getting NuGet Error NU1108 when trying to install package.

Hello there. Getting the following error when trying to install this package to a new console project.

https://learn.microsoft.com/en-us/nuget/reference/errors-and-warnings/NU1108

Apparently, it is reporting some sort of circular dependency. Let me know if you need any more information.

Validator error message - add column name

It would be helpful to the user to understand which column is invalid.

Handle quoted columns

When a file uses a comma to separate fields, commas in quoted fields are not escaped so they get processed as a new column.

Might be worth looking at CsvHelper as they seem to handle this

Console message for success is red and should be green

NuGet package

Hi, this looks awesome.

Any chance of deploying as a NuGet package to use as a library within another application.

Quoted column regex weirdness

Looks like the Regex isn't a bit too capture happy :(

Example with the following input:
somthing,before,\"just, a string\",seperated,after

It actually returns: