corpglory / csv2influx Goto Github PK

It would be pretty cool if there were an option to replace NULL values with a predefined one such as 0 in the config file. One of the csv files I have to deal with is full of nulls instead of zeros and I have no control over the generator.

Guess it would be useful to be able to set number of CSV-rows to import to DB. Something like:
csv2influx --rows 100 --config airQuality.conf.json airQuality.csv

Save when

csv2influx init

Load config and say something

csv2influx file.csv

Write about merging columns support

Want to be able to load script from CLI

First of all, a huge "Thank you" for this splendid tool.
I do have an issue: I can import the examples without any issues, if my InfluxDB runs in http mode. As soon as I switch to https, I get (using a self-signed certificate):

csv2influx -c ./traffic_violations.conf.json ./traffic_violations.csv
Reading ./traffic_violations.conf.json
ok
checking format
ok
lines count:96
Connecting to http://www_go.jail.vlan:8086/traffic_violations
Schema: events
Importing
Fields: {"coordinates":","}
Tags: {"agency":"MCP"}
Time: 1380035460000000000
1 BAD_WRITE
{
  "Date Of Stop": "09/24/2013",
  "Time Of Stop": "17:11:00",
  "Agency": "MCP",
  "SubAgency": "3rd district, Silver Spring",
  "Description": "DRIVING VEHICLE ON HIGHWAY WITH SUSPENDED REGISTRATION",
  "Location": "8804 FLOWER AVE",
  "Latitude": "",
  "Longitude": "",
  "Accident": "No",
  "Belts": "No",
  "Personal Injury": "No",
  "Property Damage": "No",
  "Fatal": "No",
  "Commercial License": "No",
  "HAZMAT": "No",
  "Commercial Vehicle": "No",
  "Alcohol": "No",
  "Work Zone": "No",
  "State": "MD",
  "VehicleType": "02 - Automobile",
  "Year": "2008",
  "Make": "FORD",
  "Model": "4S",
  "Color": "BLACK",
  "Violation Type": "Citation",
  "Charge": "13-401(h)",
  "Article": "Transportation Article",
  "Contributed To Accident": "No",
  "Race": "BLACK",
  "Gender": "M",
  "Driver City": "TAKOMA PARK",
  "Driver State": "MD",
  "DL State": "MD",
  "Arrest Type": "A - Marked Patrol",
  "Geolocation": ""
}
Error: Parse Error
{
  "bytesParsed": 0,
  "code": "HPE_INVALID_CONSTANT"
}

Any thoughts? What am I missing here?
Thank you so much,
Chris

standard js can't handle nanoseconds.
so a new format needed

• Load config
• Try to connect to influx
• Show error if something wrong

Hi,

How can I declare more than one "TagSchema" in my config file?

Below how I declare the Tags in my config file:
"tagSchema": {
"update":{
"from": "update",
"type": ""
}
"team":{
"from": "team",
"type": ""
}
}
But I receive the message: "
"team":{
^
SyntaxError: Unexpected string in JSON at position 646"

Any thoughts? What am I missing here?
Thank you so much,
Diego

It would be nice to automatically generate config from csv file.
eg: csv2influx init data.csv would automatically add fields to config according to columns' headers.

• create /example folder with data
• add readme to example
• link from main readme

• Create test command in project
• Add simple tests

I want to add a new example where for this task: #11
But before that I would refactor example folder:

• Move /example to /examples
• Add docs to examples in examples/README.md
• Make "merge-column" example
• Docs to new example
• Fix main README

I think we should be able to run the tool in "quite" mode where I see only percentage of imported rows.

In quiet mode progress bar ends at (n-1)/n, where n is linesCount.
We can create progressBar with total: linesCount-1 or function countFileLines should return linesCount-1.

I want to type csv2influx -v to get version
because sometimes you just need to know it :)

@rozetko maybe you would find time to make it

It just fails on my machine

Not it's possible to map only fieldsShema, want to map tagShema as well:

"tagsShema": {
  "name": {
    "from": "First Name"
  }
}

Hi,
I got this error when trying to import the csv file

root@foxy-01:/home/csv2influx# csv2influx --c csv2influx.conf.json test.csv
Reading csv2influx.conf.json
ok
checking format
ok
lines count:8
Connecting to http://user:[email protected]:8086
Error: Connect URI is wrong, eg: http://user:pass@localhost:port,anotherhost:port,yetanother:port/mydatabase
at new Client (/usr/lib/node_modules/csv2influx/node_modules/influxdb-nodejs/lib/client.js:73:13)
at Promise (/usr/lib/node_modules/csv2influx/importer.js:94:22)
at new Promise ()
at Importer._import (/usr/lib/node_modules/csv2influx/importer.js:92:12)
at Importer.run (/usr/lib/node_modules/csv2influx/importer.js:88:17)
at importer.countFileLines.then.linesCount (/usr/lib/node_modules/csv2influx/index.js:55:13)
at
at process._tickCallback (internal/process/next_tick.js:189:7)

in csv file you might get something like

Date Of Stop, ....
09/24/2013 17:11:00, ....

and I have no clue in which timezone is it.

I would add

"timestamp": {
  "name": "Date Of Stop",
  "timezone": "UTC-5",
   ...
}

Now tool is not working if you run it like "csv2influx data.csv --config path/to/config.json". We should parse CLI arguments using some lib like https://www.npmjs.com/package/minimist

It's not got to stop progress in code.
It's better throw exception and then stop in index.js

I have two column Date and Time, Date looks like 20180101 and Time looks like 12:01:01,
how can I merge these two column into timestamps?

This csv would break importing

Date Of Stop,Time Of Stop,Agency,SubAgency,Description
08/28/2017,23:12:00,MCP,"3rd district, Silver Spring","PEDESTRIAN FAIL TO OBEY  ""UPRAISED HAND"" SIGNAL"

Imagine you have a file with fields
foo, boo, goo , but you want to map roo, boo, goo. There is no roo in the csv file.
We should show error.

Now it's possible to merge only date-related columns like this

"time": ["Date", "Time"]

It would be cool to be able merge all columns

"name": ["First Name", "Last Name"]

There is a

"mapping": {
    "timestamp": "Date",
    .....

in csv2influx.conf.json . I wand to be allowed to import data without timestamp param

We need to add authorization support

If we parce csv file and find ony one column, we should show message what maybe delimiter is wrong

It would be nice to have a file we write errors and warning during importing

Not sure how to create a pull request for this; but this works for me to parse a "MM/DD/YYYY" date.

const moment = require('moment');

function parseValue(recordValue, mappingObject) {
  if(mappingObject.format === 'jsDate') {
    // convert millisconds to nanoseconds
    return (new Date(recordValue).getTime()) * 1000 * 1000;
  }
  if(mappingObject.format != '') {
	return (moment(recordValue,mappingObject.format)) * 1000 * 1000;
  }
  return recordValue;
}

$ nodejs --use_strict index.js init
$ cat csv2influx.conf.json 
{
  "shemaName": "SHEMA_NAME",
  "influxdbUrl": "http://127.0.0.1:8086/INFLUXDB_URL",
  "mapping": {
    "timestamp": "Date",
    "fieldShema": {
      "lat": "float",
      "lng": "float",
      "name": "string",
      "description": "string",
      "location": "string"
    }
  },
  "csv": {
    "delimiter": ","
  }
$ nodejs --use_strict index.js example/airQuality.csv 
Reading csv2influx.conf.json
ok
checking format
Config format error: mapping.fieldShema should contain 'Date' field

Fixed by adding field Date:

$ cat csv2influx.conf.json 
{
  "shemaName": "SHEMA_NAME",
  "influxdbUrl": "http://127.0.0.1:8086/INFLUXDB_URL",
  "mapping": {
    "timestamp": "Date",
    "fieldShema": {
      "Date": "Date",
      "lat": "float",
      "lng": "float",
      "name": "string",
      "description": "string",
      "location": "string"
    }
  },
  "csv": {
    "delimiter": ","
  }
}

Now database from config has to exist.
influxdbUrl: 'http://127.0.0.1:8086/INFLUXDB_URL'
We can create database automatically if it doesn't exist.

It's possible to make mapping like this:

{
  "fieldNameA": { "name": "fieldName", ... },
  "fieldNameB": { "name": "fieldName", ... },
}

So both fields fieldNameA and fieldNameB map to same fieldName.
It should be forbidden.

I have a csv dataset like this:

Date Of Stop,Time Of Stop, ...
08/28/2017,23:41:00, ...

But I need to "merge" two columns "08/28/2017,23:41:00" to "08/28/2017 23:41:00" so be able to parse it.

I don't know how to make it better. We can something like this "timestamp": ["Date Of Stop", "Time Of Stop"] and concatenate two string columns to one, but it if we were able to convert one csv file to another with mapper, then it would be a solution too. Don't want to make this tool too complex.

Hi,

This is a great tool. But I encountered with a problem, after csv data imported into the series, I noticed that 'show tag keys' will return empty thereafter.

I tried one more time using influxdb-nodejs directly, also no tag created at all.