ess-dive-community / essdive-csv-structure Goto Github PK

• Documentation reference (URL): https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_quick_guide.md
• Submitter: Rob Crystal-Ornelas
• Justification (why is this change necessary?): Changing language from standard to reporting format

I suggest the following changes:
Make language consistent with reporting formats

• Documentation reference (URL): https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_detailed_guide.md#field-names
• Submitter: Kim Ely
• Justification (why is this change necessary?): could cause coding/data assimilation issues

I suggest the following changes: It's not the end of the world, but apparently R doesn't particularly like column names that start with a number (i.e. 0-9). Perhaps recommend to not use numbers as the first character?

Submitter: @robcrystalornelas and @JEDamerow

I suggest the following changes:
The "N/A" requirement that the CSV format recommends for missing values requires additional step in R.

R specifically uses "NA" for missing values rather than N/A

Submitter: Rob Crystal-Ornelas

I suggest the following changes: Currently, the reporting format says that variable names should only include: letters, numbers, hyphen, underscores.

With these restrictions, would have to reformat variable named:

Per recommendation of @vchendrix, keeping variable names restricted to letters, numbers, hyphen and underscore helps ensure that files are portable to as many languages as possible.

Options to consider for removing slash from variable name

• chloride_umol_per_l (this follows recommendations from: https://units-of-measurement.org/umol)
• chloride_micromol_per_l (the mu symbol is not ascii character so should be swapped out)
• chloride_micromol-per-l (in this case the underscore separates the variable being measured and the unit)
• Other option is to write a script to separate the unit from the variable being measured so that we fully comply with CSV guidelines.

With all the options listed above, we need to consider how much it may disrupt researchers already ingesting and using data that have variables in their original format.

• Documentation reference (URL): https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_quick_guide.md#column-or-row-name-orientation
• Submitter: Dylan O'Ryan
• Justification (why is this change necessary?): The CSV Quick Guide incorrectly describes vertical and horizontal orientation.

I suggest the following changes to the Reporting Format Description for Column or Row Name Orientation:

Tabular data can be organized: 1) Horizontally meaning that data are organized in rowscolumns and there's a new name describing the data at the start of every rowtop of every column 2) Vertically meaning that data are organized in columnsrows and there is a new columnrow name describing the data at the top of every columnstart of every row.

• Documentation reference (URL): https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_quick_guide.md; https://github.com/ess-dive-community/essdive-hydrologic-monitoring/blob/main/template_DataFile.csv; https://github.com/ess-dive-community/essdive-soil-respiration/blob/main/data_reporting_format_guide.md; https://github.com/ess-dive-community/essdive-leaf-gas-exchange/blob/master/docs/definedVariables.md; https://github.com/ess-dive-community/essdive-sample-id-metadata/blob/master/guide.md#collection-date
• Submitter: Kristin Boye
• Justification (why is this change necessary?): there is no consistency in the format for how date and time is reported between the various "reporting formats"

I suggest the following changes: Could we agree on an "ESS-DIVE central" format for how to report date and time? Looking through the currently uploaded instructions (CSV, Sample ID, Soil Respiration, Hydrologic Monitoring, Leaf Gas exchange) there are as many ways of reporting date and time as there are reporting formats (although all recommend UTC and request UTC offset if using local time). I personally like (and think it would make things more script-friendly as we are trying to limit the use of special characters????) to keep delimiters out of it (as proposed in the soil respiration reporting format) and simply use up to 12 digits (YYYYMMDDHHMMSS) to the level of appropriate resolution for the data, but as long as we agree on what we should require/recommend I think that would help a lot...

• Documentation reference (URL): https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_quick_guide.md#delimiter
• Submitter: Rob C-O
• Justification (why is this change necessary?):

Is there a reason why we shouldn't use semicolon in (for example) a list of terms within a CSV cell? I reviewed notes on this reporting format and did not find a reason why semicolons shouldn't be used to separate words in a list.

I suggest the following changes:
Right now, the CSV reporting format suggests: "For commas not meant to be a delimiter (e.g. used within a cell), use a vertical bar"

could be changed to: "For commas not meant to be a delimiter (e.g. used within a cell), use a vertical bar or semicolon"

@charuleka and @vchendrix Is there a use case we can think of where semicolon's should not be used as part of a list of terms?

• Documentation reference (URL): https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_detailed_guide.md
• Submitter: Ben Bond-Lamberty
• Justification (why is this change necessary?): Specification is ambiguous on two important points.

Hi all, in my experience with CSV files, there are two issues that come up that can affect portability/reading that this reporting format might want to address:

1. Last line in file. Should files end with the last line of data or have a single empty line at the end? I'd suggest the latter, which is the POSIX standard and generally recommended. Note however that Excel saves CSVs without such an empty line. 😞
2. Line separator. Windows for example separates them by CR+LF, whereas Unix-y systems use LF. Not sure if the format wants to get this far into the weeds, but if yes, I'd recommend the latter.

Anyway for your consideration. The first is probably more important than the second.

Submitter:

I suggest the following changes:
The requirement in https://github.com/ess-dive-community/essdive-csv-structure/blob/master/csv_detailed_guide.md#temporal-data
to "Do not report time using Daylight Savings Time." is actually problematic for Alaska work. This is standard "local time" and we have been using this for many years. To edit could cause many errors. Can this be OK as long as the UTC offset is properly reported?