$ cmake .
Configuring xseed-utils version: 1.0.0
-- Found MSEED: /usr/local/lib/libmseed.dylib (found suitable version "3.0.0", minimum required is "3.0") 
-- 
...

but:

$ ls -l /usr/local/lib/libmseed.dylib
lrwxr-xr-x  1 chad  staff  21 Mar 24 19:26 /usr/local/lib/libmseed.dylib -> libmseed.2.19.5.dylib

But, when a build is actually triggered that installed library is not used so the build actually works. Something is wrong with that detection mechanism and it's unclear that if the right version where present on the system that it would be used.

The programs do not have any user-visible version, which would be very useful to know whether one has the current version, etc.

Each of these programs should have a command line option of -V to print the program name and version.

Version should be in #.#.# form starting with 1.0.0 and generally follow the scheme defined here: https://semver.org/

The -W command line option for xseed-validator is documented in the usage message as:

	 -W         Option flag  *e.g* -W error,skip-payload

It remains undocumented what other "flags" are supported.

The source file src/xseed-validator/parse_warn_options.c, which appears to deal with these flags, has more options. Based on the amount of commented-out and incomplete code elsewhere, there is little confidence that these flags are honored and review is needed in addition to documentation.

mseed2text flips the order of the bits in flags I think.

For example using reference data file reference-sinusoid-steim2.xseed mseed2json gives:

{
    "SID": "XFDSN:XX_TEST__L_H_Z",
    "RecordLength": 956,
    "FormatVersion": 3,
    "Flags": {
        "RawUInt8": 4,
        "ClockLocked": true
    },
    "StartTime": "2012-01-01T00:00:00.000000000Z",
    "EncodingFormat": 11,
    "SampleRate": 1,
    "SampleCount": 500,
    "CRC": "0x5CFF0548",
    "PublicationVersion": 1,
    "ExtraLength": 0,
    "DataLength": 896
}%     

But mseed2text gives:

XFDSN:XX_TEST__L_H_Z, version 1, 956 bytes (format: 3)
             start time: 2012,001,00:00:00.000000
      number of samples: 500
       sample rate (Hz): 1
                  flags: [00100000] 8 bits
                         [Bit 2] Clock locked
                    CRC: 0x5CFF0548
    extra header length: 0 bytes
    data payload length: 896 bytes
       payload encoding: STEIM-2 integer compression (val: 11)

but 4 in binary should be 00000100 not 00100000. IE bit 2 is 3rd from right, not 3rd from left.

 git clone [email protected]:EarthScope/mseed3-utils.git
cd mseed3-utils
mkdir build
cd build
cmake ..
make

copy reference-sinusoid-int32.mseed3 from miniseed repo

bin/mseed3-validator reference-sinusoid-int32.mseed3
zsh: segmentation fault  bin/mseed3-validator reference-sinusoid-int32.mseed3

This is on a M3 Max macbook, so new chip may be part of the problem.

Oddly, mseed3-text works just fine:

bin/mseed3-text reference-sinusoid-int32.mseed3
FDSN:XX_TEST__V_H_Z, version 1, 2059 bytes (format: 3)
             start time: 2022-06-05T20:32:38.123456789Z (156)
      number of samples: 500
       sample rate (Hz): 0.1
                  flags: [00000100] 8 bits
                         [Bit 2] Clock locked
                    CRC: 0x37223EA2
    extra header length: 0 bytes
    data payload length: 2000 bytes
       payload encoding: 32-bit integer (val: 3)

Both diagnostic output, i.e. output controlled with verbose (-v) flags, and the converted data content are printed to stdout. They are hopelessly mixed this way.

The diagnostic output should go to stderr and leave stdout solely for data output.

A mseed3 file with multiple records should still generate valid json.

Example using reference-data:

cat reference-sinusoid-int32.xseed reference-sinusoid-int16.xseed > combine.xseed
mseed2json combine.xseed                                      
{
    "SID": "XFDSN:XX_TEST__L_H_Z",
    "RecordLength": 2060,
    "FormatVersion": 3,
    "Flags": {
        "RawUInt8": 4,
        "ClockLocked": true
    },
    "StartTime": "2012-01-01T00:00:00.000000000Z",
    "EncodingFormat": 3,
    "SampleRate": 1,
    "SampleCount": 500,
    "CRC": "0x727BF4BD",
    "PublicationVersion": 1,
    "ExtraLength": 0,
    "DataLength": 2000
},{
    "SID": "XFDSN:XX_TEST__L_H_Z",
    "RecordLength": 860,
    "FormatVersion": 3,
    "Flags": {
        "RawUInt8": 4,
        "ClockLocked": true
    },
    "StartTime": "2012-01-01T00:00:00.000000000Z",
    "EncodingFormat": 1,
    "SampleRate": 1,
    "SampleCount": 400,
    "CRC": "0x106EAFA5",
    "PublicationVersion": 1,
    "ExtraLength": 0,
    "DataLength": 800
}%            

It looks like this was meant to be a json array at the top level, but the open and close brackets [ ] are not there.

The main() for each xseed-validator, xseed2json, and xseed2text are contained in files named main.c. Unless there is a really good reason for this super generic and same name, these should be renamed to the actual name of the program.

Heaven help the developer that has more than one of the main.c files open in an editor and confuses them. Nope, hasn't happened to me, not ever, nope, not five minutes ago either.

Just a thought, but since this is dealing with mseed3, it is a little confusing to have some commands named like mseed2json and some like mseed3-validator. Perhaps mseed3-json and mseed3-text. Otherwise they look like they might deal with miniseed version 2.

Somewhere in xseed2json/main.c:

    ierr = json_object_set_number (jsonObj, "reclen", msr->reclen);

    if (ierr == JSONFailure)
    {
      printf ("Something went wrong parsing to JSON : reclen");
      return EXIT_FAILURE;
    }

    ierr = json_object_set_number (jsonObj, "reclen", msr->formatversion);

    if (ierr == JSONFailure)
    {
      printf ("Something went wrong parsing to JSON : formatversion");
      return EXIT_FAILURE;
    }

    ierr = json_object_set_number (jsonObj, "reclen", msr->formatversion);

    if (ierr == JSONFailure)
    {
      printf ("Something went wrong parsing to JSON : formatversion");
      return EXIT_FAILURE;
    }

Would be useful if time output in mseed2text printed nanoseconds instead of microseconds.

Also, perhaps the output time should be more ISO friendly, like YYYY-DDD instead of YYYY,DDD. See "Ordinal Dates" in ISO8601.

Maybe add Z to be clear date is UTC.

Change:

             start time: 2012,001,00:00:00.000000

to

             start time: 2012-001T00:00:00.000000000Z

Note mseed2json already does some of this:

    "StartTime": "2012-01-01T00:00:00.000000000Z",

but might be good if it used day of year instead of month day since that is what is stored in the header?

Currently the payload checks are performed by re-reading and re-parsing the entire file being checked (in check_file.c), i.e. all files are read twice.

This should be improved by performing the payload check for each record while it is already read into memory, in the range noted by:

//TODO check payload via buffer, for now payloads are checked via libmseed after all headers are checked
...
//TODO validate payload using the buffer contains

The JSON Schema language allows $ref values to be "includes" of other schema components, either in the same schema structure or separate files. This is used internally in the ExtraHeaders-FDSN.schema.json schema file, so it works within a file. The intended method of combining FDSN and non-FDSN schemas is to create a simple schema document that combines all schemas using the $ref mechanism. xseed-validator does not work in this case.

The reference schema area contains and FDSN schema and some example schemas for Manufacturer123 and OperatorXYZ and a all-schemas.json file that combines all schemas. Using this file as the schema for validation should work for validating all the extra headers against all schemas.

The all-schemas.json file simply contains:

{
  "allOf": [
    {
      "$ref": "ExtraHeaders-FDSN.schema.json"
    },
    {
      "$ref": "ExtraHeaders-Manufacturer123.schema.json"
    },
    {
      "$ref": "ExtraHeaders-OperatorXYZ.schema.json"
    }
  ]
}

Validating a single record that includes an un-allowed extra header:

$ xseed-validator -j ../ExtraHeaders/ExtraHeaders-FDSN.schema.json reference-detectiononly.xseed.invalidEH

reports:

**********xseed-validator STARTING validation**********
---------------------------------------------------------

Reading file reference-detectiononly.xseed.invalidEH
Error in Schema Validation- Detection[0]: extra property 'DetectionWave' found.
*** Completed processing 1 records ***
xseed-validator RESULT - file reference-detectiononly.xseed.invalidEH is VALID xSEED

----------------------------------------------------------
*xseed-validator COMPLETE - 1 record(s) processed in 1 file(s)*
****xseed-validator SUCCESSFULLY validated 1 file(s)****

The summary messages quite (EMPHATICALLY) indicates that validation was successful even though it reported a validation error for the file.

When the validator is provided with a non-xSEED file (to generate a validation error), so the validator knows the difference:

*xseed-validator COMPLETE - 0 record(s) processed in 1 file(s)*
*xseed-validator FAILED to validate 1 file(s) out of the 1 file(s) processed*

The default (minimum) verbosity is too verbose, containing at least three lines for each file. This validator should be able to run on thousands of files and millions of records and this level of output quickly becomes overwhelming. The validator also produces some header and footer bracketing:

**********xseed-validator STARTING validation**********
---------------------------------------------------------

...

----------------------------------------------------------
*xseed-validator COMPLETE - 11 record(s) processed in 11 file(s)*
****xseed-validator SUCCESSFULLY validated 11 file(s)****

All but the last two lines are useless and the copious asterisks are visual noise. Perhaps the intention was to create parsable output, which could be useful, but this could be achieved with much less noise.

The current verbosity levels should all be moved up one, e.g. default becomes -v, and -v becomes -vv, etc.

The default minimum verbosity should only include the filename and validation errors when they occur, i.e. no output for files that are valid, and the final status of all files with a file count.

bin/mseed3-validator -v file_that_does_not_exist.ms3 ~/Downloads/seismograms/CO.BIRD.00.HHZ_2021-11-1_2.ms3 ~/Downloads/seismograms/CO.BIRD.00.HHZ_2021-11-1_3.ms3
Error! Cannot read file: file_that_does_not_exist.ms3, File Not Found! 
Reading file /Users/crotwell/Downloads/seismograms/CO.BIRD.00.HHZ_2021-11-1_2.ms3
mseed3-validator RESULT - file /Users/crotwell/Downloads/seismograms/CO.BIRD.00.HHZ_2021-11-1_2.ms3 is VALID miniSEED 3
Reading file /Users/crotwell/Downloads/seismograms/CO.BIRD.00.HHZ_2021-11-1_3.ms3
mseed3-validator RESULT - file /Users/crotwell/Downloads/seismograms/CO.BIRD.00.HHZ_2021-11-1_3.ms3 is VALID miniSEED 3

----------------------------------------------------------
mseed3-validator COMPLETE - 6 record(s) processed in 2 file(s)

Might be better to fail hard on a file not exists instead of continuing to process. Otherwise the error can be easy to miss in all the other text.

Alternatively, maybe change the COMPLETE message to FAILED. Basically the last output line probably should give an indication if there was a failure of any type.

I see the -W error arg, but even that doesn't halt on file not found.

The use of libmseed's internal msr3_unpack_mseed3() and msr3_unpack_mseed2() needs review. Ideally, these should not be used directly and the public API of msr3_parse() would be used.