Already builds on Windows, but requires the third-party dependencies repo from ScreamingUdder. First get the ESS conan packages working on Windows (googletest and ??).

In NexusFileReader.cpp:

For ISIS files the name of NXlogs are in the parent classes. We should add a check for ISIS files to set the name of the log to the log object's parent instead of the log itself. ESS files use the name of the log with the log object itself

When using a SANS_test.nxs to stream event data to a topic in the first frame all the detector IDs are set to 0. This is due to the memcpy function in the flatbuffers builder stage.

If NXlog is called value_log and it is an ISIS file then the parent group name should be used as the source name when publishing the data. Add a unit test for this.

Document in the readme and provide example configuration file (cli11 .ini)

Requested by Torben 05/09/2018

Currently makes assumptions based on format used at ISIS, for example that event_time_zero is a double in units of seconds.

Check if there are units of ns or nanoseconds, and falling back on ISIS behaviour if not, would be enough to support ISIS and example ESS files. I think this is sufficient for now as supporting all possible types and units which NeXus allows would add significant complexity.

• Don't assume raw_data_1 as the NXentry name.
• Don't rely on there being a good_frames dataset in the file
• Find sample environment information in NXlogs (remove dependence on finding selog group)
• Remove dependence on detector_1_events/total_counts as total_counts is not required by the NeXus standard, use size of one of the other datasets in the `NXevent_data instead.
• Generally try to search for particular NX_classs, rather than assuming a particular group name.

edit: Streaming event data from ESS files should now be in a working state. Just need to support reading sample env logs from NXlogs

Use /// and no separator between variable name and description.

An error is displayed at runtime with the containerised version.

/nexus_streamer/bin/nexus-streamer: error while loading shared libraries: libboost_filesystem.so.1.65.1: cannot open shared object file: No such file or directory

Containers section of readme: https://hub.docker.com/r/screamingudder/nexus-streamer/

To NeXusStreamer?

Events are published with correct timing based on pulse timestamps, but sample environment has a very crude implementation which assumes a constant 10Hz source.

This line should use proper pulse times somehow:

Pulse times in event data messages should look like "live" data, not be based on the offset attribute of the frame times in the NeXus file. This will make it more straightforward to use NeXus-Streamer with the NeXus File Writer which checks the pulse timestamp is within its start-stop time range.

Requested by Torben

Refactor CMake.

Currently half the unit tests fail and it isn't obvious what the problem is.

We have the hs00 schema now for histogram data. Owen has mentioned that support for this would be useful.

Message size grows when using random option and there are multiple event groups in the file. Observed with generated example files for LOKI.

Discovered by @LamarMoore - Report to him when fixed.

The date library https://github.com/HowardHinnant/date is already in use by the forwarder and file writer. It is a good choice as it is the implementation which will be included in C++20.

First need to conanise it!

Conan registry file is now .json not .txt.

The PR builds get deleted after merge so we'll have to revert to creating them for now commit builds.

Profiling shows a lot of time is spent in flatbuffers::createVector(). Jonas suggests instead using CreateUninitializedVector. Maybe something like this:

std::vector<things> dataVector = {0, 1, 2, 3, 4, 5};
size_t dataSizeBytes = dataVector.size() * sizeof(things);
std::uint8_t *tempPtr;
auto payload = builder.CreateUninitializedVector(dataSizeBytes, &tempPtr);
std::memcpy(tempPtr, dataVector.data(), dataSizeBytes);

• document

This is for the situation that the numbers in NXevent_data/event_id are detector numbers and not spectrum numbers. Send a one to one map using detector_number in the NXdetector group.

• east const

Add a CLI option to not use/send a detector spectrum map file.

Also create a docker image for this for use in mohican-yak

The map is actually in ISIS NeXus files, we don't actually need the separate map file.

Initial histogram support was implemented in #90 but this needs to be tested against the histogram listener in Mantid. Also check for consistency with just-bin-it and maybe how the schema is used at PSI too?
For example dimension names need to match what is expected by the listener in Mantid. May also require particular type for time-of-flight bin edges, or other fields?

Update Kafka version, remove host network mode in places it is not necessary etc.

Remove the spectra associated with the integration card from the sample map file: https://github.com/ess-dmsc/NeXus-Streamer/blob/master/data/spectrum_gastubes_01.dat
(Top half of the spectra numbers, not including monitors)

These otherwise result in a lot of wasted memory downstream for Mantid as the empty spectra get rebinned along with the populated ones. Freddie will solve the issue for the real instrument.

It seems to be mostly the print statements that need changing, so should be straightforward.

Rather than the HDF5 library directly. This should reduce the verbosity of NeXusFileReader.cpp.
It will also make it easier to create in-memory test "files" for unit testing changes in #3.

The scripts directory and its contents can be deleted; they are obsolete now that we have kafkacow.

As only 5 are kept it makes sense for them to be only from PR builds.

Document what to do if conan doesn't pick up compiler settings.
Include compiler version requirements in the build information.

Requested by Torben 05/09/2018

Requested by Torben 05/09/2018

If available in stream Mantid would make use of this information instead of loading an IDF for the instrument.

This is part of the work required to support streaming geometry for Torben. Also need to convert IDF from McStas to NeXus geometry, or directly implement NeXus geometry (+ Nxevent_data) in McStas?

sans_demo.ini?

By putting the conan installation in a separate step and before the source is copied they will only be built if outdated instead of every time the source code changes.

"Unsupported datatype" should be warnings for example.

Causes kafkacow to try to print the message as a string which results in an error.

We don't use spectrum numbers for ESS file (yet, at least), so generate a 1 to 1 mapping (in other words treat the IDs in the NeXus files as detector IDs rather than spectrum numbers) if no file is provided.

This is not currently high priority.

This would simulate the situation that a different EFU is running for different detector panels of an instrument.

Consider if it should be an option rather than always being the case.
Would require the topic to already exist with the correct number of partitions, or if topic did not exist it could be created with the new Admin API functionality (which I think requires the next release of librdkafka).

Requested by Torben 05/09/2018

Note, can mark mandatory arguments with CLI11:

  App.add_option("-f,--filename", settings.filename,
                 "Full path of the NeXus file")
      ->check(CLI::ExistingFile)
      ->required();
  App.add_option("-d,--det_spec_map", settings.detSpecFilename,
                 "Full path of the detector-spectrum map")
      ->check(CLI::ExistingFile)
      ->required();
  App.add_option("-b,--broker", settings.broker,
                 "Hostname or IP of Kafka broker")
      ->required();
  App.add_option("-i,--instrument", settings.instrumentName,
                 "Used as prefix for topic names")
      ->required();
  App.add_option("-m,--compression", settings.compression,
                 "Compression option for Kafka messages");
  App.add_option("-e,--fake_events_per_pulse", settings.fakeEventsPerPulse,
                 "Generates this number of fake events per pulse instead of "
                 "publishing real data from file");
  App.add_flag("-s,--slow", settings.slow,
               "Publish data at approx realistic rate (detected from file)");
  App.add_flag("-q,--quiet", settings.quietMode, "Less chatty on stdout");
  App.add_flag(
      "-z,--single_run", settings.singleRun,
      "Publish only a single run (otherwise repeats until interrupted)");
  App.set_config("-c,--config_file", "", "Read configuration from an ini file",
                 false)
      ->check(CLI::ExistingFile);

Good to be consistent

The streamer should not require good_frames dataset or total_counts dataset.
It should warn but not cause a problem if it does not find sample env logs in the file.

Allow proton_charge and period_number and run_state to be missing from file, in which case don't add ISIS-specific data to event data messages.

This will make life easier for Torben and co using it for the DMSC integration pipeline.

Currently only data from the first NXevent_data that the NeXus Streamer finds is published to Kafka. It should publish from all, keep in mind that it should eventually have the option to publish data from different NXevent_data groups into different partitions, but that is not required for this ticket.

This ticket is a request from Lamar 17/01/2019.