The objective is to avoid downstream processes needing to use heuristics to tell if a run has completed (and completed copying too). The sequencing_summary.txt should be the last file to be created by the run. However, it may not be the last file to arrive at the destination when there are concurrent copy operations in progress. One option is to copy sequencing_summary.txt copy only after a period of idle time. This doesn't guard against the run being restarted (but does that create a new run folder?)

Loose files in /data get picked up and archived if they match the archive criteria. These should be excluded. Only files under runfolders should be archived.

valet must have a manpage.

Document that valet respects TMPDIR and that it should be set appropriately (to the same filesystem as the data).

Capture the new JSON and HTML files created by MinKNOW. Also forthcoming pod5 files (which we do not see yet)

The large, important data files must have checksums calculated at rest. We should not do this for everything, e.g. small text files, to avoid an explosion of small files for no gain in data integrity.

https://github.com/kjsanger/valet/blob/ce0b7086f8d9717cc01a2a92a93573f681791c3b/.travis.yml#L17

It is possible for race conditions to arise where e.g. filesystem sweeps are frequent relative to job runtime, resulting in more than one work function operating in parallel on the same path. valet should prevent this.

In the meantime, work functions should be written to fail gracefully in these cases, or to recover from errors internally.

The parser currently supports GridION device IDs only.

Parsing the report Markdown file at the end of publishing fails with the error message e.g. Failed to parse device ID '2-A7-D7'.

The channel for non-fatal runtime errors has a buffer size of 1, intended to allow at least one error to be sent and a non-zero exit to be possible. However, if mutiiple errors are sent by by the watcher or finder, they get blocked on sending and can become impossible to cancel cleanly.

There should be an additional goroutine to constanly drain the error channel and act on the errors appropriately. Mostly we want to keep on working, if we can, but also have a clean cancel.

There will be some additonal work to act appropriately on errors that we really want to be fatal.

In some cases there are deep sub-hierachies within the target hierarchy that are never interesting and are simply a waste of monitoring resources. We need to be able to prune the tree for both monitoring and interval sweeps. This should be settable from the command line.

TMPDIR is defined, but not used, in valet_archive_create.sh

Runs on the PromethION-24 (MinKNOW 4.5.4) that are restarted do not get the usual Markdown report file created and sometimes do not get the final PDF report either.

We rely on the Markdown report for automated metadata extraction. The final report .txt file seems always to be present and contains most of the same data. However, it does not contain the MinKNOW or Guppy software versions.

If the archive root is set to a non-existent directory, cancelling hangs because WatchFiles returns before the cancel has been set up at

https://github.com/kjsanger/valet/blob/devel/valet/pathwatch.go#L61

The solution is to capture the error and allow the cancel to be set up before returning.

A successful archive operation means the local copy is deleted. There should be a switch to turn this on and the default should be off.

The GridION and PromethION run folders contain additional text and PDF files that are not handled by the archiver. They should be archived too, perhaps not using md5 checksum files .

The md5 checksum files generated locally for data files will remain in place after archiving, unless deleted.

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

verifying github.com/kjsanger/extendo/[email protected]/go.mod: checksum mismatch
	downloaded: h1:ghBNpbUzejGnKbx54isdXlmXfMxh4vNih0olZyPTA8A=
	go.sum:     h1:Xibv5xvLjDUcqCB2dh4uuLxjY2NYa4inOVRCTwCJOcY=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

The ONT pipeline creates uncompressed Fastq files. We should compress and checksum these locally before archiving.

If TMPDIR is set to be within the /data filesystem, automatically exclude it from directory walks, rather than have the user be required to use a --exclude CLI option if they have set things up this way.

When adding ONT metadata to iRODS, use the ont namespace for the attributes to avoid clashes with existing (non-namespaced) attributes e.g. sample_id

Experiment name is currently added under the ONT nomenclature of protocol_group_id (from the report.md file). The position is currently added as device_id (also from the report.md file), however the value is of the form e.g. GA10000 or X1 for position 1, and needs to be normalised.