junkybyte / iris Goto Github PK

Check the possibility to continuously sync a file being written continuously e.g. by a bash script.

echo "avg_walltime" > file.csv
for i in $(eval echo {0..K});
do
  echo $i >> file.csv
  ... sleep
done

this will only be sync by watchdog on file creation / first change

Manual read write was useful when this project started because allowed full control over the process.
asyncssh has put and get methods for sftp now, they should be faster than current method and allow for similar progress callback.

Deleting dir while watchdog is running doesn't delete it on remote. TODO

Especially with mirror=True if the initial file sync is interrupted a truncated file is written. When running iris again it will have newer mtime / atime than og files and will break the sync by overwriting the original files.

A solution might be to use tmp files on write and renaming them only on completion. This way is possible to do cleanup on run and be sure that files are only present when complete (and with correct atime mtime)