qbicsoftware / dropboxhandler Goto Github PK

The existance of used directories in the config file should be checked when the conifg check is applied. Otherwise the config check might run through but starting the dropboxhandler still fails if a specified directory does not exist.

Since the requirements have changed, we also accept datasets that have the necessary information in an additional metadata file.

The availability of the correct metadata and dataset structure is part of an own validation process now, so no need to explicitly check this in the dropboxhandler anymore.

The latest version does not allow datasets without a valid QBiC sample id in the folder name, therefore we currently can't route the new datasets properly along the registration path, the transfer is interrupted.

Sometimes, after successful copying of a dataset - at least from one mount point to another - the process fails, because shutils.rmtree complains about a non-empty tmp folder:

2022-11-28 20:58:20,429 dropboxhandler line:274 (levelname)-8s An error occured while handeling file. Creating error marker file /mnt/nfs/qbic/dropboxes/qeana03_imgagdna/incoming/MARKER_error_20221127232052_[...]_nanopore, remove if you fixed the error. Error was: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/dropboxhandler-1.8.1-py2.6.egg/dropboxhandler/dropboxhandler.py", line 248, in _handle_file self.to_openbis(origin, file, perms=perms) File "/usr/lib/python2.6/site-packages/dropboxhandler-1.8.1-py2.6.egg/dropboxhandler/dropboxhandler.py", line 163, in to_openbis file, dest_file, tmpdir=self._tmpdir, perms=perms File "/usr/lib/python2.6/site-packages/dropboxhandler-1.8.1-py2.6.egg/dropboxhandler/fstools.py", line 249, in recursive_copy shutil.rmtree(tmpdir) File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree rmtree(fullname, ignore_errors, onerror) File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree rmtree(fullname, ignore_errors, onerror) File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree rmtree(fullname, ignore_errors, onerror) File "/usr/lib64/python2.6/shutil.py", line 221, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/usr/lib64/python2.6/shutil.py", line 219, in rmtree os.rmdir(path) OSError: [Errno 39] Directory not empty: '/mnt/nfs/qbic/dropboxes/tmp/tmp6ENlDz/[...]_nanopore/20221122_[...]/fast5_pass'

While the data is complete, the process has to be repeated, as the checksums were not created due to the exception. It is likely that setting ignore_errors=True in the following call would fix the problem:

See documentation: https://docs.python.org/3/library/shutil.html#shutil.rmtree