JobTimeoutException Leaves Temp Files about ckanext-xloader HOT 3 OPEN

Not even sure if this is possible to fix in Xloader here. I think you need to use rq's push_exc_handler inside of the ckan.cli.jobs.worker method. So doing some debugging and seeing if I can make an implement in Core code to add exception handlers for the jobs worker.

And then it would be a matter of figuring out how to get the temp file path/name into the implemented exception handler in Xloader here.

from ckanext-xloader.

Sorry for the sporadic comments on this one. More debugging, I found that we could catch JobTimeoutException during the downloading of the file into the temp file, and clear it there.

The issue I am still having is during the process of when the temp file contents are being copied into the database in the loader.py script. This is when, if JobTimeoutException is raised, the temp file remains.

Currently I am debugging all of this with load_table and not load_csv.

from ckanext-xloader.

Okay yeah we can fix this in Xloader. (e.g. open-data@62ed5a0#diff-69e6ff3cab84fe327b715b7b1d65f7cb9660b09e076ec36dbdb5e12ffeebe3f6)

Will make a PR next week if I have time. But we need to catch the rq timeout exception in a couple places and then just close the tmp file.

from ckanext-xloader.