all pending jobs killed after Flux update about flux-accounting HOT 5 CLOSED

OK, I actually think I was able to reproduce this without having to restart Flux and instead just unloading/reloading the plugin. If I have a number of jobs in SCHED state (i.e they've received a priority and are waiting to run) and I reload the plugin without updating it with flux-accounting information and call reprioritize on all jobs:

(flux.Flux().rpc("job-manager.mf_priority.reprioritize"))

they will have an exception raised on them saying that the plugin cannot find a valid user/bank entry for those previously held jobs.

Is there a process in restarting Flux where jobs could be reprioritized? My thinking is that might have been what caused this.

In any case, the plugin should probably handle this case more gracefully so a bunch of users' jobs don't get canceled if Flux gets restarted.

I'll need to test this, but off the top of my head, I think I can include a check in the callback for job.state.priority that checks the plugin's internal map for data before deciding what to do with the job going through reprioritization.

If the plugin's internal map is empty (i.e it is waiting for flux-accounting information), it can continue to hold the job in PRIORITY until it loads some information. This would be similar to the behavior in the callback for job.validate.

from flux-accounting.

Nice job debugging @cmoussa1!

All jobs are prioritized any time a jobtap plugin is loaded, so I had assumed this would happen after mf_priority.so is loaded.

from flux-accounting.

Without trying to construct a reproducer (yet), I believe this type of exception message is raised when a job's user/bank information is being updated in job.state.priority and the plugin cannot find a valid user/bank entry for the user that this job is submitted under:

Looking at the timestamps of the eventlog, it looks like the job was:

1. submitted successfully (and received a priority)
2. Flux was restarted
3. the priority plugin was loaded
4. jobs were reprioritized before the plugin received any flux-accounting data, so it rejected this job and presumably all pending jobs.

I'll see if I can reproduce this behavior.

from flux-accounting.

Closed by #407?

from flux-accounting.

Ah, yes, should be closed by #407 - sorry that I didn't close this yesterday. Closing now

from flux-accounting.