POST-BATCH-MODULE not executed for empty uris about corb2 HOT 8 CLOSED

Yes, the POST-BATCH-MODULE currently only executes if there are no errors and the count of items is greater than 0, but that sounds reasonable.

We should be able to relax that and execute the POST-BATCH-MODULE when there are no results. We just need to verify that the PostBatchUpdateFileTask would not throw exceptions and adjust as necessary to accommodate.

from corb2.

Also, when this feature was added, this behavior was intended i.e., we didn't want to automatically execute both pre- and post- batch modules when the uris count is '0' and the job will leave no trace on the system.

However, there is a work around that we often use - If there are no uris to return, uris module can return a single pre-defined dummy URI that transform can simply ignore. This will ensure that both pre and post match modules will be executed. Please let us know if this works.

If we do relax the constraint and execute the pre and post batch modules, we may need to do this conditionally (as an option, which defaults to 'false') as we do not want to break other use cases.

from corb2.

@bbandlamudi What about exposing an option POST-BATCH-MINIMUM-COUNT that allows the user to set a threshold for how many results there must be in order for the POST-BATCH (module or task) to execute?

We would set the default value to 1, in order to ensure backwards compatibility.

from corb2.

from corb2.

I don't see anything that would currently prevent the PRE-BATCH module/task from executing.

We invoke runPreBatchTask() without any conditions, and inside that method the only condition is that it isn't null.

from corb2.

Corb actually exits a little before that - if the total count from uris module is <= 0, then corb prints 'nothing to process' and exits with returns code 0. I believe, pre- and post- batch modules should go hand in hand.

from corb2.

Yes, of course (two lines up prior to invoking the PRE-BATCH).

That makes the current condition verifying that count > 0 for the POST-BATCH duplicative and irrelevant, but would also be useful for specific conditions to control PRE/POST execution.

We could have a global property to enable/disable the PRE/POST regardless of what the count is(default false), and also allow users to customize the behavior with a specific properties to control the PRE-BATCH and POST-BATCH execution (either booleans or minimum count).

I'll get something working, and will think about a sensible name for the global flag.

from corb2.

There is a little bit of backwards compatibility risk with the global property that enables or disables pre/post batch module. Also, we could always argue that this is simply equivalent to specify/not-specifying the modules in the options file. We have a set of inter dependent corb jobs where one should create a report even with 0 uris count (execute pre- batch module too generate the header) and another shouldn't create any report file when uris count is 0 (as it generates an xml file, which cannot be just open/close tags as it fails schema validation). I would like to keep this flexible.

May be we should simply execute pre- and post- batch modules all the time and pass in a variable called 'uris-count' to pre- and post batch modules and let them decide what to do.

from corb2.