Comments (8)
Yes, the POST-BATCH-MODULE currently only executes if there are no errors and the count of items is greater than 0, but that sounds reasonable.
We should be able to relax that and execute the POST-BATCH-MODULE when there are no results. We just need to verify that the PostBatchUpdateFileTask would not throw exceptions and adjust as necessary to accommodate.
from corb2.
Also, when this feature was added, this behavior was intended i.e., we didn't want to automatically execute both pre- and post- batch modules when the uris count is '0' and the job will leave no trace on the system.
However, there is a work around that we often use - If there are no uris to return, uris module can return a single pre-defined dummy URI that transform can simply ignore. This will ensure that both pre and post match modules will be executed. Please let us know if this works.
If we do relax the constraint and execute the pre and post batch modules, we may need to do this conditionally (as an option, which defaults to 'false') as we do not want to break other use cases.
from corb2.
@bbandlamudi What about exposing an option POST-BATCH-MINIMUM-COUNT that allows the user to set a threshold for how many results there must be in order for the POST-BATCH (module or task) to execute?
We would set the default value to 1, in order to ensure backwards compatibility.
from corb2.
from corb2.
I don't see anything that would currently prevent the PRE-BATCH module/task from executing.
We invoke runPreBatchTask() without any conditions, and inside that method the only condition is that it isn't null
.
from corb2.
Corb actually exits a little before that - if the total count from uris module is <= 0, then corb prints 'nothing to process' and exits with returns code 0. I believe, pre- and post- batch modules should go hand in hand.
from corb2.
Yes, of course (two lines up prior to invoking the PRE-BATCH).
That makes the current condition verifying that count > 0
for the POST-BATCH duplicative and irrelevant, but would also be useful for specific conditions to control PRE/POST execution.
We could have a global property to enable/disable the PRE/POST regardless of what the count is(default false), and also allow users to customize the behavior with a specific properties to control the PRE-BATCH and POST-BATCH execution (either booleans or minimum count).
I'll get something working, and will think about a sensible name for the global flag.
from corb2.
There is a little bit of backwards compatibility risk with the global property that enables or disables pre/post batch module. Also, we could always argue that this is simply equivalent to specify/not-specifying the modules in the options file. We have a set of inter dependent corb jobs where one should create a report even with 0 uris count (execute pre- batch module too generate the header) and another shouldn't create any report file when uris count is 0 (as it generates an xml file, which cannot be just open/close tags as it fails schema validation). I would like to keep this flexible.
May be we should simply execute pre- and post- batch modules all the time and pass in a variable called 'uris-count' to pre- and post batch modules and let them decide what to do.
from corb2.
Related Issues (20)
- Automatically use ExportBatchToFileTask when setting EXPORT-FILE-NAME
- Update SSL config to use TLSv1.2 HOT 3
- Issues running Corb via AppServicesPort HOT 5
- SchemaValidateBatchToFileTask not appending content, overwriting with last item
- Better diagnostics for easily detectable developer errors in QueryUrisLoader
- Corb runs selector even when the job is started with COMMAND=STOP
- Option UseConcMarkSweepGC deprecated
- QueryUrisLoader uses Integer.parseInt to obtain total count
- mangled output when default character encoding is not UTF-8
- Job server web page should update after the job completes HOT 2
- CORB running mjs modules issue HOT 1
- Adjustments for Java 11+ and deprecated JAXB javax.xml.bind package HOT 1
- Require PROCESS-MODULE to be set when using Export*ToFileTask
- TLS certificate failing to connect (but the same cert works fine with MLCP) HOT 4
- Suggestion: add deprecated bind libraries to build.gradle HOT 1
- Support external header (and footer) for Module Executor
- expose Manager execError with hasExecError() method
- improve usage messages
- Illegal base64 character encountered in PrivateKeyDecrypter.doDecrypt for unencrypted values that contain - HOT 1
- OPTIONS_FILE assumes ISO 8859-1 encoding HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from corb2.