Option to De-duplicate output about corb2 HOT 3 CLOSED

It may be more efficient to do this using simple shell command than doing it in java (ex: sort | uniq) Report files can get pretty big and entire file might need to be read into HashSet or LinkedHashSet. May be we can add this option and warn about the memory consumption by checking the free mem.

from corb2.

We could evaluate the size of the file and the available memory. If small enough, perform in-memory operations. If the file is extremely large, then split it into small temp files and perform an external merge sort.

from corb2.

I agree, it would be a good idea to evaluate memory, we should do this in the next version. Please note that post batch task first writes the de-duplicated/sorted results temp file before renaming it to original. So, if XMX is too low for the data it is trying to handle, post batch task will fail with out of memory exception, but the original file with duplicates (or unsorted) will remain untouched. This may not be too bad for now. When we try to handle, we should check memory requirements and throw a warning message instead of failing with out of memory error.

from corb2.