Comments (3)
It may be more efficient to do this using simple shell command than doing it in java (ex: sort | uniq) Report files can get pretty big and entire file might need to be read into HashSet or LinkedHashSet. May be we can add this option and warn about the memory consumption by checking the free mem.
from corb2.
We could evaluate the size of the file and the available memory. If small enough, perform in-memory operations. If the file is extremely large, then split it into small temp files and perform an external merge sort.
from corb2.
I agree, it would be a good idea to evaluate memory, we should do this in the next version. Please note that post batch task first writes the de-duplicated/sorted results temp file before renaming it to original. So, if XMX is too low for the data it is trying to handle, post batch task will fail with out of memory exception, but the original file with duplicates (or unsorted) will remain untouched. This may not be too bad for now. When we try to handle, we should check memory requirements and throw a warning message instead of failing with out of memory error.
from corb2.
Related Issues (20)
- Automatically use ExportBatchToFileTask when setting EXPORT-FILE-NAME
- Update SSL config to use TLSv1.2 HOT 3
- Issues running Corb via AppServicesPort HOT 5
- SchemaValidateBatchToFileTask not appending content, overwriting with last item
- Better diagnostics for easily detectable developer errors in QueryUrisLoader
- Corb runs selector even when the job is started with COMMAND=STOP
- Option UseConcMarkSweepGC deprecated
- QueryUrisLoader uses Integer.parseInt to obtain total count
- mangled output when default character encoding is not UTF-8
- Job server web page should update after the job completes HOT 2
- CORB running mjs modules issue HOT 1
- Adjustments for Java 11+ and deprecated JAXB javax.xml.bind package HOT 1
- Require PROCESS-MODULE to be set when using Export*ToFileTask
- TLS certificate failing to connect (but the same cert works fine with MLCP) HOT 4
- Suggestion: add deprecated bind libraries to build.gradle HOT 1
- Support external header (and footer) for Module Executor
- expose Manager execError with hasExecError() method
- improve usage messages
- Illegal base64 character encountered in PrivateKeyDecrypter.doDecrypt for unencrypted values that contain - HOT 1
- OPTIONS_FILE assumes ISO 8859-1 encoding HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from corb2.