exmala
Comparing different approaches to parse large XML messages.
Streaming
When not buffering and writing to multiple parsers at once we risk running out of memory since all these parsers keep a stack and other stateful things around that cost us memory.
Depending on the parser lib we use that happens at a specific level of concurrency.
htmlparser2
settings:
{ parser: 'htmlparser',
streaming: true,
concurrency: 1000,
number: 2000,
gc: false,
file: '/Users/thlorenz/dev/js/exmala/data/ibm-request.soap.xml',
resultsFile: '/Users/thlorenz/dev/js/exmala/results/streaming-htmlparser-c1000-n2000_ibm-request.soap.xml.json' } }
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
xml2js
{ settings:
{ parser: 'xml2js',
streaming: true,
concurrency: 500,
number: 2000,
gc: false,
file: '/Users/thlorenz/dev/js/exmala/data/ibm-request.soap.xml',
resultsFile: '/Users/thlorenz/dev/js/exmala/results/streaming-xml2js-c500-n2000_ibm-request.soap.xml.json' } }
FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory
License
MIT
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent