Comments (5)
kputnam
commented on June 11, 2024
Never say never! I have a separate branch that drastically improves resource consumption (memory and CPU) and Iām making further improvements by implementing some parts in C.
The library has been very stable without major bugs for a long time, so adding native code is a big risk. Iām currently taking a break, but the next task is writing a thorough randomized test suite. I hope to resume work this summer.
Sorry there is not a great workaround in the meantime. Others have had success with spotting large files apart into small single ST/SE transactions. That cannot be done efficiently using stupidedi however.
ā¦On Thu, Jun 17, 2021, at 6:44 AM, rob mathews wrote:
Because the parser parses the entire input file completely into memory, instead of streaming it section by section.
So it worked on the 400K test file from my client in reasonable, but failed on the 176MB actual file, taking 18GB of memory and never actually finishing.
Otherwise, a very nice library.
ā
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#237>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABZSFRMZDJSQX6B6NFUNGDTTHNZHANCNFSM463P2ZLQ>.
from stupidedi.
robmathews
commented on June 11, 2024
I've looked at your performance improvements in your other branch, but they are not sufficient. A 10% or even 100% improvement doesn't address the basic problem.
Basically, unless the code ceases to attempt to parse the entire tree, it will never work for files with ~600K records in them.
But, hey, feel free to defy the laws of physics, I'm not stopping you!
from stupidedi.
kputnam
commented on June 11, 2024
Okay Rob, thanks for your feedback!
from stupidedi.
robmathews
commented on June 11, 2024
wow, an actual real time answer. I'm stunned.
from stupidedi.
robmathews
commented on June 11, 2024
And seriously, you can't just close this; your library doesn't work for real life situations like parsing 834 elibilitiy files, so it would be nice if your documentation noted that.
from stupidedi.
Related Issues (20)
- Question: Is it possible to override or monkey patch validations? HOT 2
- Need help for parsing a very simple piece of file HOT 2
- Easy Guide HOT 3
- Support for pattern matching (Ruby 2.7+)
- Unable to set CSH segment HOT 2
- non-deterministic machine state revisitied HOT 4
- problems customizing a transaction set HOT 3
- Transaction splitting using notes/split.rb script: out of date? HOT 3
- iterating over files with multiple ISAs HOT 3
- Duplicate Keys HOT 3
- Unknown patient gender in 270 causes validation error HOT 2
- value PR is not allowed in element NM101 HOT 2
- Segment ST Does not occur HOT 2
- Parse EDI 850 ERROR HOT 1
- Accept Tempfile in Stupidedi::Reader.build HOT 1
- Segment N1*PR~ cannot be reached
- Parsing an 837 and grabbing segment values HOT 2
- Is there a way to skip validations and build without some of the segments HOT 2
- 004010 version - B3-07 precision doesn't work as intended HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stupidedi.