How to avoid Out of memory about edifact HOT 13 CLOSED

Is possible do some optimisation:

• https://github.com/php-edifact/edifact/blob/master/src/EDI/Parser.php#L238 replace with while($line = array_shift($file)) {
• https://github.com/php-edifact/edifact/blob/master/src/EDI/Parser.php#L102 implement also array_shift() and result collect direct in $this->parsedfile
• "public function parse($file2)" change to "public function parse(&$file2)"
• public function loadString(&$string)
  {
  $this->rawSegments = $this->unwrap($string);
  return $this->parse($this->rawSegments);
  }

from edifact.

Are there multiple messages (so multiple UNH) or is a single message (one UNH) with a lot of repetitions?

from edifact.

It's one UNH but a plethora of LIN elements. It's basically a big catalogue file.

from edifact.

@uldisn
Thx for your input.

I've got the first and third bullet point implemented.

But I am lost at second and last bullet point

public function loadString(&$string) { $this->rawSegments = $this->unwrap($string); return $this->parse($this->rawSegments); }
Is this a new function in Parse.php? Or do I need to edit another file?

https://github.com/php-edifact/edifact/blob/master/src/EDI/Parser.php#L102 implement also array_shift() and result collect direct in $this->parsedfile

That's the line:

foreach ($file2 as $x => &$line) {
simply replace with

while($line = array_shift($file)) { ??? that doensn't seem right
while($line = array_shift($file2)) { perhaps? but x is gone.

Any hints?

from edifact.

Did I write something wrong?

from edifact.

Sorry! I do not have time validate it, but main idea is use while($r = array_shift($a)) instead foreach ($a as $r) and avoiding duplicating data in several variables

from edifact.

@teflontoni could you share this file privately?

from edifact.

@sabas here you go:
bn_new.zip
it's just an article catalogue, so I think it's ok to share it right here.

@uldisn
Okay, I'll give it a try, thanks for your response.

from edifact.

I tried last night increasing to 512 MB of allocated RAM and it started processing, but I haven't yet timed how much it lasts...

from edifact.

@teflontoni I pushed a commit which "solves" the out of memory problem with your file. I increased the allocated RAM to 1 GB and it terminated after less than 1 hour (still a bit too much I think :D).

@uldisn I used some of your ideas, but I noticed a potential problem on your while loop https://stackoverflow.com/questions/49212963/why-array-shift-on-an-array-with-an-empty-string-breaks-the-loop

from edifact.

... With 2GB of ram allocated the Parser completes almost immediately now....

from edifact.

A thing worth noting is that all lines of a file are held in memory twice in the Parser: once in the rawSegments property and once in the parsedfile one. It might be a good idea to unset the rawSegments property when Parser::parse() is executed to save memory. If you need the rawSegments, you could retrieve them right after executing one of the load methods. Of course, then Parser::loadString() would need to stop running the parse() method, and you'd need to run it explicitly yourself.

from edifact.

@gaxweb we could do that marking it a breaking change and tagging a new release, I'm open to it!

from edifact.