FfDL logging issues with Elastic Search about ffdl HOT 6 CLOSED

Probably first thing to try is to see if we can monkey with templates/infrastructure/storage.yml to give it a bit more memory.

from ffdl.

@sboagibm dont see any memory field here
https://github.com/IBM/FfDL/blob/master/templates/infrastructure/storage.yml

from ffdl.

@whummer do you know off the top of your head how to allocate more memory there?

from ffdl.

Shouldn't allocating more memory work the same way as with containers, i.e. something like the following?

      requests:
        memory: "500Mi"
      limits:
        memory: "1000Mi"

What are your specs? How much RAM do you have and how much space does /var/lib/elasticsearch have left?

You could also try increasing the Java heap size with -Xmx (maximum heap size) and -Xms (initial heap size) in jvm.options. jmap -histo <pid> should provide a heap histogram, cmp. https://docs.oracle.com/javase/10/tools/jmap.htm

Finally: What are bootstrap.memory_lock, indices.memory.index_buffer_size and MAX_OPEN_FILES?

[If all of those check out, this might just be normal garbage collection, if I'm not mistaken. 25% does not seem too critical if it does not occur too frequently. The threshold for hard warnings is 50%, iirc.]

from ffdl.

Thanks @k0105 - We will take a look @Tomcli @whummer

from ffdl.

Thanks @k0105, this issue was caused by some helper pods that has insufficient memory. I did a quick PR that will fix this issue.

from ffdl.