Troubleshooting on Java program execution about isolate HOT 9 CLOSED

Are you trying to limit the address space of the JVM? Don't do that, they assume they are able to use a lot of virtual memory when running. Limit the maximum resident memory instead.

from isolate.

Well I also provided --mem option in isolate, does that limit the max RSS? Also why are limiting stack size and heap size through JVM options not recommended?

My primary goal is to make java program run (by any way / without limits) and I haven't succeeded it through isolate yet

from isolate.

--mem limits the address space, as documented here. You want to use --cg-mem to limit the resident memory. I advise you to never use --mem, it's most of the time useless to limit the address space of the programs you're running, except maybe for programs that don't do a lot of virtual memory mapping.

from isolate.

Thanks a lot! Let me post a followup comment after i try it

from isolate.

For reference, here's how we use Java in CMS: https://github.com/cms-dev/cms/blob/master/cms/grading/languages/java_jdk.py

from isolate.

I'm still getting the same exception even though I haven't made any resource limits and just ran java -version

$ isolate --init  # o.k.
$ isolate --run -- /usr/bin/java -version  # error

The error occurs immediately after I enter the command and error message is identical as the one in gist link above

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (gcTaskThread.cpp:48), pid=2, tid=0x00002b451f523b88
#
# JRE version:  (8.0_151-b12) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.6.0
# Distribution: Custom build (Tue Nov 21 11:22:36 GMT 2017)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

Some other notes...

• Docker image is based on openjdk/8u151-jdk-alpine3.7, running in privileged mode
• It works well if I just run /usr/bin/java -version without isolate.
• Specifying following JVM options yielded same results
  • GC Thread options -XX:ParallelGCThreads=2
  • Reserved code cache size option -XX:ReservedCodeCacheSize=16M
  • Heap and stack size option Xss??M, Xmx??M
• ulimit -a inside the container (not isolate sandbox) gives you the following result

-f: file size (blocks)             unlimited
-t: cpu time (seconds)             unlimited
-d: data seg size (kb)             unlimited
-s: stack size (kb)                8192
-c: core file size (blocks)        0
-m: resident set size (kb)         unlimited
-l: locked memory (kb)             82000
-p: processes                      unlimited
-n: file descriptors               1048576
-v: address space (kb)             unlimited
-w: locks                          unlimited
-e: scheduling priority            0
-r: real-time priority             0

from isolate.

Well, it worked after I set --processes=X where X >= 11
I'm trying to figure out why so much default processes are required (even if I specify -XX:ParallelGCThreads=1) but finally it worked

from isolate.

I was going to say one of the pitfalls of isolate is running it without options thinking it won't add any restrictions, because the --processes option is the only one that have a limit by default.

I'd strongly advise to never use --processes to isolate the JVM, as the errors it will produce will be very very hard to see. At some point, we had Java just hang without dumping any state to stderr, and we couldn't debug the issue at all until I just randomly bumped the process limit from 20 to 100 and it worked.

There are not only GC threads, also JIT compilation threads, you can also specify ConcGCThreads etc... Java is just really bad at being isolated (https://jaxenter.com/nobody-puts-java-container-139373.html) either because of the number of processes required, or because of its habit of looking at the available physical RAM to see how much virtual memory it is going to allocate (it's really a percentage of the total RAM...).

For what it's worth, it's supposed to be better in Java 10: https://blog.docker.com/2018/04/improved-docker-container-integration-with-java-10/

from isolate.

from isolate.