Kafdrop 4.0.0 Docker image doesn't start about kafdrop HOT 15 CLOSED

Well, this is the Docker image, we are not modifying it anyhow. We are not using it as a base image for our image; it's used "as is". So there are no our libraries involved. I believe Kotlin is used indirectly as a dependency of some module in your project; most probably - its somehow related to SpringBoot. But it's definitely exists there: please check the screenshot of contents of obsidiandynamics/kafdrop:latest Docker image I made in Docker Desktop:

On the right side, I filtered used packages by the 'kotlin' keyword. You can see there are some of them. The previous (working) version, 3.31.0, also contains them, but the lower versions, of course:

More of that: I tried to run docker-compose.yaml your project has in docker-compose/kafka-kafdrop folder with exactly the same result: the kafka container was started successfully, while the kafdrop failed with exactly the same error.
I'm not saying the error is somewhere in the Kafdrop code. It could be that it's somewhere in the base image Kafdrop using for the own image: eclipse-temurin:17.0.8.1_1-jdk. Or it's something related to the way how the image prepared on your side; we experienced a similar problem in the past in our building environment.

from kafdrop.

Yes @mehdihasan, I can confirm, that with the -Xss360K settings Kafdrop4 image starts w/o problems. Thank you! I did test it on my local system so far.

Upd: On the remote systems it started to work as well. Thanks again!

from kafdrop.

I just got the same Error while trying to start the image. Downgraded to 3.31.0 - and no issues with it.

from kafdrop.

Hi @mehdihasan, thank you for the efforts, first of all!

Regarding the used configuration, etc.:
Actually, there are nothing special. Our solution's services are represented as the Docker images and runs in the Docker Swarm mode. Each Swarm has few physical hosts. At the moment, hosts are running under CentOS 7.9.2009, the Docker daemon version is 20.10.5. The compose.yml, which is responsible for the solution's configuration prepared with Compose V3.7. The Swarm starts with the command:
docker stack deploy --with-registry-auth -c compose.yml <swarm_name>
The specifically Kafdrop service configured almost in the same way, as in your own compose file: docker-compose/kafka-kafdrop/docker-compose.yaml. Service has got only 1 instance and here its config:

  kafdrop:
    image: obsidiandynamics/kafdrop:latest
    networks:
      - confluent_network
    environment:
      - KAFKA_BROKERCONNECT=confluent-kafka1:9092,confluent-kafka2:19092,confluent-kafka3:29092
      - JVM_OPTS=-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify
      - SERVER_SERVLET_CONTEXTPATH=/kafdrop
    depends_on:
      - confluent-kafka1
      - confluent-kafka2
      - confluent-kafka3
    deploy:
      restart_policy:
        condition: on-failure

As I already mentioned above, we are using your image "as is", so, in general, it should contain everything for the correct work and other services aren't really able to influence the used libraries/dependencies within Kafdrop container. However, when the error occurs, I tried to start your docker compose file on my local box. My box is running under the macOS Monterey (12.6.4), for Docker related stuff I'm using Docker Desktop v.4.22.0 (117440). Your compose was started with the following command:
docker-compose -f docker-compose.yaml -p obsidiandynamics up
When you've requested for a log, I just run your compose file one more time and copied the log from kafdrop service; that's why it was so quick ;).

p.s. It looks like we are three, who experienced the same problem; at least I considering the thumb emoji from the @moein7 user like that ;).

from kafdrop.

I'm also seeing this issue.

from kafdrop.

Sorry for not being explicit.

I started with a working situation, then made this change:

-      JVM_OPTS: "-Xms16M -Xmx48M -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
+      JVM_OPTS: "-Xms16M -Xmx48M -Xss360K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"

That also worked.

from kafdrop.

To answer your question first: I would stick with an old version if a new one is available.

Kafdrop 4.0 is based on JDK 17 and SpringBoot 3, so you might need to update the libraries you're loading. To the best of my knowledge, we are not using any Kotlin out of the box, so I guess you're having a custom config that loads this.

from kafdrop.

Hi @OGudkov-TechHub and @pandamorphism, is it possible for you to provide the full log file for analysis? You may can export the container log and attach as a file. Thanks.

from kafdrop.

Hi @mehdihasan, sure, np! Here you are!

Full_Kafdrop4_docker_image_log_2023-10-16.log

from kafdrop.

Regarding the Kotlin dependency - which might be really misleading - @OGudkov-TechHub guessed right that, it is being used by one of the project dependency: kafka-protobuf-serializer. However, probably it is not the cause of the issue.

It looks more like a configuration issue - how the container is spinning up - with which configurations. It might be helpful for further investigation if you can share the run configurations, environment information and the commands you use to stat your containers.

I am asking this as this issue is not consistently occurring for all the Kafdrop users, instead only 2 of you. It suggests that the problem might be related to some specific conditions or configurations that are unique to your environments.

from kafdrop.

I compared the logs with what I get. An interesting difference is that the VM issues this warning:

OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.

That's in my log right after the ASCII-art.
After "No active profile set, falling back to 1 default profile: "default"", you get "Application run failed", while it proceeds this way for me:

  INFO 1 [           main] i.u.s.s.ServletContextImpl               : Initializing Spring embedded WebApplicationContext
  INFO 1 [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2042 ms
  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking truststore file kafka.truststore.jks
  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking keystore file kafka.keystore.jks
  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking properties file kafka.properties
  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking truststore file kafka.truststore.jks
  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking keystore file kafka.keystore.jks
  INFO 1 [           main] k.c.KafkaConfiguration                   : Checking properties file kafka.properties
  INFO 1 [           main] k.s.BuildInfo                            : Kafdrop version: 4.0.1, build time: 2023-11-02T13:01:08.419Z
  INFO 1 [           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 13 endpoint(s) beneath base path '/actuator'
  INFO 1 [           main] i.u.Undertow                             : starting server: Undertow - 2.3.8.Final
  INFO 1 [           main] o.x.Xnio                                 : XNIO version 3.8.8.Final
  INFO 1 [           main] o.x.n.NioXnio                            : XNIO NIO Implementation Version 3.8.8.Final
  INFO 1 [           main] o.j.t.Version                            : JBoss Threads version 3.5.0.Final
  INFO 1 [           main] o.s.b.w.e.u.UndertowWebServer            : Undertow started on port(s) 9000 (http)
  INFO 1 [           main] o.s.b.StartupInfoLogger                  : Started Kafdrop in 3.965 seconds (process running for 5.202)
  INFO 1 [  XNIO-1 task-2] i.u.s.s.ServletContextImpl               : Initializing Spring DispatcherServlet 'dispatcherServlet'
  INFO 1 [  XNIO-1 task-2] o.s.w.s.FrameworkServlet                 : Initializing Servlet 'dispatcherServlet'
  INFO 1 [  XNIO-1 task-2] o.s.w.s.FrameworkServlet                 : Completed initialization in 1 ms
  INFO 1 [  XNIO-1 task-8] k.c.BasicErrorController                 : errorAtts: {timestamp=2023-11-02T19:02:53.495+0000, status=404, error=Not Found, message=Not Found, path=/favicon.ico}

The missing VM warning is intriguing. Anyone an idea?

from kafdrop.

I think we have figured it out.

It appears it is due to the JVM thread stack size.

The JVM thread stack size primarily affects the depth of the call stack for each thread. A smaller stack size may lead to StackOverflowError exceptions If the application uses a large number of threads or has deeply nested method calls. This is exactly what we are facing here.

We need to update the stack size for each thread in the JVM, which is actually defined by -Xss option. Currently the value is 180K. Assigning 360K works fine for me. So, the changed docker compose file should be look like the following:

version: "2"
services:
  kafdrop:
    image: obsidiandynamics/kafdrop
    restart: "no"
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka:29092"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss360K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - "kafka"
  kafka:
    image: obsidiandynamics/kafka
    restart: "no"
    ports:
      - "2181:2181"
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: "INTERNAL://:29092,EXTERNAL://:9092"
      KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka:29092,EXTERNAL://localhost:9092"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
      KAFKA_INTER_BROKER_LISTENER_NAME: "INTERNAL"
      KAFKA_ZOOKEEPER_SESSION_TIMEOUT: "6000"
      KAFKA_RESTART_ATTEMPTS: "10"
      KAFKA_RESTART_DELAY: "5"
      ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL: "0"

@lpoultersky , @OGudkov-TechHub , @pandamorphism and @Bert-R can you please confirm if the above mentioned docker compose file (setting -Xss to 360K) works fine for you?

from kafdrop.

With that -Xss option, it still works for me

from kafdrop.

@Bert-R, did you mean it still works with -Xss value to 180K for you? Or, the issue still there though you set the -Xss value to 360K?

from kafdrop.

Ya! Both suppose to work.

In the first option, the JVM will use a default thread stack size. The default stack size can vary between different operating systems and JVM implementations. These defaults are typically chosen to provide a reasonable balance between memory usage and application performance for typical use cases. Most cases, relying on the default stack size should work well.

from kafdrop.