Comments (11)
I'm actually pretty interested in this topic (memory and cpu requests/limits for java apps in kubernetes) and very interested in your opinions. Currently, I've been following advice from jsravn from his experience building a kubernetes stack from scratch. He says:
-Xmx
to roughly half the size of the memory limits of the pod. There is a lot of memory overhead in hotspot. This value will depend on the application as well, so it takes some work to find the right amount of max heap to prevent an OOM kill.-XX:+AlwaysPreTouch
so we can more quickly ensure our max heap setting is reasonable. Otherwise we might only get OOM killed under load.- Never limit CPU on a JVM container, unless you really don’t care about long pause times. The CPU limit will cause hard throttling of your application whenever garbage collections happen. The right choice almost always is to use burstable CPU. This let’s the scheduler soft clamp the application if it uses too much CPU in relation to other applications running on the node, while also letting it use as much CPU as available on the machine.
I would be very interested in other people's experience and opinions.
from kubernetes-kafka.
Since using a JVM in containers is already somewhat scary
agreed :)
In the docs
Do you mean in the readme? I think it will always have to be tweaked per installation based on the load etc. In #44 (comment) I'm looking at how to minimize the footprint, for testing.
from kubernetes-kafka.
Great input. The first task I had to battle with was to select a JVM vendor+version. Am I wrong to assume that the choice there affects behavior with memory flags?
The default kafka image here uses Zulu because Confluent does so. I'm obviously not a lawyer but I think that anyone who builds their own images can use Oracle's server-jre.
Are there other options? Is openjdk:8-jre-slim a better option than Zulu? Or should we go straight for Java9 before we start tweaking, given the improvement in kafka SSL performance? I imagine that the java community has efforts underway to tweak JVMs for use in Docker, and even with Kubernetes memory limits. Haven't researched that topic though.
Now back to the JVM args... I battled with them around #49 (comment) and was quite unhappy about the outcome (hadn't seen jsravn's advice at that time). After that I didn't dare to merge anything with memory limits on broker pods. But I'm willing to un-conclude that :)
from kubernetes-kafka.
With regards to Kafka, doing whatever Confluent does is generally the best advice. Sadly, probably best to avoid Oracle for legal reasons and pick Zulu openJDK8. JDK 9 shows great promise, and is fun for experiments, and maybe individual apps in production, but I would not run critical infrastructure like kafka in prod on it until Confluent does.
I can only speak to tuning web apps in java in kubernetes, and I worry that Kafka is different enough to invalidate that experience.
from kubernetes-kafka.
Haven't read it yet, but https://dzone.com/articles/why-my-java-application-is-oomkilled could be relevant.
from kubernetes-kafka.
The results of #128 will be very interesting. There, for the metrics
container I've only enabled memory limit awareness. No other flags that affect memory use.
I'll keep a close watch on the Prometheus metric container_memory_usage_bytes{namespace="kafka",container_name="metrics"}
, to get a baseline for experiments with memory flags later.
from kubernetes-kafka.
@solsson You have a prometheus exporter for the image now? I had to extend you image to add one, pretty important in my setup. Thanks
from kubernetes-kafka.
I didn't add the exporter to the image. It's a separate container in the pod, added using kubectl patch, see #128. Did you find an exporter that plugins into kafka without JMX? I did a couple of experiments based on https://github.com/arnobroekhof/kafka-http-metrics-reporter, https://github.com/prepor/kafka-prometheus and https://github.com/prometheus/client_java/blob/master/simpleclient_dropwizard but I failed to get any useful results.
from kubernetes-kafka.
@solsson No, I currently use the JMX exporter. Is there a downside to using the JMX exporter? I'm not hugely experienced with the JVM.
Thanks,
On a side note, some of my consumers seem to lag behind when there is a lot of data ( 20k/sec ). However they aren't using much cpu or network. Obviously consumer lib etc could affect this, but on the actual Kafka side is there anything I should be monitoring to possible see whats slowing it down?
Thanks x 2
from kubernetes-kafka.
Is there a downside to using the JMX exporter?
@qrpike Ok, I think we're all ending up with JMX as the only viable alternative. As a sidecar it's relatively resource heavy, and before #128 I had failed to set a reasonable memory limit that didn't affect stability of the kafka pod.
some of my consumers seem to lag behind when there is a lot of data ( 20k/sec ). However they aren't using much cpu or network. Obviously consumer lib etc could affect this, but on the actual Kafka side is there anything I should be monitoring to possible see whats slowing it down?
This is a great topic and quite core to Kafka ops. I'm no expert but I'd like to see the community discuss this. I suggest you open a new issue with the stats you have. Please clarify if it is the cpu & network of consuming pods that is low. Also what metrics you have from Kafka pods and nodes.
from kubernetes-kafka.
See f9162a6 and don't hesitate to open new discussions.
from kubernetes-kafka.
Related Issues (20)
- Zookeeper properties file needs an empty line at the end of the file HOT 3
- Run JMX exporter as a Java Agent (how to?) HOT 1
- Pod, Service and Statefull pending
- Error connecting to node kafka-0.broker.kafka.svc.cluster.local:9092 HOT 1
- Error processing /etc/kafka/zookeeper.properties.scale-5.pzoo-0 HOT 5
- Can you tell me about 10 brokers in Kafka- config.yml File parameters log.retention.hours= -1 and log.retention.hours=168 What's the difference?
- Release v6.0.4 Seems to be a Breaking Release? HOT 6
- ZooKeeper produce a zombie processes HOT 4
- Error processing /etc/kafka/zookeeper.properties.scale-5.pzoo-1 HOT 5
- Can't produce/consume with outside brokers HOT 1
- [Question] Getting started but no resources created?
- upstream bug: zookeeper 3.5.7 leader election seriously broken HOT 1
- How do I specify my own volumeclass / volume mount locations?
- Zookeeper Init:Error "/etc/kafka-configmap/init.sh: No such file or directory"
- Issue on external service (Kafka) HOT 1
- Incompatible with newer kustomize/kubectl
- Quickstart is broken (v6.0.3) HOT 1
- Auto scale Kafka partitions HOT 1
- Unable to successfully start pods - CrashLoopBackOff error HOT 1
- ARM64 Images for Kafka JMX Prometheus Exporter
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kubernetes-kafka.