Comments (5)
agemooij
commented on May 30, 2024
1
Hi @eschizoid . EKS support is mostly a matter of updating the install scripts. The sbt plugin just creates images and pushes them to a configurable Docker registry. The kubectl-cloudflow plugin can deploy Cloudflow apps to any cluster running the Cloudflow operator. So all we are missing for EKS support is an installation script that supports it.
I am not aware of any major differences when it comes to EKS and GKE so very likely the install script only needs some minor adjustments to be compatible with EKS. Perhaps you could try to install cloudflow on EKS and let us know where it breaks?
from cloudflow.
eschizoid
commented on May 30, 2024
Hey @agemooij, I would be more than happy to work on this. Before that, I have a couple of questions for you :)
So all we are missing for EKS support is an installation script that supports it.
By that, you mean a script that provisions an initial EKS cluster similar to this script https://github.com/lightbend/cloudflow/blob/master/installer/create-cluster-gke.sh?
from cloudflow.
eschizoid
commented on May 30, 2024
@agemooij let me know what you think about #65
It seems that the zookeper and kafka pods are not coming up correctly when running on EKS:
$ kubectl get pods -n lightbend
NAME READY STATUS RESTARTS AGE
cloudflow-flink-flink-operator-7b74f75745-lsvqm 1/1 Running 0 47s
cloudflow-nfs-fdp-nfs-6dff799c57-rgrxp 1/1 Running 0 15s
cloudflow-operator-dc8867b7-t677j 1/1 Running 0 18s
cloudflow-sparkoperator-fdp-sparkoperator-5f45cfb497-t649k 1/1 Running 0 39s
cloudflow-strimzi-zookeeper-0 0/2 Pending 0 15s
cloudflow-strimzi-zookeeper-1 0/2 Pending 0 15s
cloudflow-strimzi-zookeeper-2 0/2 Pending 0 15s
strimzi-cluster-operator-5d7d946c6d-dpcpt 1/1 Running 0 43s
$ kubectl describe pod cloudflow-strimzi-zookeeper-2 -n lightbend`
Name: cloudflow-strimzi-zookeeper-2
Namespace: lightbend
Priority: 0
Node: <none>
Labels: app.kubernetes.io/managed-by=helm
app.kubernetes.io/name=cloudflow-operator
app.kubernetes.io/part-of=cloudflow
app.kubernetes.io/version=33-7703ce5
cloudflow.lightbend.com/build-number=33-7703ce5
cloudflow.lightbend.com/release-version=1.2.0
controller-revision-hash=cloudflow-strimzi-zookeeper-77dd64856c
statefulset.kubernetes.io/pod-name=cloudflow-strimzi-zookeeper-2
strimzi.io/cluster=cloudflow-strimzi
strimzi.io/kind=Kafka
strimzi.io/name=cloudflow-strimzi-zookeeper
Annotations: kubernetes.io/psp: eks.privileged
strimzi.io/cluster-ca-cert-generation: 0
strimzi.io/generation: 0
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/cloudflow-strimzi-zookeeper
Containers:
zookeeper:
Image: strimzi/kafka:0.13.0-kafka-2.3.0
Port: 9404/TCP
Host Port: 0/TCP
Command:
/opt/kafka/zookeeper_run.sh
Limits:
memory: 3Gi
Requests:
memory: 1Gi
Liveness: exec [/opt/kafka/zookeeper_healthcheck.sh] delay=15s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [/opt/kafka/zookeeper_healthcheck.sh] delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
ZOOKEEPER_NODE_COUNT: 3
ZOOKEEPER_METRICS_ENABLED: true
STRIMZI_KAFKA_GC_LOG_ENABLED: true
DYNAMIC_HEAP_FRACTION: 0.75
DYNAMIC_HEAP_MAX: 2147483648
ZOOKEEPER_CONFIGURATION: autopurge.purgeInterval=1
tickTime=2000
initLimit=5
syncLimit=2
Mounts:
/opt/kafka/custom-config/ from zookeeper-metrics-and-logging (rw)
/var/lib/zookeeper from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from cloudflow-strimzi-zookeeper-token-v9zng (ro)
tls-sidecar:
Image: strimzi/kafka:0.13.0-kafka-2.3.0
Ports: 2888/TCP, 3888/TCP, 2181/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/opt/stunnel/zookeeper_stunnel_run.sh
Liveness: exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3
Readiness: exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
TLS_SIDECAR_LOG_LEVEL: notice
ZOOKEEPER_NODE_COUNT: 3
Mounts:
/etc/tls-sidecar/cluster-ca-certs/ from cluster-ca-certs (rw)
/etc/tls-sidecar/zookeeper-nodes/ from zookeeper-nodes (rw)
/var/run/secrets/kubernetes.io/serviceaccount from cloudflow-strimzi-zookeeper-token-v9zng (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-cloudflow-strimzi-zookeeper-2
ReadOnly: false
zookeeper-metrics-and-logging:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: cloudflow-strimzi-zookeeper-config
Optional: false
zookeeper-nodes:
Type: Secret (a volume populated by a Secret)
SecretName: cloudflow-strimzi-zookeeper-nodes
Optional: false
cluster-ca-certs:
Type: Secret (a volume populated by a Secret)
SecretName: cloudflow-strimzi-cluster-ca-cert
Optional: false
cloudflow-strimzi-zookeeper-token-v9zng:
Type: Secret (a volume populated by a Secret)
SecretName: cloudflow-strimzi-zookeeper-token-v9zng
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: dedicated=StrimziKafka:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 48s (x8 over 5m50s) default-scheduler 0/3 nodes are available: 3 node(s) didn't match node selector.
from cloudflow.
eschizoid
commented on May 30, 2024
hey @agemooij after deploying the application sensor-data-scala it seems that i need kafka up and running:
$ kubectl get pods -n sensor-data-scala
NAME READY STATUS RESTARTS AGE
sensor-data-scala-file-ingress-bb4f6f96-bt5xj 0/1 Pending 0 13m
sensor-data-scala-http-ingress-557ddb966f-spd8s 1/1 Running 0 13m
sensor-data-scala-invalid-logger-5b8f8b5b67-4r74f 0/1 CrashLoopBackOff 7 13m
sensor-data-scala-merge-749b695f59-c2br8 1/1 Running 0 13m
sensor-data-scala-metrics-55f697f64c-8sxst 1/1 Running 0 13m
sensor-data-scala-rotor-avg-logger-58d6ff8455-zsks5 0/1 Pending 0 13m
sensor-data-scala-rotorizer-6f84fb4479-ptv2r 1/1 Running 0 13m
sensor-data-scala-valid-logger-78d4d94875-7k4cf 0/1 Pending 0 13m
sensor-data-scala-validation-6b8cf4fbd7-ffhsw 1/1 Running 0 13m
$ kubectl logs -f sensor-data-scala-invalid-logger-5b8f8b5b67-4r74f -n sensor-data-scala
...
[ERROR] [12/08/2019 23:02:36.453] [akka_streamlet-akka.actor.default-dispatcher-5] [akka://akka_streamlet/system/kafka-consumer-1] Failed to construct kafka consumer
akka.actor.ActorInitializationException: akka://akka_streamlet/system/kafka-consumer-1: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:202)
at akka.actor.ActorCell.create(ActorCell.scala:696)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:547)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:569)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:293)
at akka.dispatch.Mailbox.run(Mailbox.scala:228)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:624)
at akka.kafka.ConsumerSettings$.createKafkaConsumer(ConsumerSettings.scala:178)
at akka.kafka.ConsumerSettings$.$anonfun$apply$3(ConsumerSettings.scala:94)
at akka.kafka.ConsumerSettings.createKafkaConsumer(ConsumerSettings.scala:573)
at akka.kafka.internal.KafkaConsumerActor.preStart(KafkaConsumerActor.scala:403)
at akka.actor.Actor.aroundPreStart(Actor.scala:550)
at akka.actor.Actor.aroundPreStart$(Actor.scala:550)
at akka.kafka.internal.KafkaConsumerActor.aroundPreStart(KafkaConsumerActor.scala:202)
at akka.actor.ActorCell.create(ActorCell.scala:676)
... 7 more
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:90)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:49)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:721)
... 16 more
Anyways, I feel I am really close getting the app running on EKS, just need some suggestions on how to solve the kafka strimzi installation. Thanks again!
from cloudflow.
eschizoid
commented on May 30, 2024
Closed by #65
from cloudflow.
Related Issues (20)
- Publish the CLI to Krew
- Redact and redirect Console references
- Configure command breaks akka server streamlets
- Make it possible to mount configmaps.
- Docs are missing a step to install CF CRD to be able to install CF from helm charts since v2.2.2 HOT 2
- CloudflowLocalRunnerPlugin adds log4j:log4j:1.2.7 to test dependencies HOT 1
- Multiple kafka clusters not working HOT 4
- Support logback for local runner in place of log4j
- Kafka brokers not reconnecting HOT 1
- The apiextensions.k8s.io/v1beta1 API version of CustomResourceDefinition is no longer served as of v1.22. HOT 6
- Add the ability to declare k8s pod `tolerations` that allow (but do not require) the pods to schedule onto nodes with matching taints HOT 1
- Cloudflow CRD not found issue HOT 2
- Support for Java 17
- Support more recent versions of Kubernetes HOT 1
- Docs need attention for Protobuf in v2.3.1 HOT 5
- Docs need attention for: "Download and install the Cloudflow CLI"
- Cloudflow CLI: incomplete pods status on deployment (1/0)
- Checkpointing configurations are not taking effect for flink streamlet HOT 1
- Communication on deprecation of cloudflow
- create the SensorDataJsonSupport object
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudflow.