Comments (6)
jainhemant163
commented on May 28, 2024
2
Hii @amotl
I just figured out that:
I was missing out to run this command on the Linux Machine and was in the pre-assumption that I won't be requiring to run this command:
sudo sysctl -w vm.max_map_count=262144But, now I am able to replicate the complete article with cratedb on AWS EC2 instance with the above docker and docker-compose versions installed, as it is not supportable on the lower versions.
from cratedb-examples.
amotl
commented on May 28, 2024
Dear @jainhemant163,
thank you for sharing your outcome. Apologies that following the tutorial didn't work out of the box with you. We verified it works on Linux, macOS and Windows 10 using the software versions outlined at [2], so I am very curious why it wouldn't work on your end. With respect to Docker, I am using Docker 20.10.5 and docker-compose version 1.29.0 on my machine.
So, can I humbly ask you to share more details about the environments on both your workstation and the AWS EC2 instance(s) you are using? That is, operating system flavors, release versions and your Docker / Docker Compose versions? It would be nice if we could solve the problem together.
With kind regards,
Andreas.
[2] https://github.com/crate/cratedb-examples/blob/main/spikes/kafka-flink/README.rst#about
from cratedb-examples.
jainhemant163
commented on May 28, 2024
Hii @amotl
I tried setting up the latest version on the Amazon EC2 Instance Ubuntu machine for docker and docker-compose.
The command " docker-compose up -d " ran successfully but the cratedb and other container instances got destroyed up.
Below are the logs which I received on the AWS EC2 Instance machine:
root@ip-172-31-49-181:~/sandbox/kafka-flink-cratedb/cratedb-examples/spikes/kafka-flink# docker-compose up
Creating network "scada-demo" with driver "bridge"
Creating network "kafka-flink_default" with the default driver
Pulling kafka-zookeeper (confluentinc/cp-zookeeper:6.1.1)...
6.1.1: Pulling from confluentinc/cp-zookeeper
8f403cb21126: Pull complete
65c0f2178ac8: Pull complete
f531aa5f4b59: Pull complete
e8f004375cc7: Pull complete
62400f0f3d19: Pull complete
3b70c889979a: Pull complete
90d755e41546: Pull complete
5190561ca80d: Pull complete
e9740bd870ac: Pull complete
a65e45d3569e: Pull complete
Digest: sha256:dc04584a644b09ec00046a9acd352660b54cd358094a88945802a3444a392586
Status: Downloaded newer image for confluentinc/cp-zookeeper:6.1.1
Pulling kafka-broker (confluentinc/cp-kafka:6.1.1)...
6.1.1: Pulling from confluentinc/cp-kafka
8f403cb21126: Already exists
65c0f2178ac8: Already exists
f531aa5f4b59: Already exists
e8f004375cc7: Already exists
62400f0f3d19: Already exists
3b70c889979a: Already exists
90d755e41546: Already exists
5190561ca80d: Already exists
51978feb47e0: Pull complete
39e8c3cb0f91: Pull complete
Digest: sha256:b356b5138a8f31dceb9b62061a8e7862bfb653819fdb3469f1bba7bdb0db5b2b
Status: Downloaded newer image for confluentinc/cp-kafka:6.1.1
Pulling flink-jobmanager (flink:1.12)...
1.12: Pulling from library/flink
bd8f6a7501cc: Pull complete
44718e6d535d: Pull complete
efe9738af0cb: Pull complete
c647a8e650d3: Pull complete
4c209a1e5186: Pull complete
2ea976dfda62: Pull complete
041000bb441d: Pull complete
88bdee250f27: Pull complete
92f72c6d9548: Pull complete
f22995480563: Pull complete
777a2b4f35db: Pull complete
07fdc83dd943: Pull complete
Digest: sha256:3f91d579113e57be6d21b5869be1664f8085a3793af8df310cd51d78e03b2f7a
Status: Downloaded newer image for flink:1.12
Pulling cratedb (crate:4.5.0)...
4.5.0: Pulling from library/crate
2d473b07cdd5: Pull complete
926db4efe32a: Pull complete
2732c2d08b5d: Pull complete
b12306c186da: Pull complete
ba11d3730a19: Pull complete
b8bd594f0158: Pull complete
c1aac5f55c86: Pull complete
c6c800063847: Pull complete
Digest: sha256:cc871ce71e20d65c949482bf12204700366baaed664406b0e35e4c17a06cf88e
Status: Downloaded newer image for crate:4.5.0
Pulling start-dependencies (dadarek/wait-for-dependencies:)...
latest: Pulling from dadarek/wait-for-dependencies
3489d1c4660e: Pull complete
38737c1480a6: Pull complete
Digest: sha256:b38e28108dee0aad90984e176adcc014dbf31b528602640f6c9a280cdc7ce2f0
Status: Downloaded newer image for dadarek/wait-for-dependencies:latest
Creating kafka-flink_flink-jobmanager_1 ... done
Creating kafka-flink_kafka-zookeeper_1 ... done
Creating kafka-flink_cratedb_1 ... done
Creating kafka-flink_kafka-broker_1 ... done
Creating kafka-flink_flink-taskmanager_1 ... done
ERROR: for start-dependencies Container "00903382f4c7" is unhealthy.
ERROR: Encountered errors while bringing up the project.
----------------------------------------------------------------------------------------------------------------
cratedb_1
7e8d5baf3b90 confluentinc/cp-zookeeper:6.1.1 "/etc/confluent/dock…" 16 minutes ago Up 15 seconds (health: starting) 2181/tcp, 2888/tcp, 3888/tcp kafka-flink_kafka
Yes, if possible we can work together to replicate the same scenario and look around the issue with the alternative solutions to be discussed.
from cratedb-examples.
amotl
commented on May 28, 2024
Dear @jainhemant163,
thank you for reporting back that you have been able to make it work. Congratulations, enjoy!
@carlotasoto, @carlotas19: Maybe you can extend this line on the tutorial to read like:
This tutorial uses Kafka 6.1.1, Apache Flink 1.12, and CrateDB 4.5.0 on a macOS system. It is important to use recent versions of Docker like Docker 20.10.5 and Docker Compose 1.29.0 to invoke the infrastructure sandbox successfully.
Adding these details to the bullet point 2 of the preliminary notes section would also fit, I believe.
With kind regards,
Andreas.
from cratedb-examples.
carlotas19
commented on May 28, 2024
Excellent suggestion @amotl. I have included the remark about the Docker / Docker Compose versions in the original post.
And thank you @jainhemant163 to report to us and help us improve our tutorial!
Best,
Carlota
from cratedb-examples.
amotl
commented on May 28, 2024
Thank you Carlota and Jain! I am closing this now.
from cratedb-examples.
Related Issues (20)
- AutoML: Test harness trips when installing `catboost` on macOS with Python 3.11, works with Python 3.10
- Add examples on how to use CrateDB with Polars in Write and Read operations.
- Improve tutorials about Apache Superset HOT 1
- Dependency installation fails on Google Colab for `cratedb-vectorstore-rag-openai-sql.ipynb`
- LLM example `document_loader.ipynb` fails HOT 4
- ODBC on Windows, pyodbc, and unixODBC
- [Testing] Demonstrate CrateDB test layers with parameterization
- LangChain: `cratedb_rag_customer_support.ipynb` trips with `CellTimeoutError` HOT 2
- AutoML: CI trips with `ValueError: Input contains NaN.` HOT 7
- AutoML: CI trips with `CellTimeoutError` HOT 2
- Issue with requirements.txt HOT 3
- CI job trips on Apache Superset 2.x: `AttributeError: module 'flask.json' has no attribute 'JSONEncoder'` HOT 5
- [Tech Writing] CrateDB for Pythonistas with SQLAlchemy HOT 1
- [Java] crate-java-testing
- Issue with time series forecasting with pycaret notebook HOT 2
- Time Series: Modernize notebooks to use recent versions of pandas and SQLAlchemy
- Apache Kafka Streaming with PyFlink
- Use Python 3.10 on CI, at least for Jupyter Notebooks HOT 1
- RAG: Problems resolving dependencies on Google Colab HOT 3
- TSML: Error in `timeseries-anomaly-detection.ipynb` HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cratedb-examples.