big-data-europe / docker-flink Goto Github PK
View Code? Open in Web Editor NEWApache Flink docker image
Apache Flink docker image
Master and worker docker containers should dump the logs into stdout.
At the moment, logs are not available. For instance, in a case of failing flink-worker, there will be empty output in docker logs:
docker logs dockerflink_flinkworker_1
Missing streaming mode (batch|streaming). Using 'batch'.
Starting taskmanager daemon on host flink-worker.
Is this deprecated?
The last commit is 19 Feb 2019.
Is there a way to get the latest docker image?
Thx
Hi, thank you for the image. I found a bug and would like to share it with you
FLINK_NUM_TASK_SLOTS=12
docker-compose up -d
docker-compose restart
expected task slots 12
actual task slots 122
the problem place in worker/worker.sh
it can be fixed by replacing
sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: ${FLINK_NUM_TASK_SLOTS}/g" /usr/local/flink/conf/flink-conf.yaml
with
sed -i -e "s/taskmanager.numberOfTaskSlots: [0-9]\+/taskmanager.numberOfTaskSlots: ${FLINK_NUM_TASK_SLOTS}/g" /usr/local/flink/conf/flink-conf.yaml
i.e. by changing "1" to "[0-9]\+"
docker-compose on 'master' branch should always use the last version of the image
Flink is not published at the Docker Hub.
At the moment the config is hardcoded as follows:
flinkmaster:
image: bde2020/flink-master:0.10.1-hadoop2.7
hostname: flink-master
environment:
- INIT_DAEMON_STEP=setup_flink
# networks:
# - flink
flinkworker:
image: bde2020/flink-worker:0.10.1-hadoop2.7
hostname: flink-worker
# networks:
# - flink
links:
- "flinkmaster:localhost"
The worker is connected to flinkmaster using hardcoded "localhost" host name (in master/slaves file?). The proper docker-compose.yml should look like:
flinkmaster:
image: bde2020/flink-master:0.10.1-hadoop2.7
environment:
- INIT_DAEMON_STEP=setup_flink
flinkworker:
image: bde2020/flink-worker:0.10.1-hadoop2.7
links:
- "flinkmaster"
environment:
- FLINK_MASTER=flinkmaster
Please provide docker-compose v2 definition compatible with docker swarm and instructions on how to run it inside swarm.
I'm trying to setup a Flink cluster using docker run commands found in READE.
First I deploy the master using
$ docker run --name flink-master -h flink-master -e ENABLE_INIT_DAEMON=false -d bde2020/flink-master:0.10.1-hadoop2.7
$ docker logs flink-master
Starting Flink webclient
Missing streaming mode (batch|streaming) argument. Using 'batch'.
Starting jobmanager daemon on host flink-master.
Cluster started.
Then I deploy two workers
$ docker run --name flink-worker-1 --link flink-master:flink-master -e ENABLE_INIT_DAEMON=false -d bde2020/flink-worker:0.10.1-hadoop2.7
$ docker logs flink-worker-1
Missing streaming mode (batch|streaming). Using 'batch'.
Starting taskmanager daemon on host 80bb50f151cd.
$ docker run --name flink-worker-2 --link flink-master:flink-master -e ENABLE_INIT_DAEMON=false -d bde2020/flink-worker:0.10.1-hadoop2.7
$ docker logs flink-worker-2
Missing streaming mode (batch|streaming). Using 'batch'.
Starting taskmanager daemon on host 6cda68eb7512.
When I check the container status both workers seem to exit right after they are deployed.
$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6cda68eb7512 bde2020/flink-worker:0.10.1-hadoop2.7 "/bin/bash /worker.sh" 7 seconds ago Exited (0) 6 seconds ago flink-worker-2
80bb50f151cd bde2020/flink-worker:0.10.1-hadoop2.7 "/bin/bash /worker.sh" 13 seconds ago Exited (0) 13 seconds ago flink-worker-1
c69c45ebe9a1 bde2020/flink-master:0.10.1-hadoop2.7 "/bin/bash /master.sh" 25 seconds ago Up 24 seconds 22/tcp, 6123/tcp, 8080/tcp flink-master
Is this the proper way to deploy the cluster? Should I do something else to keep the workers alive?
The platform release is already in 3 months. Need to provide sbt template or remove this statement from the README file.
A simple edit to docker-compose.yaml needs to be done
ports: - "8081:8081"
That's just to make the dashboard accessible from the localhost.
README file does not include docker-compose snippet.
P.S. To have this first publish the image on the docker hub
The tag of bde2020/flink-master and bde2020/flink-worker image does not exist anymore! Please update.
(docker-compose)โ docker-flink git:(master) docker-compose up
Creating dockerflink_flinkmaster_1
Creating dockerflink_flinkworker_1
Attaching to dockerflink_flinkmaster_1, dockerflink_flinkworker_1
flinkmaster_1 | Starting Flink webclient
flinkmaster_1 | Missing streaming mode (batch|streaming) argument. Using 'batch'.
flinkmaster_1 | Starting jobmanager daemon on host flink-master.
flinkmaster_1 | Cluster started.
flinkworker_1 | Missing streaming mode (batch|streaming). Using 'batch'.
flinkworker_1 | Starting taskmanager daemon on host flink-worker.
dockerflink_flinkworker_1 exited with code 0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.