big-data-europe / docker-flink Goto Github PK

Flink is not published at the Docker Hub.

Please provide docker-compose v2 definition compatible with docker swarm and instructions on how to run it inside swarm.

docker-compose on 'master' branch should always use the last version of the image

README file does not include docker-compose snippet.

P.S. To have this first publish the image on the docker hub

Is this deprecated?
The last commit is 19 Feb 2019.
Is there a way to get the latest docker image?
Thx

Hi, thank you for the image. I found a bug and would like to share it with you

Steps to reproduce

1. in docker-compose.yml set

FLINK_NUM_TASK_SLOTS=12

2. then run

docker-compose up -d

3. finally, restart services

docker-compose restart

expected task slots 12
actual task slots 122

Possible solution

the problem place in worker/worker.sh
it can be fixed by replacing

sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: ${FLINK_NUM_TASK_SLOTS}/g" /usr/local/flink/conf/flink-conf.yaml

with

sed -i -e "s/taskmanager.numberOfTaskSlots: [0-9]\+/taskmanager.numberOfTaskSlots: ${FLINK_NUM_TASK_SLOTS}/g" /usr/local/flink/conf/flink-conf.yaml

i.e. by changing "1" to "[0-9]\+"

The platform release is already in 3 months. Need to provide sbt template or remove this statement from the README file.

I'm trying to setup a Flink cluster using docker run commands found in READE.

First I deploy the master using

$ docker run --name flink-master -h flink-master -e ENABLE_INIT_DAEMON=false -d bde2020/flink-master:0.10.1-hadoop2.7
$ docker logs flink-master
Starting Flink webclient
Missing streaming mode (batch|streaming) argument. Using 'batch'.
Starting jobmanager daemon on host flink-master.
Cluster started.

Then I deploy two workers

$ docker run --name flink-worker-1 --link flink-master:flink-master -e ENABLE_INIT_DAEMON=false -d bde2020/flink-worker:0.10.1-hadoop2.7
$ docker logs flink-worker-1
Missing streaming mode (batch|streaming). Using 'batch'.
Starting taskmanager daemon on host 80bb50f151cd.
$ docker run --name flink-worker-2 --link flink-master:flink-master -e ENABLE_INIT_DAEMON=false -d bde2020/flink-worker:0.10.1-hadoop2.7
$ docker logs flink-worker-2
Missing streaming mode (batch|streaming). Using 'batch'.
Starting taskmanager daemon on host 6cda68eb7512.

When I check the container status both workers seem to exit right after they are deployed.

$ sudo docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6cda68eb7512 bde2020/flink-worker:0.10.1-hadoop2.7 "/bin/bash /worker.sh" 7 seconds ago Exited (0) 6 seconds ago flink-worker-2
80bb50f151cd bde2020/flink-worker:0.10.1-hadoop2.7 "/bin/bash /worker.sh" 13 seconds ago Exited (0) 13 seconds ago flink-worker-1
c69c45ebe9a1 bde2020/flink-master:0.10.1-hadoop2.7 "/bin/bash /master.sh" 25 seconds ago Up 24 seconds 22/tcp, 6123/tcp, 8080/tcp flink-master

Is this the proper way to deploy the cluster? Should I do something else to keep the workers alive?

A simple edit to docker-compose.yaml needs to be done
ports: - "8081:8081"
That's just to make the dashboard accessible from the localhost.

At the moment the config is hardcoded as follows:

flinkmaster:
    image: bde2020/flink-master:0.10.1-hadoop2.7
    hostname: flink-master
    environment:
      - INIT_DAEMON_STEP=setup_flink
#    networks:
#      - flink
flinkworker:
    image: bde2020/flink-worker:0.10.1-hadoop2.7
    hostname: flink-worker
#    networks:
#     - flink
    links:
      - "flinkmaster:localhost"

The worker is connected to flinkmaster using hardcoded "localhost" host name (in master/slaves file?). The proper docker-compose.yml should look like:

flinkmaster:
    image: bde2020/flink-master:0.10.1-hadoop2.7
    environment:
      - INIT_DAEMON_STEP=setup_flink
flinkworker:
    image: bde2020/flink-worker:0.10.1-hadoop2.7
    links:
      - "flinkmaster"
    environment:
       - FLINK_MASTER=flinkmaster

Master and worker docker containers should dump the logs into stdout.

At the moment, logs are not available. For instance, in a case of failing flink-worker, there will be empty output in docker logs:

docker logs dockerflink_flinkworker_1 
Missing streaming mode (batch|streaming). Using 'batch'.
Starting taskmanager daemon on host flink-worker.

The tag of bde2020/flink-master and bde2020/flink-worker image does not exist anymore! Please update.