Code Monkey home page Code Monkey logo

flink-k8s's Introduction

在Kubernetes(K8s)上搭建基于Zookeeper的Flink高可用集群(HA)


版本

  • Flink-1.7.2
  • Hadoop-2.6.0
  • Docker 18.03.1-ce
  • K8s v1.10.4
  • 系统-CentOS 7

非HA搭建

该过程Flink官网有提供,也比较简单,可以点击查看,也可以参考我这里的配置文件

Flink HA搭建

官方没有提供在K8s上搭建基于ZK的HA的镜像的脚本文件,所以我们需要自己修改构建Flink镜像的相关配置。我们可以 参考/借用 官方提供Flink非HA的构建脚本。根据需要搭建的版本,我们以这个目录作为我们添加HA配置的基础。

  • 开始操作:

修改该目录下的docker-entrypoint.sh,添加如下配置(可以做一下非空判断,下面就直接echo了):

echo "high-availability: ${HIGH_AVAILABILITY}" >> "${CONF_FILE}"
echo "high-availability.zookeeper.quorum: ${HIGH_AVAILABILITY_ZOOKEEPER_QUORUM}" >> "${CONF_FILE}"
echo "high-availability.zookeeper.path.root: ${HIGH_AVAILABILITY_ZOOKEEPER_PATH_ROOT}" >> "${CONF_FILE}"
echo "high-availability.cluster-id: ${HIGH_AVAILABILITY_CLUSTER_ID}" >> "${CONF_FILE}"
echo "high-availability.storageDir: ${HIGH_AVAILABILITY_STORAGEDIR}" >> "${CONF_FILE}"

修改后的docker-entrypoint.sh可以参考我这里的

然后通过该目录下的Dockerfile文件构建属于自己的镜像(这里就过多介绍构建过程啦),我构建出来的镜像名:leen/flink-ha:v1,在hub.docker上可以拉取到

然后部署用这两个配置文件执行下面命令就可以搭建Flink HA集群了

kubectl apply -f jobmanager-statefulset.yaml -f taskmanager-deployment.yaml

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.