antoinemeheut / rokc-mk1 Goto Github PK

The purpose of this project is to build a Kubernetes cluster on a couple of Raspberry Pi to run Openshift, deploy applications and work on security to improve and share my knowledge on the Sec of DevSecOps.

Report Bug · Request Feature

• About the Project
  • My goals
  • Features
  • Feedback
  • Experience sharing
• Hardware
  • Shopping list
  • Why not Intel
• Software
  • Clone this repo
  • Setup SD Cards
  • Install flash
  • Start your Raspberry pi
  • Configure the network
  • How to Connect to a Raspberry
• Cluster creation with Kubernetes
  • Install Ansible 2.9
  • Create the hosts file
  • Setup your pis system and configuration
    • Installing sshpass
    • Launching your Pis configuration
  • Install Kubernetes, star your master and nodes
  • Some useful commands on kubernetes
    • Install Kubectl
    • Configure Kubectl
    • Kubectl commands
  • Deploy default applications
    • Create a simple anonymous user
    • Download Kubernetes dashboard
    • Launch the proxy for accessing the dashboard
    • Access to Kuberbetes dashboard
• Openshift installation
  • Compiling OpenShift for ARM
    • Check out OpenShift Origin
• Other scripts
  • Reboot your cluster
  • Halt your cluster
• Roadmap
• Contributing
• License
• Contact
• Acknowledgements

This project is largely based on the :

• great repository made by Roland Huß: Project31/ansible-kubernetes-openshift-pi3.
• great repository made by SitoCH: rpi-kubernetes-cluster.
• great repository made by StefanScherer: flash.

Their work was very inspiring and allowed me to learn and above all helped me a lot to achieve my goal of having a private demonstration cloud equipped for my dual Openshit and Kubernetes installation purpose.

• Create a private cloud cluster from 4 Raspberry Pi
• Connect my MacBook to this cluster to manage and monitor it
• Create an arm Openshift image
• Install Kubernetes and Openshift with Ansible
• Deploy applications on this cluster
• Being able to move this cluster, use it for demos and therefore : all works in environments where WiFi and network are not accessible to the cluster
• Share my knowledge and recipes acquired

• Hardware pattern
• Use a bash script to prepare Raspberry Pi SD Card with Hypriot
• Explain Pi networking from a Freebox
• Use a bash script to prepare and start a running cluster with a few Ansible playbook
• Install Docker
• Install Kubernetes
• Create an arm Openshift image FAILED
• Cluster aware storage available out of the box using GlusterFS
• Dashboard and Heapster deployed by default along with Traefik

• Building the raspberry cluster is easy, flashing SD cards with Hypriot makes it really fast and makes booting machines very fast
• Once you get to this point, your cluster is not quite active and this is where the really complicated things start
• The bash scripts that I built and the Ansible playbooks that I used from projects that inspired me and that I modified are really good, but time goes by and the structuring evolutions of the versions of Docker, Kubernetes, Kubeadm and others make it necessary to update the playbooks
• We must therefore be aware that being DevSecOps is also having to maintain these playbooks and therefore consider them as a real computer application that evolves with the changes that occur in its environment
• Building infrastructure like code and rolling out IT applications or microservices continuously involves thinking of building and deploying scripts as code
• It's therefore necessary to apply to this code all the good development practices and permanently manage versions that can be used according to the roadmap of the products used

• You may need to adapt the playbooks and scripts from this project to create your own cluster
• The way I do it's simple, I open a terminal to run scripts and playbooks, at the same time I open the project in an IDE and adapt the code of the project when necessary
• To save time in problem analysis, you can add to each ansible command the --vvv option, which allows you to have verbose mode

Here is the first part of a shopping list for a Raspberry Pi 3 cluster, with links (not affiliated) to references that I have used. I let you search for other equipment and especially how to buy this equipment according to your favorite stores.

This cluster must be able to operate without a network other than the local loop constituted by the cluster itself and the macbook which is connected to it, this allows the cluster to be taken for demonstrations in environments where the network is closed for reasons of security.

The router is not compulsory, it allows demonstrations outside your home.

Below is a link to a more powerful configuration based on an Intel® architecture. I started with Raspberry I had already bought for other projects and I must admit that I had some problems because of the ARM architecture, some software that I use are often built in priority for Intel® type architectures.

SBC de type Intel®

git clone https://github.com/AntoineMeheut/Rokc.git Rokc
cd Rokc

A few preliminary words on Hypriot's flash.

It's a project which allows to burn images of the Linux kernels, already ready, which embed a version of Docker, all for arm.

These images are stable and light which is an advantage to have the maximum possible space on the SD card of each Raspberry for future application deployments. For the moment there are no HDDs connected to the cluster.

You will quickly understand the advantages of using such a project, this allows you to have formatted and correctly configured cards for the cluster. Because flash has a number of options, which I let you discover to customize the flashed bone.

I will still detail the options used here to flash the cards.

• -- ssid : your wifi ssid, to allow the pi to connect to wifi
• -- password : your wifi password
• -- version : Hypriot version, something like 1.7.1 or higher, for this project I used 1.7.1

Flash will flash your SD Card, if you don't have it on your computer.

Download the appropriate version for Linux or Mac with this command

curl -LO https://github.com/hypriot/flash/releases/download/2.5.0/flash

chmod +x flash

sudo mv flash /usr/local/bin/flash

In the scripts directory, you will use

./0-flash-4sd-cards.sh

Run this script, it's configured to use Hypriot and flash 4 SD cards. If you need to flash more cards, you just need to change the length of a loop in the script.

At the end of the execution of this script, your 4 SD cards will be ready to be inserted in the Raspberry Pi.

It's not yet a private cloud cluster, but what a joy to finally start your assembly!

Well done! We are not yet deploying thousands of microservices, but we already have a nice sound of fans.

To continue it will be necessary to assign fixed DHCP addresses to each of the Raspberries. For that nothing very complicated, you can use the management console of your Internet provider router.

For me it's : http://mafreebox.freebox.fr/index.php

I'm not using a WLAN router, because when I'm at home, I benefit from the box router of my access provider. When I move my cluster for demos, I do not connect it to internet, I reconfigure the addresses in the host file and if necessary I run this script.

./1-cleanup-known-hosts.sh

The addresses I have chosen are:

Before going further you should note that it's possible to access nodes from the MacBook using ssh with user: pirate and password: hypriot

ssh [email protected]

On a MacBook if it's not already installed you will have to execute the following commands, assuming that you have already installed brew on your computer.

brew install ansible

cp hosts.example hosts
vi hosts

• pis contains all members of your cluster where one is marked as "master" in the field host_extra. This group will be added to every node in its /etc/hosts. It's important that one host is marked as "master", since the playbooks rely on this host alias for accessing the API server.
• master IP address of the Master
• nodes All nodes which are not Master
• volumes Volumes to create on GlusterFS

Example

[pis]
192.168.1.10 name=n0 host_extra="master"
192.168.1.11 name=n1
192.168.1.12 name=n2
192.168.1.13 name=n3

[master]
192.168.1.10

[nodes]
192.168.1.11
192.168.1.12
192.168.1.13

[volumes]
volume-1
volume-2
volume-3
volume-4
volume-5

To avoid sshpass Error you will have to install:

brew install http://git.io/sshpass.rb

The ssh password: hypriot

ansible-playbook -k -i hosts 2-setup.yml

ansible-playbook -k -i hosts 3-kubernetes.yml

And now a good cup of coffee, because it will take a little time.

brew install kubernetes-cli

You will have to connect in ssh to the master node, than copy the config file that is in ~/.kube to your Mac abd then install this file.

mkdir ~/.kube
vim ~/.kube/config

and copy the config file from master node to this file.

Now that the cluster is properly deployed and started, you want to know if it works!

Nothing simpler, type this command on your computer or after an ssh on one of the nodes of your cluster.

kubectl get nodes

This command is also useful for checking the configuration of your cluster and checking the url on which Kubernetes is configured.

kubectl cluster-info

If you want to know what is running on your cluster, use this command.

kubectl -n kube-system get pod

Once that the cluster is up and running this playbook will deploy GlusterFS endpoints, Heapster, Dashboard and Traefik:

ansible-playbook -k -i hosts 4-applications.yml

For testing we can do that.

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

Attention use RBAC if you want to secure your cluster access

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml

kubectl proxy

http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/node?namespace=default

git clone https://github.com/openshift/origin

git checkout v3.11.0

However, please note that the cross compile build only works on current master (because of this introduced lately). So the step above is probably best used when on v1.1.5 or later.

Attention : After many tries, I was unable to compile Openshift to make it work on ARM. So I decided to repernser this project to make it work on Intel type cards.

For convenience useful deployments can be found in the directory manual-deployments, please note that you must install the Registry in order to use other manual deployments because the images will be centrally cached.

This container installs a Docker registry that acts as a pass-trough cache, this way only the first node will download an image from Internet and all the other requests will be served from "inside" the cluster.

kubectl --kubeconfig run/admin.conf create -f manual-deployments/registry/registry.yml

Simple container that deploys on 3 nodes a website that prints it's container ID.

Thanks to Traefik you can access it on http://master-node/whoami/

kubectl --kubeconfig run/admin.conf create -f manual-deployments/whoami/whoami.yml

MySQL database server with persistent storage.

kubectl --kubeconfig run/admin.conf create -f manual-deployments/mysql/mysql.yml

If you need to reboot your cluster you can use this script, Ansible will take care of asking you to ask each Raspberry in your cluster to restart.

ansible pis -i hosts --become --args "/sbin/reboot" --background 30 --forks 4 --user pi

The operating system of raspberry pis supports very well a shutdown by power cut. It still leaves me perplexed and if you are like me, use this script, ansible will ask each of the operating systems in your cluster to stop, then you can cut the power.

ansible pis -i hosts --become --args "/sbin/halt" --forks 4 --user pi

See the Project for a list of proposed features (and known issues).

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Distributed under the MIT License. See LICENSE for more information.

If you want to contact me just clic

Project Link: https://github.com/AntoineMeheut/Rokc

• Roland Huß for doing the heavy lifting and creating a project ready to use and easy to deploy.

• Lucas Käldström for porting Kubernetes to the Raspberry Pi / ARM.

• SitoCH for doing a great job of clarifying ansible playbooks, adding Traefik installation and manual deployments

• StefanScherer for doing a great job on Hypriot and flash, which really save a lot of time