Code Monkey home page Code Monkey logo

ceph-salt's Introduction

Ceph cluster deployment

Salt states for Ceph cluster deployment.

Support for:

  • Ceph MON
  • Ceph OSD
  • Ceph MDS
  • Ceph Clients
  • Ceph REST API

Details:

  • Support for Ceph multi-environment deployment from one salt master node.
  • Deploy any number of MONs, OSDs or MDS services. Also, those states could be used to add new nodes after a cluster is created.
  • Support to select which disks are OSDs or Journals.
  • Support for cluster and public network.

Those states are tested on Ubuntu 14.04 with Ceph Hammer release and Salt v2016.3.2.

Vagrant

If you want to test this deployment on your local machine inside VMs, the easiest way is to use Vagrant with VirtualBox provider. All you need is to go inside vagrant directory and run:

cd vagrant && vagrant up

This will bring up 3 VMs, one master, and 3 minion nodes. Ceph will be deployed on all three nodes. Also, those VMs will have two additional network interfaces to emulate public and cluster network for Ceph and three additional drives attached to them. Two will be used for OSDs and one for a journal.

Test the connectivity between master and minions:

vagrant ssh master
sudo salt -G 'environment:VAGRANT' test.ping

If everything is OK you can proceed with the Ceph deployment step: https://github.com/komljen/ceph-salt#deployment

Local environment

First, you need Salt master and minions installed and running on all nodes and minions keys should be accepted. The easiest way to install SaltStack is using bootstrap script:

Master:

curl -L https://bootstrap.saltstack.com | sudo sh -s -- -M -g https://github.com/saltstack/salt.git git v2016.3.2

Minions:

curl -L https://bootstrap.saltstack.com | sudo sh -s -- -g https://github.com/saltstack/salt.git git v2016.3.2

Master configuration

On the master node you need to include additional options. Edit master config file /etc/salt/master. Replace <USER> with username where this repository will be cloned:

file_recv: True
file_roots:
  base:
    - /home/<USER>/ceph-salt/salt
    - /var/cache/salt/master/minions
pillar_roots:
  base:
    - /home/<USER>/config
    - /home/<USER>/ceph-salt/pillar
worker_threads: 10
hash_type: sha256
jinja_trim_blocks: True
jinja_lstrip_blocks: True

New options will make sure that minions can send files to the master and other minions to be able to get those files. Also here you can change where your salt states and config files are located. Salt master restart is required:

sudo service salt-master restart

Minions configuration

On all minion nodes, you need to edit the configuration file. Edit minion config file /etc/salt/minion. Replace <ENV_NAME> and master IP address to match with your environment:

master: 192.168.33.10
hash_type: sha256
grains:
  environment: <ENV_NAME>

Salt minion restart is required:

sudo service salt-minion restart

NOTE: To add new Ceph environment just install minions and choose new environment name!

Connection check

On master node accept all minions with:

sudo salt-key -A

Now all minions are connected and you should be able to send any command to a particular environment. Examples:

sudo salt -G 'environment:PROD' test.ping
sudo salt -G 'environment:STAGE' test.ping

If everything is fine clone this git repository on the master node. Use the same user you specified in master configuration file:

git clone https://github.com/komljen/ceph-salt.git -b master

Copy configuration files for each environment except top.sls file:

mkdir -p ~/config
cp ~/ceph-salt/pillar/environment-EXAMPLE.sls ~/config/environment-<ENV_NAME>.sls
cp ~/ceph-salt/pillar/ceph-EXAMPLE.sls ~/config/ceph-<ENV_NAME>.sls
cp ~/ceph-salt/pillar/top.sls ~/config/top.sls

Edit ~/config/top.sls file and replace ENV_NAME with your environment:

  'environment:<ENV_NAME>':
    - match: grain
    - environment-<ENV_NAME>
    - ceph-<ENV_NAME>

If you have more environments add it here. Example:

  'environment:PROD':
    - match: grain
    - environment-PROD
    - ceph-PROD

  'environment:STAGE':
    - match: grain
    - environment-STAGE
    - ceph-STAGE

Configuration options

Edit ~/config/environment-<ENV_NAME>.sls file to match with your environment. For node names use hostnames (not FQDN):

nodes:
  master:
    roles:
      - ceph-osd
      - ceph-mon
      - ceph-mds
      - ceph-client
      - ceph-rest-api
    osds:
      sdc:
        journal: sdb
      sdd:
        journal: sdb
  node01:
    roles:
      - ceph-osd
      - ceph-mon
      - ceph-mds
    osds:
      sdc:
        journal: sdb
      sdd:
        journal: sdb
  node02:
    roles:
      - ceph-osd
      - ceph-mon
      - ceph-mds
    osds:
      sdc:
        journal: sdb
      sdd:
        journal: sdb

Now edit ~/config/ceph-<ENV_NAME>.sls if you want to make additional changes to ceph configuration. Take a look at those options to match with your machines:

ceph:
  version: hammer
  cluster_name: ceph
  rest_api:
    port: 5000
  global:
    cluster_network: 192.168.36.0/24
    fsid: 294bc494-81ba-4c3c-ac5d-af7b3442a2a5
    public_network: 192.168.33.0/24
  mon:
    interface: eth1 # Should match public_network

Proceed with deployment step after all changes are done.

NOTE: Generate your FSID with uuidgen command!

Deployment

First, you need to run high state to add roles to minions based on environment-<ENV_NAME>.sls file. All roles for all environments will be applied:

sudo salt '*' state.highstate

To start Ceph cluster deployment run orchestrate state:

sudo salt-run state.orchestrate deploy.ceph pillar='{environment: ENV_NAME}'

It will take few minutes to complete. Then you can check ceph cluster status:

sudo ceph -s

ceph-salt's People

Contributors

komljen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ceph-salt's Issues

terraform + saltstack + ceph

Ciao @komljen first congrats for the project seems really well done ๐Ÿ‘

I am creating a new opensource project which will aim to deploy ceph via terraform (kvm, cloud, etc), and provision it with saltstack or ansible.

For the saltstack part, i want to create the prerequisites to run https://github.com/SUSE/DeepSea.

The project is opensource and claim to support all OS that provide an upstream Cloudimage. ( i did a table for that)

https://github.com/MalloZup/ceph-open-terrarium

You might be interested or want to contribute there step by ๐Ÿ‘ cu

typo in cache config

"flash" should be "flush".

vagrant@master:/srv/pillar$ grep writethrough data/ceph.sls
rbd_cache_writethrough_until_flash: "true"

vagrant@master:/srv/pillar$ sudo salt 'node02' cmd.run 'ceph daemon osd.0 config show | grep rbd_cache_write'
node02:
"rbd_cache_writethrough_until_flush": "false",

Error connecting to cluster: PermissionError

I seem to run into an issue with your scripts. The orchestration finishes, but when I try to 'sudo ceph -s' or 'ceph -s' I get the error in the title. When I 'sudo chown harold:harold ceph.client.admin.keyring' , and then try to 'ceph -s' everything works fine. I suspect there is something wrong with the ownership of the automatically generated keyring? 'chmod 777' does not fix it, it has to be owned by the calling user...

I would like to know if there is a way to fix this in the code?

cluster comes up with all pgs stuck inactive/unclean

I've tested both with vagrant and physical machines with more osds/pgs, the result is in both cases the cluster comes up with all pgs stuck unclean/inactive. Output of ceph -s :

root@node01:/var/log/ceph# ceph -s
cluster 294bc494-81ba-4c3c-ac5d-af7b3442a2a5
health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean; clock skew detected on mon.node02
monmap e3: 3 mons at {master=192.168.33.10:6789/0,node01=192.168.33.20:6789/0,node02=192.168.33.30:6789/0}, election epoch 6, quorum 0,1,2 master,node01,node02
osdmap e7: 4 osds: 4 up, 4 in
pgmap v16: 192 pgs, 3 pools, 0 bytes data, 0 objects
130 MB used, 16209 MB / 16339 MB avail
192 creating

sync client.admin.keyring

you want sync ceph.client.admin.keyring to ceph node by bellow code in mon.sls with this:

{% for mon in salt'mine.get' -%}

cp.get_file {{ mon }}{{ conf.admin_keyring }}:
module.wait:
- name: cp.get_file
- path: salt://{{ mon }}/files{{ conf.admin_keyring }}
- dest: {{ conf.admin_keyring }}
- watch:
- cmd: {{ conf.admin_keyring }}

{% endfor -%}

and push the cleint.admin.keyring by

cp.push {{ conf.admin_keyring }}: module.wait: - name: cp.push - path: {{ conf.admin_keyring }} - watch: - cmd: gen_admin_keyring

the cp.push will collect the {{conf.admin_keyring}} to /var/cache/salt/master/minions/{{mnion}}/files/etc/ceph/ceph.client.admin.keyring; but how could cp.get_file get this file?

is not that cp.get_file just send the file at /srv/salt to the minion,
how could send the file of "/var/cache/salt/master/minions"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.