piraeusdatastore / piraeus Goto Github PK

View Code? Open in Web Editor NEW

418.0 418.0 50.0 39.94 MB

High Available Datastore for Kubernetes

Home Page: https://piraeus.io/

License: Apache License 2.0

Dockerfile 22.33% Shell 59.94% Makefile 17.73%

piraeus's People

Contributors

Stargazers

Watchers

Forkers

linbit pigmej lalinsky ivan4th eugenewolf alexzhc bratao renzhengeek lsadehaan huangkaiyan10 aland-zhang wanzenbug orlandoparser orlandobrea cevalogistics frankkkkk vavere hwaastad hurdonkey neujie kryptonite-ru tnaganawa datanela timyinshi jukie simonyangchao envooch nathanawmk moteesh-reddy xmlgrg 0xflotus kvaps schmidax neilluo1 yankay naveenkumarhacker rid clrxbl plaidcloud akurz basfeo phoenix-bjoern daocloud mtb-xt philipp-reisner aenix-io retinio windsonsea webclinic017

piraeus's Issues

When several PVCs are created simultaneously, linstor-satelite returns ChildProcessTimeoutException

Hi,

I set up 3 node kubernetes cluster with piraeus-operator.

When I create PVC one by one, everything works fine, and I can create up to 10 PVCs,
but when I create 10 PVCs at the same time, some linstor-satelites show ErrorReport.log with 'External command timed out' for pvdisplay command.

root@ip-172-31-128-5:/var/log/linstor-satellite# cat ErrorReport-600586FF-39674-000001.log
ERROR REPORT 600586FF-39674-000001

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Satellite
Version:                            1.11.1
Build ID:                           fe95a94d86c66c6c9846a3cf579a1a776f95d3f4
Build time:                         2021-01-13T08:34:55+00:00
Error time:                         2021-01-18 13:54:00
Node:                               ip-172-31-128-5.ap-northeast-1.compute.internal

============================================================

Reported error:
===============

Description:
    Failed to get physical devices for volume group: drbdpool
Cause:
    External command timed out
Additional information:
    External command: pvdisplay --columns -o pv_name -S vg_name=drbdpool --noheadings --nosuffix

Category:                           LinStorException
Class name:                         StorageException
Class canonical name:               com.linbit.linstor.storage.StorageException
Generated at:                       Method 'genericExecutor', Source file 'Commands.java', Line #121

Error message:                      Failed to get physical devices for volume group: drbdpool

When this behavior occurred, pvdisplay command and pvs command actually takes long time to return some output.

root@ip-172-31-128-5:/var/log/linstor-satellite# pvdisplay --columns -o pv_name -S vg_name=drbdpool --noheadings --nosuffix
^C
root@ip-172-31-128-5:/var/log/linstor-satellite# pvs
^C
root@ip-172-31-128-5:/var/log/linstor-satellite#

and that satelite node becomes OFFLINE and volume will be Uknown status ..

root@piraeus-op-cs-controller-67bccfd9bc-ppzbd:/# linstor --controller 127.0.0.1 v list
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                                              ┊ Resource                                 ┊ StoragePool ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊ Allocated ┊ InUse  ┊        State ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-0b9dbca3-e55a-4088-bf72-1675a02641cd ┊ lvm-thick   ┊     0 ┊    1004 ┊ None          ┊           ┊        ┊      Unknown ┊
┊ ip-172-31-128-202.ap-northeast-1.compute.internal ┊ pvc-0bfec7d8-0fbc-42e4-8184-95d5bb800424 ┊ lvm-thick   ┊     0 ┊    1000 ┊ /dev/drbd1000 ┊  2.00 GiB ┊ Unused ┊     UpToDate ┊
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-29079ec4-f476-442e-980d-546bfe924e83 ┊ lvm-thick   ┊     0 ┊    1003 ┊ None          ┊           ┊        ┊      Unknown ┊
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-40315b55-7dda-4847-9672-61b9da6e9fd7 ┊ lvm-thick   ┊     0 ┊    1007 ┊ None          ┊           ┊        ┊      Unknown ┊
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-64f637ba-5d25-443b-a9f0-cbf4529a3c90 ┊ lvm-thick   ┊     0 ┊    1002 ┊ None          ┊           ┊        ┊      Unknown ┊
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-7978d086-62ba-4568-87f2-fae081fdb8b3 ┊ lvm-thick   ┊     0 ┊    1005 ┊ None          ┊           ┊        ┊      Unknown ┊
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-82b01996-ad41-4a2f-b011-bf663bf65827 ┊ lvm-thick   ┊     0 ┊    1006 ┊ None          ┊           ┊        ┊      Unknown ┊
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ pvc-c49ffc32-ae18-472c-a5e9-4a2df68fc75d ┊ lvm-thick   ┊     0 ┊    1001 ┊ /dev/drbd1001 ┊  2.00 GiB ┊ Unused ┊ Inconsistent ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

root@piraeus-op-cs-controller-67bccfd9bc-ppzbd:/# linstor --controller 127.0.0.1 n list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                                              ┊ NodeType   ┊ Addresses                   ┊ State   ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ ip-172-31-128-5.ap-northeast-1.compute.internal   ┊ SATELLITE  ┊ 172.31.128.5:3366 (PLAIN)   ┊ OFFLINE ┊
┊ ip-172-31-128-90.ap-northeast-1.compute.internal  ┊ SATELLITE  ┊ 172.31.128.90:3366 (PLAIN)  ┊ Online  ┊
┊ ip-172-31-128-202.ap-northeast-1.compute.internal ┊ SATELLITE  ┊ 172.31.128.202:3366 (PLAIN) ┊ Online  ┊
┊ piraeus-op-cs-controller-67bccfd9bc-ppzbd         ┊ CONTROLLER ┊ 10.47.255.243:3366 (PLAIN)  ┊ Online  ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@piraeus-op-cs-controller-67bccfd9bc-ppzbd:/#

Since strace command returns 'EMEDIUMTYPE (Wrong medium type)' for drbd device which are currently being created (so the state is secondary),

[root@ip-172-31-128-5 ~]# strace pvs
(snip)
stat("/dev/drbd1002", {st_mode=S_IFBLK|0660, st_rdev=makedev(147, 1002), ...}) = 0
open("/dev/drbd1002", O_RDONLY|O_DIRECT|O_NOATIME) = -1 EMEDIUMTYPE (Wrong medium type)
open("/dev/drbd1002", O_RDONLY|O_NOATIME) = -1 EMEDIUMTYPE (Wrong medium type)
stat("/dev/drbd1003", {st_mode=S_IFBLK|0660, st_rdev=makedev(147, 1003), ...}) = 0
open("/dev/drbd1003", O_RDONLY|O_DIRECT|O_NOATIME) = -1 EMEDIUMTYPE (Wrong medium type)
open("/dev/drbd1003", O_RDONLY|O_NOATIME) = -1 EMEDIUMTYPE (Wrong medium type)
stat("/dev/drbd1004", {st_mode=S_IFBLK|0660, st_rdev=makedev(147, 1004), ...}) = 0
open("/dev/drbd1004", O_RDONLY|O_DIRECT|O_NOATIME) = -1 EMEDIUMTYPE (Wrong medium type)
open("/dev/drbd1004", O_RDONLY|O_NOATIME) = -1 EMEDIUMTYPE (Wrong medium type)
stat("/dev/drbd1005", {st_mode=S_IFBLK|0660, st_rdev=makedev(147, 1005), ...}) = 0

I guess this issue is the same with this one,
LINBIT/linstor-server#8
LINBIT/linstor-server#108
and setting lvm.conf in ns-node containers,
pvdisplay command works fast and up to 10 parallel PVC creation worked fine.

Piraeus/linstor configuration lost after full cluster reboot

The current deployment uses a statefullset with 3 etcd pods and is setup to always start without any data. It seems that this is the reason why we loose our linstor configuration after a full reboot of the cluster (reproducable by deleting all etcd pods simultaniously).

We were planning to "just" back the etcd pods with a hostPath volume for data. However since there has been put considered effort in the init container (and stopHook) in making sure that the data folder is moved/archived we are hesitant in doing so.

Would it be possible to support durable etcd data (surviving a cluster wide outage of the etcd service)? Or is there an other way envisioned to retain the linstor configuration after such an event?

Prestop-etcd is creating empty dumps

Hello,
on dockerfiles/piraeus-init/bin/prestop-etcd.sh you do a dump of keys using:

etcdctl get --prefix "LINSTOR/" -w simple > "${data_dir}_${mod_time}/keys.txt"
etcdctl get --prefix "LINSTOR/" -w json > "${data_dir}_${mod_time}/keys.json"

This creates a empty dump for me. In my database the prefix is "/LINSTOR/"

kernel crashes at Oracle Linux 8

Kubernetes v1.27.5
Bare metal nodes
LVM Thinpool
piraeus-operator v2.4.1
Oracle Linux 8
Kernel 5.15.0-204.147.6.2.el8uek.x86_64 + default drbd image drbd9-jammy
Also reproduced with kernel 4.18 + drbd image drbd9-almalinux8

How to reproduce:
Create and subsequently delete a number of volumes and attach them. I tested with about 8 pvc-s and pod-s and made around 20 operations of creation and then deletion of them. Randomly the server goes to reboot because of crash. Most often it happened during volumes deletion but also it was reproduced during a new pvc creation.

UEK kernel Makefile (/usr/src/kernels/5.15.0-204.147.6.2.el8uek.x86_64/Makefile) patched to be able to build drbd:

--- Makefile	2024-01-15 12:24:44.452296691 +0000
+++ Makefile	2024-01-15 12:25:36.325543428 +0000
@@ -853,18 +853,18 @@
 endif
 
 # Initialize all stack variables with a 0xAA pattern.
-ifdef CONFIG_INIT_STACK_ALL_PATTERN
-KBUILD_CFLAGS	+= -ftrivial-auto-var-init=pattern
-endif
+#ifdef CONFIG_INIT_STACK_ALL_PATTERN
+#KBUILD_CFLAGS	+= -ftrivial-auto-var-init=pattern
+#endif
 
 # Initialize all stack variables with a zero value.
-ifdef CONFIG_INIT_STACK_ALL_ZERO
-KBUILD_CFLAGS	+= -ftrivial-auto-var-init=zero
-ifdef CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO_ENABLER
+#ifdef CONFIG_INIT_STACK_ALL_ZERO
+#KBUILD_CFLAGS	+= -ftrivial-auto-var-init=zero
+#ifdef CONFIG_CC_HAS_AUTO_VAR_INIT_ZERO_ENABLER
 # https://github.com/llvm/llvm-project/issues/44842
-KBUILD_CFLAGS	+= -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
-endif
-endif
+#KBUILD_CFLAGS	+= -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
+#endif
+#endif

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: piraeus-storage-pool
spec:
  storagePools:
    - name: piraeus-storage-pool-lvmthin
      lvmThinPool:
        volumeGroup: lvmvgthin
        thinPool: thinpool_piraeus
  podTemplate:
    spec:
      hostNetwork: true
  nodeAffinity:
    nodeSelectorTerms:
    - matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
      - key: piraeus
        operator: In
        values:
         - enabled

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: piraeus-storage-replicated-lvm
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
parameters:
  # https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-kubernetes-sc-parameters
  ## CSI related parameters
  csi.storage.k8s.io/fstype: ext4
  ## LINSTOR parameters
  linstor.csi.linbit.com/storagePool: piraeus-storage-pool-lvmthin
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/mountOpts: noatime,discard
  property.linstor.csi.linbit.com/DrbdOptions/Net/max-buffers: "11000"

---
apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstorcluster
spec:
  nodeAffinity:
    nodeSelectorTerms:
    - matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  # https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-autoplace-linstor
  properties:
    - name: DrbdOptions/Net/max-buffers # controller level
      value: "10000"
    - name: Autoplacer/Weights/MaxFreeSpace
      value: "0" # 1 default
    - name: Autoplacer/Weights/MinReservedSpace
      value: "10" # preferr nodes with minimal reserved space on thin pool
    - name: Autoplacer/Weights/MinRscCount
      value: "0"
    # - name: Autoplacer/Weights/MaxThroughput
    #   value: "0" # COOL but not today

cat /proc/drbd 
version: 9.2.8 (api:2/proto:86-122)
GIT-hash: e163b05a76254c0f51f999970e861d72bb16409a build by @srvh52.example.com, 2024-03-28 15:13:48
Transports (api:20): tcp (9.2.8) lb-tcp (9.2.8) rdma (9.2.8)

[ 4083.197349] Call Trace:
[ 4083.208990]  <TASK>
[ 4083.220334]  ? show_trace_log_lvl+0x1d6/0x2f9
[ 4083.231532]  ? show_trace_log_lvl+0x1d6/0x2f9
[ 4083.242553]  ? drbd_free_peer_req+0x99/0x210 [drbd]
[ 4083.253383]  ? __die_body.cold+0x8/0xa
[ 4083.263954]  ? page_fault_oops+0x16d/0x1ac
[ 4083.274325]  ? exc_page_fault+0x68/0x13b
[ 4083.284460]  ? asm_exc_page_fault+0x22/0x27
[ 4083.294360]  ? _raw_spin_lock_irq+0x13/0x58
[ 4083.303995]  drbd_free_peer_req+0x99/0x210 [drbd]
[ 4083.313482]  drbd_finish_peer_reqs+0xc0/0x180 [drbd]
[ 4083.322880]  drain_resync_activity+0x25b/0x43a [drbd]
[ 4083.332060]  conn_disconnect+0xf4/0x650 [drbd]
[ 4083.341017]  drbd_receiver+0x53/0x60 [drbd]
[ 4083.349787]  drbd_thread_setup+0x77/0x1df [drbd]
[ 4083.358332]  ? drbd_reclaim_path+0x90/0x90 [drbd]
[ 4083.366677]  kthread+0x127/0x144
[ 4083.374961]  ? set_kthread_struct+0x60/0x52
[ 4083.382938]  ret_from_fork+0x22/0x2d
[ 4083.390678]  </TASK>

piraeus-server,docker: Makefile only builds with bash

The Makefile was broken by introducing some bashisms and now needs a make update ... SHELL=/bin/bash, at least on Debian/dash. This should be fixed.

Old images on quay.

Hey,

Despite of fixes I see that all images even with latest tag are old on quay.io. Do you guys have any procedure to update them?

[Question] Does Piraeus now support k8s v1.18.x?

README lists that priaeus support k8s until v1.17.x.

When will Piraues support v1.18.x?

Resource status Standalone if piraeus-local-dflt-r2 is used

Hi,

since a few days i'm testing Piraeus in an rke based Kubernetes 1.15.11 cluster. The Master and Worker nodes are using Ubuntu 18.04 with the minimal (later also generic server) image and the DRBD packages from the LINBIT PPA. I'm just using https://github.com/piraeusdatastore/piraeus/blob/master/deploy/all.yaml, but i have replaced the file_thin disks with dedicated LVM disks on /dev/vdb.

All components are deployed fine, linstor node info looks good, linstor sp l looks good too and deploying a statefulset with the piraeus-local-dflt-r1 storageclass also works as expected (fio benchmark successful).

The problem starts with the storageclass piraeus-local-dflt-r2. Two volumes on two Worker nodes will be created, one volume is UpToDate and the other is Inconsistent. The drbdadm status of both nodes is Standalone. If i trigger drbdadm adjust all or linstor node reconnect, both switch into the Connecting state. Netstat shows no established connection on both nodes.

tcp        0      0 192.168.53.111:7000     0.0.0.0:*               LISTEN      0          52963      -

Later i have also updated the all.yaml with the recent image versions (linstor 1.6.1 instead of 1.4.2 and so on) and i have also switched the Ubuntu kernel from 4.15 to 5.3 (HWE). It does not change the behavior. Tcpdump on the Kubernetes nodes shows communication between the nodes on port 7000.

Any ideas?

ERROR: failed to push quay.io/piraeusdatastore/drbd9-flatcar:v9.1.14: unexpected status: 401 UNAUTHORIZE

Hi all!

I read in the documentation that there should be a drb9-flatcar image available via quay.io. Looking into the logs of the latest Github actions run of the "Build driver loader" it looks like there an authorization issue when trying to push the images for flatcar ... maybe a wrong/missing secret?

Cheers,
Andreas

Loading drivers on ubuntu broken because of invalid $

piraeus/dockerfiles/piraeus-init/config-node.sh

Line 41 in cdb550b

$MOUNT_USR_LIB='-v /usr/lib:/usr/lib:ro'

That $ clearly shouldn't be there. Will post PR shortly.

modinfo 8.4.10 vs. kernel-loader 9.0.27

We are using Ubuntu 18.04 bionic. modinfo drbd shows the version 8.4.10 while the kernel loader show a different version:

>    -v /lib/modules:/lib/modules -v /usr/src:/usr/src:ro \
>    -e "LB_INSTALL=yes" \
>    piraeusdatastore/drbd9-bionic:latest
DRBD module is already loaded

DRBD version loaded:
version: 9.0.27-1 (api:2/proto:86-118)
GIT-hash: bea41a056bb2abe4d5dfd2f69863282dfa1b2257 build by @de-fra-node38, 2021-01-21 10:18:25
Transports (api:16): tcp (9.0.27-1)

What is now the actual version? And how to release the version conflict?

DRBD_IMG_TAG: v9.0.21 does not exists for Ubuntu

Hi,
After freaking out because I could not get piraeus to work, I found out that it was trying to install a version of DRBD that do not exists.

In deploy/all.yaml it set DRBD_IMG_TAG: v9.0.21

And in init-node.sh it try to get the version or install:

elif [[ "v$( modinfo drbd \| awk '/^version: / {print $2}' )" == "${DRBD_IMG_TAG}-1" ]]; then
--

But https://launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack only contains the v9.0.22

no matches for kind "StatefulSet"

I m unable to apply
kubectl apply -f https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml

I get the following errors:

unable to recognize "https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml": no matches for kind "StatefulSet" in version "apps/v1beta1"
unable to recognize "https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml": no matches for kind "DaemonSet" in version "extensions/v1beta1"

I have changed the StatefulSet to apps/v1 (not knowing if thats the right way but it resolves it), but for the 2nd one i couldnt find a solution.

I m using Ubuntu 18.04 /rancher / kubectl

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.4", GitCommit:"224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba", GitTreeState:"clean", BuildDate:"2019-12-11T12:37:43Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

/etc/drbd.conf is a directory

kubectl apply -f https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml resulted in /etc/drbd.conf being a directory and pod/piraeus-node-* failing.

I was able to fix this by running touch /etc/drbd.conf on all nodes before launching.

How to limit Disk IO?

In K8S there is no way to limit Disk IO. So if using piraeus, it could impact K8S components. Any idea to solve this issue?

drbd9-focal compilation doesn't work on Ubuntu 20.04 5.15.0-43-generic

Cannot run drbd9-focal in compilation mode on Ubuntu 20.04 5.15.0-43-generic. It is failing because of

k logs piraeus-ns-node-vc9fg kernel-module-injector -f
Need a git checkout to regenerate drbd/.drbd_git_revision
make[1]: Entering directory '/tmp/pkg/drbd-9.1.8/drbd'

    Calling toplevel makefile of kernel source tree, which I believe is in
    KDIR=/lib/modules/5.15.0-43-generic/build

make -C /lib/modules/5.15.0-43-generic/build   M=/tmp/pkg/drbd-9.1.8/drbd  modules
  COMPAT  __vmalloc_has_2_params
  COMPAT  add_disk_returns_int
  COMPAT  before_4_13_kernel_read
  COMPAT  bio_alloc_has_5_params
  COMPAT  blkdev_issue_zeroout_discard
  COMPAT  can_include_vermagic_h
  COMPAT  fs_dax_get_by_bdev_takes_start_off
  COMPAT  genl_policy_in_ops
  COMPAT  have_BIO_MAX_VECS
  COMPAT  have_CRYPTO_TFM_NEED_KEY
  COMPAT  have_GENHD_FL_NO_PART
  COMPAT  have_SHASH_DESC_ON_STACK
  COMPAT  have_WB_congested_enum
  COMPAT  have_allow_kernel_signal
  COMPAT  have_bdev_nr_sectors
  COMPAT  have_bdgrab
  COMPAT  have_bdi_congested_fn
  COMPAT  have_bio_bi_bdev
  COMPAT  have_bio_bi_error
  COMPAT  have_bio_bi_opf
  COMPAT  have_bio_bi_status
  COMPAT  have_bio_clone_fast
  COMPAT  have_bio_op_shift
  COMPAT  have_bio_set_dev
  COMPAT  have_bio_set_op_attrs
  COMPAT  have_bio_start_io_acct
  COMPAT  have_bioset_init
  COMPAT  have_bioset_need_bvecs
  COMPAT  have_blk_alloc_disk
  COMPAT  have_blk_alloc_queue_rh
  COMPAT  have_blk_check_plugged
  COMPAT  have_blk_qc_t_make_request
  COMPAT  have_blk_qc_t_submit_bio
  COMPAT  have_blk_queue_flag_set
  COMPAT  have_blk_queue_make_request
  COMPAT  have_blk_queue_merge_bvec
  COMPAT  have_blk_queue_split_bio
  COMPAT  have_blk_queue_split_q_bio
  COMPAT  have_blk_queue_split_q_bio_bioset
  COMPAT  have_blk_queue_update_readahead
  COMPAT  have_blk_queue_write_cache
  COMPAT  have_d_inode
  COMPAT  have_disk_update_readahead
  COMPAT  have_fallthrough
  COMPAT  have_fs_dax_get_by_bdev
  COMPAT  have_generic_start_io_acct_q_rw_sect_part
  COMPAT  have_generic_start_io_acct_rw_sect_part
  COMPAT  have_genl_family_parallel_ops
  COMPAT  have_hd_struct
  COMPAT  have_ib_cq_init_attr
  COMPAT  have_ib_get_dma_mr
  COMPAT  have_idr_is_empty
  COMPAT  have_inode_lock
  COMPAT  have_ktime_to_timespec64
  COMPAT  have_kvfree
  COMPAT  have_max_send_recv_sge
  COMPAT  have_nla_nest_start_noflag
  COMPAT  have_nla_parse_deprecated
  COMPAT  have_nla_put_64bit
  COMPAT  have_nla_strscpy
  COMPAT  have_part_stat_h
  COMPAT  have_part_stat_read_accum
  COMPAT  have_pointer_backing_dev_info
  COMPAT  have_proc_create_single
  COMPAT  have_queue_flag_stable_writes
  COMPAT  have_rb_declare_callbacks_max
  COMPAT  have_refcount_inc
  COMPAT  have_req_hardbarrier
  COMPAT  have_req_noidle
  COMPAT  have_req_nounmap
  COMPAT  have_req_op_write
  COMPAT  have_req_op_write_zeroes
  COMPAT  have_req_write
  COMPAT  have_revalidate_disk_size
  COMPAT  have_sched_set_fifo
  COMPAT  have_security_netlink_recv
  COMPAT  have_sendpage_ok
  COMPAT  have_set_capacity_and_notify
  COMPAT  have_shash_desc_zero
  COMPAT  have_simple_positive
  COMPAT  have_sock_set_keepalive
  COMPAT  have_struct_bvec_iter
  COMPAT  have_struct_size
  COMPAT  have_submit_bio_noacct
  COMPAT  have_tcp_sock_set_cork
  COMPAT  have_tcp_sock_set_nodelay
  COMPAT  have_tcp_sock_set_quickack
  COMPAT  have_time64_to_tm
  COMPAT  have_timer_setup
  COMPAT  have_void_make_request
  COMPAT  have_void_submit_bio
  COMPAT  ib_alloc_pd_has_2_params
  COMPAT  ib_device_has_ops
  COMPAT  ib_post_send_const_params
  COMPAT  ib_query_device_has_3_params
  COMPAT  need_make_request_recursion
  COMPAT  part_stat_read_takes_block_device
  COMPAT  queue_limits_has_discard_zeroes_data
  COMPAT  rdma_create_id_has_net_ns
  COMPAT  sock_create_kern_has_five_parameters
  COMPAT  sock_ops_returns_addr_len
  COMPAT  struct_gendisk_has_backing_dev_info
  UPD     /tmp/pkg/drbd-9.1.8/drbd/compat.5.15.39.h
  UPD     /tmp/pkg/drbd-9.1.8/drbd/compat.h
make[4]: 'drbd-kernel-compat/cocci_cache/74688e3f60b12f8e1adfcc25debe6f57/compat.patch' is up to date.
  PATCH
patching file ./drbd_int.h
patching file drbd_req.c
patching file drbd_receiver.c
patching file drbd_nl.c
patching file drbd_main.c
patching file drbd_debugfs.c
patching file drbd_dax_pmem.c
patching file drbd_bitmap.c
patching file drbd_actlog.c
  CC [M]  /tmp/pkg/drbd-9.1.8/drbd/drbd_dax_pmem.o
./tools/objtool/objtool: error while loading shared libraries: libelf.so.1: cannot open shared object file: No such file or directory
make[3]: *** [scripts/Makefile.build:285: /tmp/pkg/drbd-9.1.8/drbd/drbd_dax_pmem.o] Error 127
make[3]: *** Deleting file '/tmp/pkg/drbd-9.1.8/drbd/drbd_dax_pmem.o'
make[2]: *** [Makefile:1875: /tmp/pkg/drbd-9.1.8/drbd] Error 2
make[1]: *** [Makefile:134: kbuild] Error 2
make[1]: Leaving directory '/tmp/pkg/drbd-9.1.8/drbd'
make: *** [Makefile:126: module] Error 2

Could not find the expexted *.ko, see stderr for more details

piraeus-controller-0 cannot connect to etcd

On deploying the Getting started example, the piraeus-controller-0 ends in a CrashLoopBackOff, with the error message:

ERROR LINSTOR/Controller - SYSTEM - No connection to ETCD server [Report number 5E69444E-00000-000000]

Etcd is deployed and accessible via it's service. Suggestions on howto debug or resolve this issue are welcome.

Logging of the piraeus-controller-0 pod:

kubectl -n kube-system logs -f piraeus-controller-0
LINSTOR, Module Controller
Version:            1.4.2 (974dfcad291e1f683941ada3d7e7337821060349)
Build time:         2020-01-27T11:15:32+00:00
Java Version:       11
Java VM:            Debian, Version 11.0.6+10-post-Debian-1deb10u1
Operating system:   Linux, Version 3.10.0-1062.9.1.el7.x86_64
Environment:        amd64, 1 processors, 247 MiB memory reserved for allocations

System components initialization in progress

20:04:31.119 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
20:04:31.295 [main] INFO  LINSTOR/Controller - SYSTEM - Linstor configuration file loaded from '/etc/linstor/linstor.toml'.
20:04:31.296 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
20:04:32.691 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 1350ms
20:04:32.692 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/linstor-server/lib/guice-4.2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20:04:36.029 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 3337ms
20:04:37.003 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
20:04:37.533 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the etcd database
20:04:49.655 [Main] ERROR LINSTOR/Controller - SYSTEM - No connection to ETCD server [Report number 5E69444E-00000-000000]

20:04:49.657 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Shutdown in progress
20:04:49.658 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'ETCDDatabaseService' of type ETCDDatabaseService
20:04:49.662 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'ETCDDatabaseService' to complete shutdown
20:04:49.662 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
20:04:49.662 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
20:04:49.662 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
20:04:49.663 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
20:04:49.663 [Thread-0] INFO  LINSTOR/Controller - SYSTEM - Shutdown complete

Etcd and other piraeus services start OK:

kube-system     piraeus-controller-0                       0/1     Error       1          66s
kube-system     piraeus-csi-controller-0                   5/5     Running     0          60m
kube-system     piraeus-csi-node-nstql                     2/2     Running     0          60m
kube-system     piraeus-csi-node-qv6rv                     2/2     Running     0          60m
kube-system     piraeus-csi-node-swzmv                     2/2     Running     0          60m
kube-system     piraeus-etcd-0                             1/1     Running     0          60m
kube-system     piraeus-etcd-1                             1/1     Running     0          60m
kube-system     piraeus-etcd-2                             1/1     Running     0          60m
kube-system     piraeus-node-6gqhs                         0/1     Init:0/1    1          60m
kube-system     piraeus-node-k64d5                         0/1     Init:0/1    1          60m
kube-system     piraeus-node-qczk6                         0/1     Init:0/1    1          60m

Etcd is accessible via the svc:

# etcdctl --endpoints=http://10.43.119.148:2379 cluster-health
member 150e6aaeb9e31b16 is healthy: got healthy result from http://piraeus-etcd-0.piraeus-etcd:2379
member ad8612fde261bb43 is healthy: got healthy result from http://piraeus-etcd-2.piraeus-etcd:2379
member e41ccd855ab0c8bf is healthy: got healthy result from http://piraeus-etcd-1.piraeus-etcd:2379
cluster is healthy

Enable DCO per CNCF IP Policy

install and enable this for all repos :)

https://github.com/apps/dco

A potential risk in piraeus that could lead to takeover of the cluster

Dear Team Members:

Greetings! Our team is very interested in your project and we recently identified a potential RBAC security risk while doing a security assessment of your project. Therefore, we would like to report it to you and provide you with the relevant details so that you can fix and improve it accordingly. I couldn't find a way to report security issues, so I made the report here. I apologize if this is not the appropriate channel for reporting.

Details:

In this Kubernetes project, there exists a ClusterRole that has been granted list secrets high-risk permissions. These permissions allow the role to list confidential information across the cluster. An attacker could impersonate the ServiceAccount bound to this ClusterRole and use its high-risk permissions to list secrets information across the cluster. By combining the permissions of other roles, an attacker can elevate the privileges and further take over the entire cluster.

we constructed the following attack vectors.

First, you need to get a token for the ServiceAccount that has this high-risk privilege. If you are already in a Pod and have this override, you can directly run the following command to get the token: cat /var/run/secrets/kubernetes.io/serviceaccount/ token. If you are on a node other than a Pod, you can run the following command to get the kubectl describe secret .

Use the obtained token information to authenticate with the API server. By including the token in the request, you can be recognized as a legitimate user with a ServiceAccount and gain all privileges associated with the ServiceAccount. As a result, this ServiceAccount identity can be used to list all secrets in the cluster.

We give two ways to further utilize ServiceAccount Token with other privileges to take over the cluster:

Method 1: Elevation of Privilege by Utilizing ServiceAccount Token Bound to ClusterAdmin

Directly use a Token with the ClusterAdmin role permissions that has the authority to control the entire cluster. By authenticating with this token, you can gain full control of the cluster.

Method 2: Create Privileged Containers with ServiceAccount Token with create pods permission You can use this ServiceAccount Token to create a privileged container that mounts the root directory and schedules it to the master node in a taint-tolerant way, so that you can access and leak the master node's kubeconfig configuration file. In this way you can take over the entire cluster.

For the above attack chain we have developed exploit code and uploaded it to github: https://github.com/HouqiyuA/k8s-rbac-poc

Mitigation methods are explored:

Carefully evaluate the permissions required for each user or service account to ensure that it is following the principle of least privilege and to avoid over-authorization.

If list secrets is a required permission, consider using more granular RBAC rules. Role Binding can be used to grant list secrets permissions instead of ClusterRole, which restricts permissions to specific namespaces or resources rather than the entire cluster.

Isolate different applications into different namespaces and use namespace-level RBAC rules to restrict access. This reduces the risk of privilege leakage across namespaces

Looking forward to hearing from you and discussing this risk in more detail with us, thank you very much for your time and attention.

Best wishes.

HouqiyuA

Unable to find way to reduce placement count

I've been trying to find a way to reduce the the placementCounts of a volume.

What I've done:

Create storageClass "lvm-thin" with setting autoPlace: 3
Replace storageClass "lvm-thin" with setting autoPlace: 2
Tried updating the resource-definition to autoplace 2 (linstor resource-definition ap pvc-xxx --place-count 2)
- Gave error: The resource would have to be deleted from nodes to reach the placement count.
Tried changing one of the disks using: https://github.com/LINBIT/linbit-documentation/blob/master/UG9/en/linstor-administration.adoc#adding-and-removing-disks
- Resulted in error: context deadline exceeded Couldn't connect to ETC, however all other commands seem to work fine

Can someone point me to the right documentation for reducing the placement count?

Update from piraeus-server 1.11.1 to 1.12.3 failed with Database initialization error

Hello,

I recently tried to update the Piraeus operator 1.4 to the new release 1.5, but the controller failed to do the upgrade. The error-report states that an "Exception occured during migration". The original Java exception seems to be a NumberFormatException. Is there something wrong in my database? How could I try to debug this issue?

Full error-report

Thank you.

piraeus-csi-controller-0 pod pending, insufficient cpu

Hi,

i wanted to test Piraeus and spawned a cluster of 1+4 worker nodes, ubuntu 18.04 on each, running on a kvm/libvrt setup, each 4GB RAM and 2 CPU (below output of /proc/cpuinfo). The host is a 32GB RAM intel nuc with 4 core Intel(R) Core(TM) i5-6260U CPU @ 1.80GHz.
The cluster was spawned for this particular test, no other workload is running, other than flannel.

It seems pod piraeus-csi-controller-0 is Pending due to

Warning FailedScheduling 3m25s (x39 over 53m) default-scheduler 0/5 nodes are available: 5 Insufficient cpu.

wrt to the Piraeus setup some nodes are also Pending as per below:

Warning FailedScheduling 23s (x44 over 56m) default-scheduler 0/5 nodes are available: 1 Insufficient cpu, 4 node(s) didn't match node selector.

Any advice ?
Thanks !

kube-system   piraeus-controller-0             0/1     Init:0/1   0          55m
kube-system   piraeus-csi-controller-0         0/5     Pending    0          55m
kube-system   piraeus-csi-node-7xht4           0/2     Pending    0          55m
kube-system   piraeus-csi-node-9d975           2/2     Running    0          55m
kube-system   piraeus-csi-node-9zkkl           2/2     Running    0          55m
kube-system   piraeus-csi-node-k8wlp           2/2     Running    0          55m
kube-system   piraeus-etcd-0                   1/1     Running    0          55m
kube-system   piraeus-etcd-1                   1/1     Running    0          55m
kube-system   piraeus-etcd-2                   1/1     Running    0          55m
kube-system   piraeus-node-dmqxh               0/1     Init:0/1   0          55m
kube-system   piraeus-node-jxm2j               0/1     Init:0/1   0          55m
kube-system   piraeus-node-ldslx               0/1     Init:0/1   0          55m
kube-system   piraeus-node-wwj2s               0/1     Init:0/1   0          55m

/proc/cpuinfo for the node.

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 61
model name	: Intel Core Processor (Broadwell, no TSX, IBRS)
stepping	: 2
microcode	: 0x1
cpu MHz		: 1799.998
cache size	: 16384 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 arat md_clear
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips	: 3599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 61
model name	: Intel Core Processor (Broadwell, no TSX, IBRS)
stepping	: 2
microcode	: 0x1
cpu MHz		: 1799.998
cache size	: 16384 KB
physical id	: 1
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 arat md_clear
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips	: 3599.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

piraeus-controller goes into NotReady constantly

readinessProbe settings are too tight that it went into NotReady then Ready loop constantly.

The default timeoutSeconds of 1s (probably injected by kubelet) caused so many of "Timeout exceeded while awaiting headers" since I noticed it took a while to respond to linstor controller list-properties.

I increased the timeoutSeconds to 5s then it stayed at ready. I think we should define it in the deployment file. I also increased the periodSeconds so it does not spam the rest-access.log.

By the way, why are we mounting /etc/localtime? It caused me headaches because /etc/localtime on Fedora CoreOS for some reason is a empty directory. I don't see the reason since other deployments of Linstor does not do this.

Losing quorum as soon as a node goes down

Hi,

I have 3 nodes, and a placementCount of 2. After quite a bit of fiddling, the third node got 'TieBreaker' volumes (or Diskless, for some) setup on it, so I'd assume I'm okay to lose one node.

But sadly as soon as any of the nodes go down, I lose quorum and the remaining two nodes get tainted with drbd.linbit.com/lost-quorum:NoSchedule.

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node          ┊ Port ┊ Usage  ┊ Conns                     ┊      State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-1a9e5a5e-fdba-4b8e-ae9f-1a7acd048184 ┊ talos-00r-fu9 ┊ 7001 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊   Diskless ┊ 2023-11-19 15:36:20 ┊
┊ pvc-1a9e5a5e-fdba-4b8e-ae9f-1a7acd048184 ┊ talos-813-fn2 ┊ 7001 ┊ InUse  ┊ Connecting(talos-ozt-z3h) ┊   UpToDate ┊ 2023-11-07 18:03:48 ┊
┊ pvc-1a9e5a5e-fdba-4b8e-ae9f-1a7acd048184 ┊ talos-ozt-z3h ┊ 7001 ┊        ┊                           ┊    Unknown ┊ 2023-10-27 12:04:28 ┊
┊ pvc-56924ed3-7815-4655-9536-6b64792182ca ┊ talos-00r-fu9 ┊ 7004 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊   Diskless ┊ 2023-11-19 15:36:23 ┊
┊ pvc-56924ed3-7815-4655-9536-6b64792182ca ┊ talos-813-fn2 ┊ 7004 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊   UpToDate ┊ 2023-11-19 09:47:12 ┊
┊ pvc-56924ed3-7815-4655-9536-6b64792182ca ┊ talos-ozt-z3h ┊ 7004 ┊        ┊                           ┊    Unknown ┊ 2023-10-27 12:04:32 ┊
┊ pvc-86499a05-3ba9-4722-9bb1-69ae47406263 ┊ talos-00r-fu9 ┊ 7005 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊ TieBreaker ┊ 2023-11-19 15:36:23 ┊
┊ pvc-86499a05-3ba9-4722-9bb1-69ae47406263 ┊ talos-813-fn2 ┊ 7005 ┊ InUse  ┊ Connecting(talos-ozt-z3h) ┊   UpToDate ┊ 2023-11-12 18:11:43 ┊
┊ pvc-86499a05-3ba9-4722-9bb1-69ae47406263 ┊ talos-ozt-z3h ┊ 7005 ┊        ┊                           ┊    Unknown ┊ 2023-11-12 18:11:43 ┊
┊ pvc-c7bdfa9e-e3c2-4dd3-ac9c-b7b2e847d30b ┊ talos-00r-fu9 ┊ 7003 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊   Diskless ┊ 2023-11-19 15:36:23 ┊
┊ pvc-c7bdfa9e-e3c2-4dd3-ac9c-b7b2e847d30b ┊ talos-813-fn2 ┊ 7003 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊   UpToDate ┊ 2023-11-07 18:03:50 ┊
┊ pvc-c7bdfa9e-e3c2-4dd3-ac9c-b7b2e847d30b ┊ talos-ozt-z3h ┊ 7003 ┊        ┊                           ┊    Unknown ┊ 2023-10-27 12:04:33 ┊
┊ pvc-e57930e5-6772-41e4-8c98-99105b77970a ┊ talos-00r-fu9 ┊ 7002 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊   Diskless ┊ 2023-11-19 15:36:23 ┊
┊ pvc-e57930e5-6772-41e4-8c98-99105b77970a ┊ talos-813-fn2 ┊ 7002 ┊ InUse  ┊ Connecting(talos-ozt-z3h) ┊   UpToDate ┊ 2023-11-07 18:03:49 ┊
┊ pvc-e57930e5-6772-41e4-8c98-99105b77970a ┊ talos-ozt-z3h ┊ 7002 ┊        ┊                           ┊    Unknown ┊ 2023-10-27 12:04:33 ┊
┊ pvc-fbdf5c3c-2d49-49b8-ac10-f8e1212c7788 ┊ talos-00r-fu9 ┊ 7000 ┊ Unused ┊ Connecting(talos-ozt-z3h) ┊ TieBreaker ┊ 2023-11-19 15:36:23 ┊
┊ pvc-fbdf5c3c-2d49-49b8-ac10-f8e1212c7788 ┊ talos-813-fn2 ┊ 7000 ┊ InUse  ┊ Connecting(talos-ozt-z3h) ┊   UpToDate ┊ 2023-11-08 08:47:17 ┊
┊ pvc-fbdf5c3c-2d49-49b8-ac10-f8e1212c7788 ┊ talos-ozt-z3h ┊ 7000 ┊        ┊                           ┊    Unknown ┊ 2023-10-27 12:05:17 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I have no idea why the above leads to loosing quorum, there's clearly two connected nodes (even if one is the TieBreaker).

I'm not sure what I'm doing wrong, but tainting the nodes like that make recovering pretty difficult as most pods won't get re-scheduled, depending on what went down I sometimes have to manually untaint a node to let pods come back up and slowly recover by hand, using drbdadm to decide which to keep for every volume.

Thanks

is etcd a must ?

Seems there is etcd container in the deploy , is it better to use k8s 's api ?
Thanks !

Two questions about drbd containerized installation solution

Hello, after reading the document, I found that there is a solution that uses drbd containerization.
https://github.com/piraeusdatastore/piraeus-operator/blob/master/doc/host-setup.md

First of all, I would like to ask whether containerized drbd charges. I checked the official website of linbit and found that containerized drbd requires commercial support.
I looked at the dockerfile of the centos8 image,
https://github.com/piraeusdatastore/piraeus/blob/master/dockerfiles/drbd-driver-loader/Dockerfile.centos8,
A bash script in the dockerfile entry.sh,
https://raw.githubusercontent.com/LINBIT/drbd/master/docker/entry.sh,
in this script only load the DRBD module,
It seems that the drbd service has not been started,
and there is no network related configuration.
Does this need to be operated on the host instead of in the container？

thanks

demo-sts-mysql.yaml penging

kubectl apply -f demo-sts-mysql.yaml pending status
lvm storageClass operate？
select "layerlist: drbd storage" sotrageClass is pengding
select "layerlist: storage" sotrageClass is ok!

Piraeus Annual Review Evaluation and next steps

I am the TOC member assigned to conduct the Evaluation of Piraeus' Annual Review. At this time the TOC is requesting a roadmap that details the dates and actions for the project to improve its overall health and continued progress towards incubation. This roadmap should address the findings below with anticipated dates of resolution. The Roadmap is due to the TOC by January 31st 2024 and may be provided as a comment on this issue within the TOC repo and tagging @TheFoxAtWork.

Please note the findings listed below are not all encompassing and are intended to focus specifically on bringing the project into alignment with expectations of a Sandbox project. For more information, please refer to the TOC Archival process.

The following were findings from the evaluation of the Annual Review:

Project needs to adopt the CNCF code of conduct. The governance file states it is aligned with the CNCF code of conduct however the CoC in the repository is the contributor covenant. Currently, TAG Contributor Strategy recommends projects link directly to the CNCF CoC within their own CoC file so they inherit any changes made in the future.
While there are new contributors to the project, it does not appear to be achieving the anticipated velocity to make substantial progress towards the stated goals and towards incubation.
While the project's README indicated active discussions take place in Slack, a quick review of Piraeus slack shows very little discussion on going beyond periodic requests for assistance in the default channel (about once a week). This is not indicative of a growing community.
The project does not appear to hold meetings to engage in synchronous discussions with contributors, adopters, community members, perform project planning, or any other content. If meetings or discussions are happening in other channels, or platforms, the project must update its documentation to reflect these discoverable and public communication mediums.
The project does not appear to engage in any long term planning.
While development is ongoing, there does not appear to be additional planning towards future releases, no open issues tracking progress or discussions on outstanding areas of improvement, etc. It does not appear tags or milestones are used.
issues with the project have been open for an extensive period of time with discussions, if they occurred, remaining unresolved.

Additional recommendations for the project:

review the definition of adopters in the TOC repo and update the adopters listing for this project to ensure those types of adopters are properly captured. A few of the adopters listed do not appear to meet this definition.
Engage with TAG Contributor Strategy to explore opportunities to increase the pool of contributors, identify potential emerging maintainers, improve retention and increase activity of existing contributors
Engage with TAG Storage to determine any additional insights and recommendations the TAG may have in bringing closure to its open goals and work areas.

Add reference in README that you're a CNCF sandbox project

Snapshotter not workin with Kubernetes 1.17

The snapshotter container is complaining:

E0207 08:43:45.373919 1 reflector.go:270] github.com/kubernetes-csi/external-snapshotter/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1alpha1.VolumeSnapshot: unknown (get volumesnapshots.snapshot.storage.k8s.io)

E0207 08:43:45.395867 1 reflector.go:126] github.com/kubernetes-csi/external-snapshotter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:piraeus-csi-controller-sa" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope

The CRDs for the snapshotter are missing. As of Kubernetes 1.17 this should be v1beta1. But may be the Alpha can be used as well if the get installed. I think the CRDs for the snapshotter should get installed with piraeus.

Build multi-arch images

Would like to use images that support multiple architectures for mixed clusters or at least additional images per architecture.

I'd like to contribute this but would need to understand the CI system for this repo.

More info here - https://docs.docker.com/engine/reference/commandline/manifest/

Outdated Readme

Hello

I'm not quite sure whether it's me doing something wrong, or README is outdated, but it says to label desired nodes with piraeus/enabled=true, while spec uses piraeus/node=true as selector

Thanks in advance

Piraeus doesn't support IPv6

Hi,
Piraeus doesn't seem to support IPv6-only clusters.

I've found at least 2 IPv4-only things to fix, but there are likely many more:

diff --git a/dockerfiles/piraeus-init/bin/init-etcd.sh b/dockerfiles/piraeus-init/bin/init-etcd.sh
index 271fa6c..11d026d 100755
--- a/dockerfiles/piraeus-init/bin/init-etcd.sh
+++ b/dockerfiles/piraeus-init/bin/init-etcd.sh
@@ -5,12 +5,19 @@ pod_sts="${POD_NAME/-[0-9]*/}"
 
 cluster=$( seq 0 $(( CLUSTER_SIZE - 1 )) | xargs -tI % echo "${pod_sts}-%=http://${pod_sts}-%.${pod_sts}:${PEER_PORT}" | paste -sd "," - )
 
+if [[ $POD_IP =~ .*:.* ]]; then
+       POD_IP="[$POD_IP]"
+       LOCALHOST='[::1]'
+else
+       LOCALHOST='127.0.0.1'
+fi
+
 # write config file
 cat > /init/etc/etcd/etcd.conf << EOF
 name:                        ${POD_NAME}
 max-txn-ops:                 1024
 listen-peer-urls:            http://${POD_IP}:${PEER_PORT}
-listen-client-urls:          http://${POD_IP}:${CLIENT_PORT},http://127.0.0.1:${CLIENT_PORT}
+listen-client-urls:          http://${POD_IP}:${CLIENT_PORT},http://${LOCALHOST}:${CLIENT_PORT}
 advertise-client-urls:       http://${POD_IP}:${CLIENT_PORT}
 initial-advertise-peer-urls: http://${POD_IP}:${PEER_PORT}
 initial-cluster-token:       ${pod_sts}
diff --git a/dockerfiles/piraeus-init/bin/lib.tools.sh b/dockerfiles/piraeus-init/bin/lib.tools.sh
index 6afea1f..4d30ace 100644
--- a/dockerfiles/piraeus-init/bin/lib.tools.sh
+++ b/dockerfiles/piraeus-init/bin/lib.tools.sh
@@ -1,9 +1,19 @@
 #!/bin/bash
 
 _get_ip_by_if() {
-    ip -f inet address | grep -w "$1" | awk '/inet / {print $2}' | sed 's#/.*##'
+    if [[ $1 =~ .*:.* ]]; then
+        inet=inet6
+    else
+        inet=inet
+    fi
+    ip -f $inet address | grep -w "$1" | awk "/$inet / {print \$2}" | sed 's#/.*##'
 }
 
 _get_if_by_ip() {
-    ip -f inet address | grep -B1 "inet $1" | head -1 | awk '{print $2}' | sed 's/://g'
-}
\ No newline at end of file
+    if [[ $1 =~ .*:.* ]]; then
+        inet=inet6
+    else
+        inet=inet
+    fi
+    ip -f $inet address | grep -B1 "$inet $1" | head -1 | awk '{print $2}' | sed 's/://g'
+}

Cheers

Piraeus supports for arm64

I have tested various functions of piraeus on x86 machines, and I am very excited about its high availability and high performance, but it seems that piraeus only supports x86 now.

I have successfully completed the arm64 images of all projects written by golang, but I cannot make the arm64 versions of the following two images:
quay.io/piraeusdatastore/piraeus-server
quay.io/piraeusdatastore/drbd9

If you have a plan for piraeus to support arm64, I will do my best to help.

drdb is reverted to 8.x version after a reboot

Hello,
After a node restart, the drdb was reverted to the ubuntu default. I just did an apt upgrade and rebooted the node.

quay.io/piraeusdatastore/piraeus-init:v0.5.4 log:

created directory: '/init/tmp'

* Initialize node

<html><title>Linstor REST server</title><body><a href="https://app.swaggerhub.com/apis-docs/Linstor/Linstor/1.0.16">Documentation</a></body></html>

... controller is UP

This node IP: XXXXX@enp35s0

WARN: This node name "worker4" is already registered

Now cluster has nodes:

[

  {

    "name": "worker4",

    "type": "SATELLITE",

    "props": {

      "CurStltConnName": "enp35s0",

      "NodeUname": "worker4"

    },

    "net_interfaces": [

      {

        "name": "enp35s0",

        "address": "XXXXX",

        "satellite_port": 3366,

        "satellite_encryption_type": "PLAIN",

        "is_active": true,

        "uuid": "c6bf14a4-b06a-447b-9f31-b088f5370ad9"

      }

    ],

    "connection_status": "OFFLINE",

    "uuid": "081234c9-2bbc-41c3-b9c6-1f1a79574535",

    "storage_providers": [

      "DISKLESS",

      "FILE",

      "FILE_THIN",

      "OPENFLEX_TARGET"

    ],

    "resource_layers": [

      "CACHE",

      "OPENFLEX",

      "STORAGE"

    ],

    "unsupported_providers": {

      "LVM": [

        "No information from satellite yet"

      ],

      "SPDK": [

        "No information from satellite yet"

      ],

      "ZFS_THIN": [

        "No information from satellite yet"

      ],

      "ZFS": [

        "No information from satellite yet"

      ],

      "LVM_THIN": [

        "No information from satellite yet"

      ]

    },

    "unsupported_layers": {

      "DRBD": [

        "No information from satellite yet"

      ],

      "LUKS": [

        "No information from satellite yet"

      ],

      "NVME": [

        "No information from satellite yet"

      ],

      "WRITECACHE": [

        "No information from satellite yet"

      ]

    }

  }

]

* Enable dm_thin_pool

Module                  Size  Used by

dm_thin_pool           69632  0

DRBD module is already loaded

Module                  Size  Used by

drbd                  364544  0

filename:       /lib/modules/4.15.0-99-generic/kernel/drivers/block/drbd/drbd.ko

alias:          block-major-147-*

license:        GPL

version:        8.4.10

description:    drbd - Distributed Replicated Block Device v8.4.10

author:         Philipp Reisner <[email protected]>, Lars Ellenberg <[email protected]>

srcversion:     7922D81D3881494EB149253

depends:        lru_cache,libcrc32c

retpoline:      Y

intree:         Y

name:           drbd

vermagic:       4.15.0-99-generic SMP mod_unload

sig_id:         PKCS#7

signer:

sig_key:

sig_hashalgo:   unknown

signature:

parm:           allow_oos:DONT USE! (bool)

parm:           disable_sendpage:bool

parm:           proc_details:int

parm:           minor_count:Approximate number of drbd devices (1-255) (uint)

parm:           usermode_helper:string

* Install local linstor cli

'/init/bin/linstor.sh' -> '/opt/piraeus/client/linstor'

'/etc/resolv.conf' -> '/opt/piraeus/client/resolv.conf'

'/usr/local/bin/linstor' -> '/opt/piraeus/client/linstor'

Upgrades packages on main node:

Start-Date: 2020-05-03  01:50:37
Commandline: apt upgrade
Upgrade: update-manager-core:amd64 (1:18.04.11.10, 1:18.04.11.12), libkmod2:amd64 (24-1ubuntu3.2, 24-1ubuntu3.3), kmod:amd64 (24-1ubuntu3.2, 24-1ubuntu3.3), python3-update-manager:amd64 (1:18.04.11.10, 1:18.04.11.12), distro-info-data:amd64 (0.37ubuntu0.6, 0.37ubuntu0.7)
End-Date: 2020-05-03  01:50:38

drdb version on satellite pod:

root@worker4:/# cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 7922D81D3881494EB149253 
root@worker4:/#

GitHub repository does not link to the project website url

As part of our ongoing effort to ensure all the websites within the CNCF meet the CLOMonitor guidelines, we noticed that piraeus does not pass the website criteria on CLOMonitor.

To fix this:
Edit piraeusdatastore/piraeus repository details. See Notary for example:

In the website section, add the link to the piraeus website. See Notary for example:

Once done, feel free to close this issue as CLOMonitor will recrawl the page and update accordingly.

Problems with Ubuntu 22.04 and drbd9-jammy

The drbd9-jammy image for Ubuntu 22.04 does not work because the libelf1 package does not exist. This should be added to the Dockerfile.

Satellite does not support the following layers: [DRBD]

Hello,

I did a install using the helm and tried to initialize a Persistent Volume using the piraeus-dflt-r1 class. But the volume is never created. Analyzing the controller I can see the following error:

Suppressed exception 1 of 2:
===============
Category:                           RuntimeException
Class name:                         OnAssemblyException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at:                       <UNKNOWN>

Error message:                      Assembly site of producer [reactor.core.publisher.MonoFlatMapMany] is identified by light checkpoint [Auto-place resource including thin pools].Error has been observed by the following operator(s):
        |_      Assembly site of producer [reactor.core.publisher.MonoFlatMapMany] is identified by light checkpoint [Auto-place resource].


Error context:
    Satellite 'ubuntu-1804-bionic-64-minimal' does not support the following layers: [DRBD]

Call backtrace:

    Method                                   Native Class:Line number

Suppressed exception 2 of 2:
===============
Category:                           Exception
Class name:                         SnapshotStackException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.SnapshotStackException
Generated at:                       <UNKNOWN>

Error message:                      Auto-place resource

Error context:
    Satellite 'ubuntu-1804-bionic-64-minimal' does not support the following layers: [DRBD]

Call backtrace:

    Method                                   Native Class:Line number


END OF ERROR REPORT.

The host is a metal dedicated server, running the Ubuntu 18.04. Do I need to do any specific configuration to enable piraeus?

Etcd pods fail to install

Pods can't seem to discover each other so they keep on crashing. Since they're being installed under kube-system, perhaps the advertisement should be like piraeus-etcd-2.kube-system instead of piraeus-etcd-2.piraeus-etcd ?

2020-02-09 08:22:09.978560 I | embed: advertise client URLs = http://piraeus-etcd-2.piraeus-etcd:2379
{"level":"warn","ts":1581232959.9791162,"caller":"netutil/netutil.go:121","msg":"failed to resolve URL Host","url":"http://piraeus-etcd-2.piraeus-etcd:2380","host":"piraeus-etcd-2.piraeus-etcd:2380","retry-interval":1,"error":"i/o timeout"}
{"level":"warn","ts":1581232959.9792569,"caller":"netutil/netutil.go:131","msg":"failed to resolve URL Host; returning","url":"http://piraeus-etcd-2.piraeus-etcd:2380","host":"piraeus-etcd-2.piraeus-etcd:2380","retry-interval":1,"error":"i/o timeout"}
2020-02-09 08:22:39.988125 C | etcdmain: failed to resolve http://piraeus-etcd-2.piraeus-etcd:2380 to match --initial-cluster=piraeus-etcd-2=http://piraeus-etcd-2.piraeus-etcd:2380 (failed to resolve "http://piraeus-etcd-2.piraeus-etcd:2380" (i/o timeout))

Schedule only on workers node

Hello,

Thank you for this awesome technology, I´m very excited to try.
I installed using the helm, but one think. It also installed on the control node. I think that would be preferred in install only on nodes with the label node-role.kubernetes.io/worker . Or I´m missing something?

If pvc goes to diskless (node without local storage) pvc goes into error state

If pvc is diskless we got several issues. Like the one bellow.

pvc goes into error state. Reboot of node does not fix this. k8smaster and k8w6 have local physical replica.

k8smaster ┊ pvc-ef126d77-8e99-4607-84e6-7cfa43f5db39 ┊ lvm-hdd ┊ 0 ┊ 1120 ┊ /dev/drbd1120 ┊ 9.05 GiB ┊ InUse ┊ UpToDate ┊
┊ k8w6 ┊ pvc-ef126d77-8e99-4607-84e6-7cfa43f5db39 ┊ lvm-hdd ┊ 0 ┊ 1120 ┊ None ┊ ┊ Unused ┊ Error

lvm not found on node with physical storage

ERROR:
Description:
Node: 'k8w6', resource: 'pvc-ef126d77-8e99-4607-84e6-7cfa43f5db39', volume: 0 - Device provider threw a storage exception
Details:
Command 'blockdev --getsize64 /dev/linstor_lvm-hdd/pvc-ef126d77-8e99-4607-84e6-7cfa43f5db39_00000' returned with exitcode 1.

Standard out:
Error message:
blockdev: cannot open /dev/linstor_lvm-hdd/pvc-ef126d77-8e99-4607-84e6-7cfa43f5db39_00000: No such file or directory

Load average on node goes up significant.

Latest release on quay.io

Hello,
I´m waiting for a new release on quay.io, but the latest version is still 1.7.1
https://quay.io/repository/piraeusdatastore/piraeus-server?tab=history

Is there still a supported location for installation?

snapshot stucks and PVC turns readOnlny

Hi,
we have a MYSQL DB running with a linstore/piraeus PVC and we try to create snapshot for the mysql-data-pvc, but it does not work stable, sometimes it works sometimes not.
Often we have the issue that the "READYTOUSE" (on snapshot) does not turn to "true" and after some time the PVC thats in use by MYSQL turns readonly and the MYSQL-Statefullset turns inactive.
It takes about 20-30mins until the PVC can be used again by the MYSQL-Pod. When we scale down and up, it takes about 20-30 mins until the PVC can be mounted.

We want to use snapshots/cloning on our MYSQL-Service to create backups/dumps of the DB, but this issue happens too often that we cant really use the snapshot/clone features.
We have a script running to create a snapshot and it sends first a "LOCK TABLES FOR BACKUP" to the MYSQL-DB that there are no write operations when a snapshot is taken.

Our clusters are running on rancher
the nodes OS is Ubuntu 18.04, we use LVM as storagepool
we tried it on K8S v1.18.14, after we had this issue, we updated our K8S cluster to v1.19.7 but we have still the same issue.

In the dmesg outputs of a node we could see this when we tried to scale down and up the MYSQL-Statefullset:

[Thu Feb 25 00:23:45 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 1808454609 (2->-1 3/1)
[Thu Feb 25 00:23:47 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Aborting cluster-wide state change 1808454609 (2036ms) rv = -10
[Thu Feb 25 00:23:47 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 3281288772 (2->-1 3/1)
[Thu Feb 25 00:23:49 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Aborting cluster-wide state change 3281288772 (2016ms) rv = -10
[Thu Feb 25 00:23:49 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 3724795384 (2->-1 3/1)
[Thu Feb 25 00:23:51 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Declined by peer node30 (id: 0), see the kernel log there
[Thu Feb 25 00:23:51 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Aborting cluster-wide state change 3724795384 (2016ms) rv = -10
[Thu Feb 25 00:23:51 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 706470052 (2->-1 3/1)
[Thu Feb 25 00:23:53 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Declined by peer node30 (id: 0), see the kernel log there
[Thu Feb 25 00:23:53 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Aborting cluster-wide state change 706470052 (2016ms) rv = -10
[Thu Feb 25 00:23:54 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 3245719846 (2->-1 3/1)
[Thu Feb 25 00:23:56 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Aborting cluster-wide state change 3245719846 (2036ms) rv = -10
[Thu Feb 25 00:23:56 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 1156981054 (2->-1 3/1)
[Thu Feb 25 00:23:58 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Aborting cluster-wide state change 1156981054 (2016ms) rv = -10
[Thu Feb 25 00:23:58 2021] drbd pvc-71e0f8aa-b391-4de0-aaf7-2615056aeb69: Preparing cluster-wide state change 4267462796 (2->-1 3/1)

CNCF TOC annual review due

Annual review due

Sandbox projects are subject to an annual review by the TOC. This is intended to be a lightweight process to ensure that projects are on track, and getting the support they need.

CLOMonitor has detected that the annual review for this project has not been filed yet. CLOMonitor relies on the information in the annual_review_url and annual_review_date fields in the CNCF Landscape configuration file for this check. If your annual review has already been presented, please make sure this information has been correctly added.

For more information about how to file your annual review please see the Sandbox annual review documentation.

Fail to compile drbd module

This is an Ubuntu 18.04 installation. During the init execution, the compilation fails

Linux 4.15.0-99-generic x86_64

* Enable dm_thin_pool

Module                  Size  Used by

dm_thin_pool           69632  0

* Compile and load drbd module by image "quay.io/piraeusdatastore/drbd9-bionic:v9.0.22"

�:Need a git checkout to regenerate drbd/.drbd_git_revision

�:make[1]: Entering directory '/tmp/pkg/drbd-9.0.22-1/drbd'

��

�K    Calling toplevel makefile of kernel source tree, which I believe is in

�.    KDIR=/lib/modules/4.15.0-99-generic/build

��

�Vmake -C /lib/modules/4.15.0-99-generic/build   M=/tmp/pkg/drbd-9.0.22-1/drbd  modules

�Farch/x86/Makefile:157: CONFIG_X86_X32 enabled but no binutils support

�"  COMPAT  before_4_13_kernel_read

�$  COMPAT  alloc_workqueue_takes_fmt

�'  COMPAT  blkdev_issue_zeroout_discard

�$  COMPAT  drbd_release_returns_void

��  COMPAT  genl_policy_in_ops

�#  COMPAT  have_SHASH_DESC_ON_STACK

�!  COMPAT  have_WB_congested_enum

�#  COMPAT  have_allow_kernel_signal

�,  COMPAT  have_atomic_dec_if_positive_linux

�   COMPAT  have_atomic_in_flight

�%  COMPAT  have_bd_unlink_disk_holder

�   COMPAT  have_bd_claim_by_disk

��  COMPAT  have_bio_bi_bdev

��  COMPAT  have_bio_bi_error

��  COMPAT  have_bio_bi_opf

��  COMPAT  have_bio_bi_status

��  COMPAT  have_bio_clone_fast

��  COMPAT  have_bio_flush

��  COMPAT  have_bio_free

��  COMPAT  have_bio_rw

��  COMPAT  have_bio_op_shift

�   COMPAT  have_bio_set_op_attrs

��  COMPAT  have_bioset_init

�"  COMPAT  have_blk_queue_flag_set

�!  COMPAT  have_bioset_need_bvecs

�!  COMPAT  have_blk_queue_plugged

�!  COMPAT  have_blk_check_plugged

�"  COMPAT  have_blkdev_get_by_path

�'  COMPAT  have_bioset_create_front_pad

�%  COMPAT  have_blk_queue_split_q_bio

�%  COMPAT  have_blk_queue_write_cache

�,  COMPAT  have_blk_queue_split_q_bio_bioset

�4  COMPAT  have_generic_start_io_acct_q_rw_sect_part

��  COMPAT  have_d_inode

�$  COMPAT  have_blk_queue_merge_bvec

�2  COMPAT  have_generic_start_io_acct_rw_sect_part

�(  COMPAT  have_genl_family_parallel_ops

��  COMPAT  have_ib_cq_init_attr

��  COMPAT  have_file_inode

�%  COMPAT  have_blk_qc_t_make_request

��  COMPAT  have_prandom_u32

��  COMPAT  have_nla_put_64bit

��  COMPAT  have_idr_alloc

�!  COMPAT  have_max_send_recv_sge

�(  COMPAT  have_pointer_backing_dev_info

�$  COMPAT  have_nla_parse_deprecated

��  COMPAT  have_ib_get_dma_mr

��  COMPAT  have_idr_is_empty

��  COMPAT  have_req_nounmap

�%  COMPAT  have_nla_nest_start_noflag

�$  COMPAT  have_ratelimit_state_init

�"  COMPAT  have_proc_create_single

��  COMPAT  have_req_write_same

�!  COMPAT  have_netlink_cb_portid

�#  COMPAT  have_ktime_to_timespec64

��  COMPAT  have_shash_desc_zero

�%  COMPAT  have_security_netlink_recv

��  COMPAT  have_refcount_inc

��  COMPAT  have_kvfree

�$  COMPAT  have_rb_augment_functions

��  COMPAT  have_inode_lock

��  COMPAT  have_signed_nla_put

�#  COMPAT  have_req_op_write_zeroes

�!  COMPAT  have_req_op_write_same

��  COMPAT  have_req_noidle

��  COMPAT  have_req_prio

��  COMPAT  have_req_op_write

��  COMPAT  have_req_hardbarrier

��  COMPAT  have_simple_positive

��  COMPAT  have_req_write

�   COMPAT  have_struct_bvec_iter

��  COMPAT  have_timer_setup

�4  COMPAT  hlist_for_each_entry_has_three_parameters

�#  COMPAT  ib_alloc_pd_has_2_params

��  COMPAT  ib_device_has_ops

��  COMPAT  have_time64_to_tm

�'  COMPAT  have_struct_kernel_param_ops

�!  COMPAT  have_void_make_request

�'  COMPAT  ib_query_device_has_3_params

�   COMPAT  kmap_atomic_page_only

�$  COMPAT  ib_post_send_const_params

�/  COMPAT  queue_limits_has_discard_zeroes_data

�&  COMPAT  need_make_request_recursion

�$  COMPAT  rdma_create_id_has_net_ns

�/  COMPAT  sock_create_kern_has_five_parameters

�$  COMPAT  sock_ops_returns_addr_len

�7  CHK     /tmp/pkg/drbd-9.0.22-1/drbd/compat.4.15.18.h

�7  UPD     /tmp/pkg/drbd-9.0.22-1/drbd/compat.4.15.18.h

�/  CHK     /tmp/pkg/drbd-9.0.22-1/drbd/compat.h

�/  UPD     /tmp/pkg/drbd-9.0.22-1/drbd/compat.h

�M./drbd-kernel-compat/gen_compat_patch.sh: line 12: spatch: command not found

�K./drbd-kernel-compat/gen_compat_patch.sh: line 45: hash: spatch: not found

�>  INFO: no suitable spatch found; trying spatch-as-a-service;

�(  be patient, may take up to 10 minutes

�@  if it is in the server side cache it might only take a second

�,  SPAAS    0b6a78a7dfa6879aa5fa45d03a5e4869

�P  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

�N                                 Dload  Upload   Total   Spent    Left  Speed

��
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 17660    0 13011    0  4649  78379  28006 --:--:-- --:--:-- --:--:--  103k

�E  You can create a new .tgz including this pre-computed compat patch

��  by calling "make unpatch ; echo drbd-9.0.22-1/drbd/drbd-kernel-compat/cocci_cache/0b6a78a7dfa6879aa5fa45d03a5e4869/compat.patch >>.filelist ; make tgz"

��  PATCH

��patching file ./drbd_int.h

��patching file drbd_bitmap.c

�#patching file drbd_transport_tcp.c

��patching file drbd_nla.c

��patching file drbd_main.c

��patching file drbd_debugfs.c

��patching file drbd_nl.c

��patching file drbd_req.c

��patching file drbd_receiver.c

�3patching file drbd-headers/linux/genl_magic_func.h

�6  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_dax_pmem.o

�5  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_debugfs.o

�4  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_bitmap.o

�2  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_proc.o

�4  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_sender.o

�6  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_receiver.o

�1  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_req.o

�4  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_actlog.o

�2  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_main.o

�2  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/lru_cache.o

�5  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_strings.o

�0  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_nl.o

�6  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_interval.o

�3  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_state.o

�I  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd-kernel-compat/drbd_wrappers.o

�1  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_nla.o

�;  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_transport_tcp.o

�3  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/kref_debug.o

�7  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_transport.o

�8  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_kref_debug.o

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�7  GEN     /tmp/pkg/drbd-9.0.22-1/drbd/drbd_buildtag.c

�6  CC [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd_buildtag.o

�Mobjdump: symbol lookup error: objdump: undefined symbol: cplus_mangle_opname

�-  LD [M]  /tmp/pkg/drbd-9.0.22-1/drbd/drbd.o

�7ld: symbol lookup error: ld: undefined symbol: xmemdup

�Zscripts/Makefile.build:578: recipe for target '/tmp/pkg/drbd-9.0.22-1/drbd/drbd.o' failed

�<make[3]: *** [/tmp/pkg/drbd-9.0.22-1/drbd/drbd.o] Error 127

�;make[2]: *** [_module_/tmp/pkg/drbd-9.0.22-1/drbd] Error 2

�NMakefile:1577: recipe for target '_module_/tmp/pkg/drbd-9.0.22-1/drbd' failed

�0Makefile:125: recipe for target 'kbuild' failed

��make[1]: *** [kbuild] Error 2

�9make[1]: Leaving directory '/tmp/pkg/drbd-9.0.22-1/drbd'

��make: *** [module] Error 2

�0Makefile:125: recipe for target 'module' failed

��make -C drbd install

�:make[1]: Entering directory '/tmp/pkg/drbd-9.0.22-1/drbd'

�FNo .drbd_kernelrelease found. Do you need to 'make' the module first?

��make[1]: *** [install] Error 1

�1Makefile:205: recipe for target 'install' failed

�9make[1]: Leaving directory '/tmp/pkg/drbd-9.0.22-1/drbd'

�1Makefile:129: recipe for target 'install' failed

��make: *** [install] Error 2

�amodprobe: FATAL: Module drbd_transport_tcp not found in directory /lib/modules/4.15.0-99-generic

��

�#Could not load DRBD kernel modules

* Install local linstor cli

'/init/bin/linstor.sh' -> '/opt/piraeus/client/linstor'

'/etc/resolv.conf' -> '/opt/piraeus/client/resolv.conf'

'/usr/local/bin/linstor' -> '/opt/piraeus/client/linstor'

Etcd can't start (failed to resolve)

Hi,
I'm trying to test piraeus on kops-created cluster (kubernetes 1.16.8)

After kubectl apply -f https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml

etcd can't start

piraeus-controller-0       0/1     Init:0/1   0          9m34s
piraeus-csi-controller-0   5/5     Running    0          9m33s
piraeus-csi-controller-1   5/5     Running    0          9m33s
piraeus-csi-controller-2   5/5     Running    0          9m33s
piraeus-csi-node-4l9j6     2/2     Running    0          9m33s
piraeus-csi-node-dcdhf     2/2     Running    0          9m33s
piraeus-csi-node-gjxsn     2/2     Running    0          9m33s
piraeus-etcd-0             0/1     Running    6          9m34s
piraeus-etcd-1             0/1     Running    6          9m34s
piraeus-etcd-2             0/1     Error      6          9m34s
piraeus-node-hg7bs         0/1     Init:0/1   0          9m34s
piraeus-node-rxf6b         0/1     Init:0/1   0          9m34s
piraeus-node-vlh8b         0/1     Init:0/1   0          9m34s
piraeus-scaler-5bv8h       0/1     Init:0/1   0          9m34s
piraeus-scaler-cn2hw       0/1     Init:0/1   0          9m34s
piraeus-scaler-spvkr       0/1     Init:0/1   0          9m34s

Errors in etcd

{"level":"warn","ts":1590732738.2479806,"caller":"netutil/netutil.go:121","msg":"failed to resolve URL Host","url":"http://piraeus-etcd-0.piraeus-etcd:2380","host":"piraeus-etcd-0.piraeus-etcd:2380","retry-interval":1,"error":"lookup piraeus-etcd-0.piraeus-etcd on 100.64.0.10:53: no such host"}

piraeusdatastore / piraeus Goto Github PK

piraeus's People

Contributors

Stargazers

Watchers

Forkers

piraeus's Issues

Annual review due

Recommend Projects

Recommend Topics

Recommend Org