hwameistor / hwameistor Goto Github PK

View Code? Open in Web Editor NEW

511.0 8.0 73.0 136.19 MB

Hwameistor is an HA local storage system for cloud-native stateful workloads.

Home Page: https://hwameistor.io/

License: Apache License 2.0

Go 98.33% Makefile 0.80% Dockerfile 0.07% Shell 0.63% Smarty 0.10% Python 0.08%

high-availability k8s-sig-storage kubernetes storage

hwameistor's Introduction

HwameiStor

English | 简体中文

HwameiStor is an HA local storage system for cloud-native stateful workloads. It creates a local storage resource pool for centrally managing all disks such as HDD, SSD, and NVMe. It uses the CSI architecture to provide distributed services with local volumes and provides data persistence capabilities for stateful cloud-native workloads or components.

Current Status

HwameiStor is a Cloud Native Computing Foundation sandbox project.

The latest release of HwameiStor is

Build Status

Release Status

See current releases.

Running Environments

Kubernetes compatibility

kubernetes	v0.4.3	>=v0.5.0	>= 0.13.0
>=1.18&&<=1.20	Yes	No	No
1.21	Yes	Yes	No
1.22	Yes	Yes	No
1.23	Yes	Yes	No
1.24	Yes	Yes	Yes
1.25	No	Yes	Yes
1.26	No	Yes	Yes
1.27	No	No	Yes
1.28	No	No	Yes

Modules and Code

HwameiStor contains several modules:

local-disk-manager
local-storage
scheduler
admission-controller
Evictor
Exporter
HA module installer
Volume Snapshot
Volume Auto Resize
Volume IO Throtting
App Failover
Audit
UI

local-disk-manager

local-disk-manager (LDM) is designed to hold the management of disks on nodes. Other modules such as local-storage can take advantage of the disk management feature provided by LDM. Learn more

local-storage

local-storage (LS) provides a cloud-native local storage system. It aims to provision high-performance persistent LVM volume with local access to applications. Learn more

Scheduler

Scheduler is to automatically schedule a pod to a correct node which has the associated HwameiStor volumes. Learn more

admission-controller

admission-controller is a webhook that can automatically determine which pod uses the HwameiStor volume and, help to modify the schedulerName to hwameistor-scheduler. Learn more

Evictor

Evictor is used to automatically migrate HwameiStor volumes in case of node or pod eviction. When a node or pod is evicted as either Planned or Unplanned, the associated HwameiStor volumes, which have a replica on the node, will be detected and migrated out this node automatically. Learn more

HA module installer

DRBD (Distributed Replicated Block Device) is one of third-party HA modules which the HwameiStor will leverage to provide HA volume. It composed of Linux kernel modules and related scripts to build high available clusters. It is implemented by mirroring the entire device over the network, which can be thought of as a kind of network RAID. This installer can directly install DRBD to a container cluster.

Exporter

Exporter will collect the system metrics including nodes, storage pools, volumes, disks. It supports Prometheus. Learn more

Volume Snapshot

HwameiStor provides the feature of snapshot and restore on the LVM volumes. Currently, the snapshot/restore feature works for LVM non-HA volume. Learn more

Volume Auto Resize

HwameiStor can automatically expand the LVM volume according the pre-defined resize policy. User can define the preferred policy and describe how and when to expand the volume, and HwameiStor will take the policy into effect. Learn more

Volume IO Throtting

HwameiStor can set a maxmium rate (e.g. bandwidth, IOPS) to access a volume. This feature is very important to prevent the Pod from crashing, especially in the low-resource condition. Learn more

App Failover

The feature of failover is to actively help the application to fail over to another health node with the volume replica, and continue the working. Learn more

Audit

HwameiStor provides the information about the resource history, including cluster, node, storage pool, volume, etc. Learn more

UI

HwameiStor provides a friendly UI to the user to operate the cluster. Learn more

Documentation

For full documentation, please see our website hwameistor.io.

For detailed adopters that have HwameiStor deployed in a production environment or a user acceptance testing environment, please check the adopters list.

Build and Preview Website Locally

It's a good practice to build and preview your modifications locally before submit a PR to HwameiStor Website. You can run the following commands to preview locally:

cd docs
npm run start  # for english
npm run start -- --locale cn # for i18n/cn

Change Navigation

To change the navigation (left sidebar) sequence, you can change the content in _category_.json that exists in each folder:

{
  "label": "Installation", // The name displayed on the left sidebar
  "position": 4, // The sequence identifier, 1 is listed at top
  "link": {
    "type": "generated-index",
    "description": "In this section, we will introduce the installation procedure:"
  }
}

For i18n/cn, you can change the nav in current.json:

{
  "sidebar.tutorialSidebar.category.Modules": {
    "message": "Modules", // The section titles displayed on the left sidebar
    "description": "The label for category Modules in sidebar tutorialSidebar"
  },
  "sidebar.tutorialSidebar.category.Modules.link.generated-index.description": {
    "message": "This chapter introduces the following modules included in HwameiStor:", // The description before sub-section cards
    "description": "The generated-index page description for category Modules in sidebar tutorialSidebar"
  }
}

Roadmap

Features	Status	Release	Description
CSI for LVM volume	Completed	v0.3.2	Provision volume with lvm
CSI for disk volume	Completed	v0.3.2	Provision volume with disk
HA LVM Volume	Completed	v0.3.2	Volume with HA
LVM Volume expansion	Completed	v0.3.2	Expand LVM volume capacity online
LVM Volume conversion	Completed	v0.3.2	Convert a non-HA LVM volume to the HA
LVM Volume migration	Completed	v0.4.0	Migrate a LVM volume replica to a different node
Volume Group	Completed	v0.3.2	Support volume group allocation
Disk health check	Completed	v0.7.0	Disk fault prediction, status reporting
LVM HA Volume Recovery	Planned		Recover the LVM HA volume in problem
HwameiStor Operator	Completed	v0.9.0	Operator for HwameiStor install, maintain, etc.
Observability	Completed	v0.9.2	Observability, such as metrics, logs, etc.
Failover	Completed	v0.12.0	Fail over the pod with HwameiStor volume
IO throttling	Completed	v0.11.0	Limit IO bandwidth to access the HwameiStor volume
Disk replacement	Planned		Replace disk which fails or will fail soon
LVM volume auto-expansion	Completed	v0.12.0	Expand LVM volume automatically
LVM volume snapshot	Completed	v0.12.0	Snapshot of LVM volume
LVM volume clone	Completed	v0.13.1	Clone LVM volume
LVM volume thin provision	Unplaned		LVM volume thin provision
LVM volume stripe mode	Unplaned		LVM volume stripe read/write
Data encryption	Unplaned		Data encryption
System Consistency	Planned		Consistent check and recovery from a disaster
Volume backup	Planned		Backup the volume data to remote server and restore
HwameiStor CLI command	Completed	v0.12.4	CLI command is to manage the HwameiStor cluster
HwameiStor GUI	Completed	v0.11.0	Manage the HwameiStor cluster

Community

We welcome contributions of any kind. If you have any questions about contributing, please consult the contributing documentation.

Blog

Please follow our weekly blogs.

Slack

If you want to join the hwameistor channel on CNCF slack, please get invite to CNCF slack and then join the #hwameistor channel.

WeChat

HwameiStor tech-talk group:

Discussion

Welcome to follow our roadmap discussions here

Pull Requests and Issues

Please feel free to raise requests on chats or by a PR.

We will try our best to respond to every issue reported on community channels, but the issues reported here on this repo will be addressed first.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

HwameiStor enriches the CNCF CLOUD NATIVE Landscape.

hwameistor's People

Stargazers

Watchers

Forkers

sun7927 niulechuan ihuibin yulng linqthomas bridgecrew-perf6 angel0507 floatxd alexzhc yunshihang michelle951 windsonsea qiupingdai zhaosimon panpan0000 daocloud-opensource yankay hangscer8 ye-ting leemingeer zgfh setsunakute rokkiter buffalo1024 bzsuni carlory sonic-chain pacoxu zizhufanqing lsq645599166 mengjiao-liu nameyuli jaredtan95 zhuzhenghao fish-pro yanzhifa samzong michael-xing vacant2333 iq-scm aland-zhang panguicai008 lucadev seaiii hadoop835 wawa0210 calvin-puram ssmallmonster yk2012985 gala-rx amazingpangwei luckymrwang yyzxw hikariwo whomobile yanggangtony lanyujiex jeanine-tw jzhupup yojay11717 megabyte875 huoxiaojun2009 fan-lin daocloud xiamuwei peng9808 lihuibng sundegan jiejaitt gamegzh little-guy-lxr

hwameistor's Issues

【New Feature Support】Add Support the Observability Feature to Expose external storage service status and Hard disk throughput or Other indicators

Output of lsblk:

# paste output here

Output of blkid:

# paste output here

Output of kubectl version:

# paste output here

Output of kubectl get lsn -o yaml:

# paste output here

Cloud Provider/Platform (AKS, GKE, Minikube etc.):

# paste output here

Better Readme

The Readme now looks like a template.

We may add the links or remove the useless title.

Uninstall processes

I could not find the uninstall processes in ReadMe, not sure if there is or not?

It is necessary sometimes to remove all the hwamei components completely for testing purpose.

pvc info still remained in lsn after lv/lvr deleted

I created a statefulSet with pods running, and each pod attached with lv/lvr.

After I deleted the statefulSet, pods, and lv/lvrs, the pvc volume info still remained in 'kubectl get lsn -o yaml'

Install hwameiistor in arm environment, local-storage and local-disk-manager will CrashLoopBackOff

[docs] Added a glossary list to keep consistency

Much inconsistency is found in docs. It's time to add a glossary list, so everyone can follow the glossary and reduce confusion. Starting from four modules of HwameiStor:

local-disk-manager
local-storage
scheduler
admission-controller
HwameiStor

Hope we add more day by day.

The node-role.kubernetes.io/master taint is deprecated and will be removed in kubeadm version 1.25

The node-role.kubernetes.io/master taint is deprecated and will be removed in kubeadm version 1.25.

It would be replace by node-role.kubernetes.io/control-plane. like:
https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/manifests/vanilla/vsphere-csi-driver.yaml#L220

[docs] Added a style guide

HwameiStor documentation has three different topics currently. It is recommended to provide a simple style guide if more than 20 topics in a docsite.

Concept
Task (steps)
Reference

References:

[Doc] Need a method of configuring LocalDiskClaim (LDC) in batch

Currently, there are two ways to configure LDC

During installation, by helm chart's storageNodes[]
After installation, by yaml's kind: LocalDiskClaim

Both have drawbacks:

Method 1 is a batch process, but cannot be modified after installation because helm upgrade ignores all CRDs and their instances. Therefore it is not suitable for node expansion and reduction.

Mothod 2 can be configured anytime but is not a batch process, therefore each node needs to be configured with a separate yaml.

We need a new method to configure LDC in batch both during installation and after it.

@SSmallMonster please comment

csi-node-driver-registrar is from k8s.gcr.io

According to the docs : https://kubernetes-csi.github.io/docs/node-driver-registrar.html, csi-node-driver-registrar is from k8s.gcr.io. But the is from the https://github.com/hwameistor/helm-charts/blob/main/charts/hwameistor/values.yaml#L1, It's from the Quay.io .

[docs] Does HwameiStor support OpenShift and Ranch

Need to test and clarify if HwameiStor can work with OpenShift and Ranch besides the Kubernetes.

Need to test if HwameiStor can work with OpenShift and Rancher
Document should be updated with this information.

Application cannot be scheduled normally

Application cannot be scheduled normally

LDC available

[docs] Build up a matrix for HwameiStor installation for different scenarios

There are multiple components in HwameiStor, including LDM, LS, Scheduler, Admission Controller. Had better to build up a matrix to describe the installation options for supporting multiple scenarios. For example:

LDM + LS + Scheduler + Admission Controller
LDM + Scheduler + MinIO
...

Need to mirror image "docker.io/jackmmzhou/self-signed:v1"

hwameistor-webhook uses image "docker.io/jackmmzhou/self-signed:v1" which needs to be mirrored to other repositories for better access:

error:

Failed to pull image "jackmmzhou/self-signed:v1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/jackmmzhou/self-signed:v1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/jackmmzhou/self-signed/manifests/sha256:c1f6eea3c47ed1fdc560530235d2abcf3e27d17664ff7a8b18e9641ff3032982: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[bug]README.md文档描述需要明确nodeName是node的hostname

作为insight tester，我在按照README.md文档### STEP 3: Claim Disk By Type On Node创建LDC时，没有将spec.nodeName设置为node的hostname以至于LDC未到Bound状态；请明确此处设置

Discussion: Should we use config instead of the "node label"

When we install the hwamei stor, the user must to use "node label" to select the node enabled

Should we change it to the config file , like https://github.com/rancher/local-path-provisioner#configuration

Benifit:

The config can be declared in the helm values.
We can use "auto policy" in config file, like " all free disk" , so that the use donot need to write the config mannaly.
Remove the Step in Installtion to make install simpler

fs PVC not working properly with disk without filesystem

PVC won't bound in HA mode

Because we watch DRBD events use nsenter use -t, so host ${PATH} is not working in container:
https://github.com/hwameistor/local-storage/blob/6668a3af1b80a5c2d5c0c499820f74161819adc4/pkg/member/node/configer/drbd.go#L507

Besides we didn't providedrbdsetup in Dockerfile:
https://github.com/hwameistor/local-storage/blob/6668a3af1b80a5c2d5c0c499820f74161819adc4/build/Dockerfile#L1-L7

So we got no DRBD events to fill up m.resourceCache map.
https://github.com/hwameistor/local-storage/blob/6668a3af1b80a5c2d5c0c499820f74161819adc4/pkg/member/node/configer/drbd.go#L507-L508

Residual LocalVolume information causes an error

time="2022-08-09T03:27:30Z" level=debug msg="Finished running command" func=RunCommand file="executor.go:92" args="[--mount=/proc/1/ns/mnt --ipc=/proc/1/ns/ipc --net=/proc/1/ns/net --uts=/proc/1/ns/uts -- lvcreate --size 1073741824B --stripes 1 LocalStorage_PoolHDD -n pvc-4deca8ca-0258-49a6-85b9-3efb5670f7f8 -y]" command=nsenter error="exit status 5" stderr=" Logical Volume "pvc-4deca8ca-0258-49a6-85b9-3efb5670f7f8" already exists in volume group "LocalStorage_PoolHDD"" stdout= timeout=30

go 1.16 would select

Hi, My go version is go version go1.18rc1 darwin/amd64,when I exec 'go mod tidy'(After I clone github.com/hwameistor/local-disk-manager)，I got this:

go: downloading github.com/onsi/ginkgo v1.12.1
go: downloading github.com/onsi/gomega v1.10.1
go: downloading k8s.io/apiextensions-apiserver v0.18.6
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: downloading go.etcd.io/bbolt v1.3.3
go: downloading github.com/gorilla/websocket v1.4.0
go: downloading golang.org/x/lint v0.0.0-20190930215403-16217165b5de
go: downloading golang.org/x/tools v0.1.1
go: downloading golang.org/x/mod v0.4.2
go: downloading sigs.k8s.io/structured-merge-diff v0.0.0-20190525122527-15d366b2352e
go: downloading github.com/docker/docker v1.4.2-0.20200203170920-46ec8731fbce
go: downloading google.golang.org/genproto v0.0.0-20200117163144-32f20d992d24
go: downloading go.etcd.io/bbolt v1.3.4
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.12.1
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.1.0
github.com/hwameistor/local-storage/cmd/scheduler imports
        k8s.io/kubernetes/cmd/kube-scheduler/app imports
        github.com/spf13/cobra loaded from github.com/spf13/[email protected],
        but go 1.16 would select v1.0.0
github.com/hwameistor/local-storage/cmd/scheduler imports
        k8s.io/kubernetes/cmd/kube-scheduler/app imports
        k8s.io/apiserver/pkg/util/term imports
        github.com/docker/docker/pkg/term loaded from github.com/docker/[email protected],
        but go 1.16 would select v1.4.2-0.20200203170920-46ec8731fbce
github.com/hwameistor/local-storage/pkg/member/csi imports
        github.com/container-storage-interface/spec/lib/go/csi imports
        google.golang.org/grpc/status imports
        google.golang.org/genproto/googleapis/rpc/status loaded from google.golang.org/[email protected],
        but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/component-base/metrics imports
        github.com/blang/semver loaded from github.com/blang/[email protected]+incompatible,
        but go 1.16 would select v3.5.1+incompatible
github.com/hwameistor/local-storage/cmd/scheduler imports
        k8s.io/kubernetes/cmd/kube-scheduler/app imports
        k8s.io/apiserver/pkg/server/routes imports
        k8s.io/kube-openapi/pkg/handler imports
        github.com/NYTimes/gziphandler loaded from github.com/NYTimes/[email protected],
        but go 1.16 would select v1.1.1
github.com/hwameistor/local-storage/cmd/scheduler imports
        k8s.io/kubernetes/cmd/kube-scheduler/app imports
        k8s.io/apiserver/pkg/util/term imports
        github.com/docker/docker/pkg/term imports
        github.com/docker/docker/pkg/term/windows loaded from github.com/docker/[email protected],
        but go 1.16 would select v1.4.2-0.20200203170920-46ec8731fbce
github.com/hwameistor/local-storage/pkg/member/csi imports
        github.com/container-storage-interface/spec/lib/go/csi imports
        google.golang.org/grpc/codes tested by
        google.golang.org/grpc/codes.test imports
        google.golang.org/genproto/googleapis/rpc/code loaded from google.golang.org/[email protected],
        but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/member/csi imports
        github.com/container-storage-interface/spec/lib/go/csi imports
        google.golang.org/grpc/status tested by
        google.golang.org/grpc/status.test imports
        google.golang.org/genproto/googleapis/rpc/errdetails loaded from google.golang.org/[email protected],
        but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        github.com/coreos/pkg/capnslog loaded from github.com/coreos/[email protected],
        but go 1.16 would select v0.0.0-20180928190104-399ea9e2e55f
github.com/hwameistor/local-storage/cmd/alerter imports
        sigs.k8s.io/controller-runtime/pkg/client/config tested by
        sigs.k8s.io/controller-runtime/pkg/client/config.test imports
        sigs.k8s.io/controller-runtime/pkg/log/zap imports
        go.uber.org/zap tested by
        go.uber.org/zap.test imports
        golang.org/x/lint/golint loaded from golang.org/x/[email protected],
        but go 1.16 would select v0.0.0-20191125180803-fdd1cda4f05f
github.com/hwameistor/local-storage/pkg/apis imports
        k8s.io/apimachinery/pkg/runtime imports
        sigs.k8s.io/structured-merge-diff/v3/value imports
        gopkg.in/yaml.v2 tested by
        gopkg.in/yaml.v2.test imports
        gopkg.in/check.v1 imports
        github.com/kr/pretty loaded from github.com/kr/[email protected],
        but go 1.16 would select v0.2.1
github.com/hwameistor/local-storage/cmd/alerter imports
        sigs.k8s.io/controller-runtime/pkg/client/config tested by
        sigs.k8s.io/controller-runtime/pkg/client/config.test imports
        sigs.k8s.io/controller-runtime/pkg/log/zap imports
        go.uber.org/zap tested by
        go.uber.org/zap.test imports
        golang.org/x/lint/golint imports
        golang.org/x/lint loaded from golang.org/x/[email protected],
        but go 1.16 would select v0.0.0-20191125180803-fdd1cda4f05f
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        github.com/grpc-ecosystem/grpc-gateway/runtime loaded from github.com/grpc-ecosystem/[email protected],
        but go 1.16 would select v1.12.1
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        github.com/tmc/grpc-websocket-proxy/wsproxy loaded from github.com/tmc/[email protected],
        but go 1.16 would select v0.0.0-20190109142713-0ad062ec5ee5
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        go.etcd.io/bbolt loaded from go.etcd.io/[email protected],
        but go 1.16 would select v1.3.4
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/etcdserver/api/v3rpc imports
        github.com/grpc-ecosystem/go-grpc-middleware loaded from github.com/grpc-ecosystem/[email protected],
        but go 1.16 would select v1.1.0
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        github.com/grpc-ecosystem/grpc-gateway/runtime imports
        github.com/grpc-ecosystem/grpc-gateway/internal loaded from github.com/grpc-ecosystem/[email protected],
        but go 1.16 would select v1.12.1
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        github.com/grpc-ecosystem/grpc-gateway/runtime imports
        github.com/grpc-ecosystem/grpc-gateway/utilities loaded from github.com/grpc-ecosystem/[email protected],
        but go 1.16 would select v1.12.1
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        github.com/grpc-ecosystem/grpc-gateway/runtime imports
        google.golang.org/genproto/googleapis/api/httpbody loaded from google.golang.org/[email protected],
        but go 1.16 would select v0.0.0-20200117163144-32f20d992d24
github.com/hwameistor/local-storage/pkg/scheduler imports
        k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1 imports
        k8s.io/kubernetes/pkg/controller/volume/scheduling imports
        k8s.io/apiserver/pkg/storage/etcd3 tested by
        k8s.io/apiserver/pkg/storage/etcd3.test imports
        go.etcd.io/etcd/integration imports
        go.etcd.io/etcd/embed imports
        github.com/grpc-ecosystem/grpc-gateway/runtime imports
        google.golang.org/genproto/protobuf/field_mask loaded from google.golang.org/[email protected],
        but go 1.16 would select v0.0.0-20200117163144-32f20d992d24

To upgrade to the versions selected by go 1.16:
        go mod tidy -go=1.16 && go mod tidy -go=1.17
If reproducibility with go 1.16 is not needed:
        go mod tidy -compat=1.17
For other options, see:
        https://golang.org/doc/modules/pruning

It seems right?
What's go version support exactly?

Disucussion: change scheduler-name in deployment yaml is not easy to use

Change scheduler-name in deployment yaml is not easy to use.

If user want to install wordpress from bitnami, he cannot run

helm install wp -n wp5 community/wordpress  --create-namespace

He should run:

helm install wp -n wp5 community/wordpress --set global.imageRegistry=docker.m.daocloud.io --set schedulerName=hwameistor-scheduler  --set mariadb.schedulerName=hwameistor-scheduler  --create-namespace

The config is very diffcult for user.

There are some Chinese in Engish Docs

In the HA volume scenario, only the latest migration target node will be reserved for LVG accessibility:node after LVR migration using localvolumegroupmigrate

Add a file system to a disk that has been managed by LD, and the status of the disk in LD will not change

Localvolumegroupconvert does not work properly

There is a problem with the namespace of CRD in helm chart

localvolumegroupmigrate has no effect on lvg in non-hwameistor namespace

time="2022-06-23T03:10:01Z" level=error msg="addLocalVolume GetLocalVolumeGroupByName err" func=addLocalVolume file="manager.go:523" Module=LocalVolumeGroupManager error="LocalVolumeGroup.hwameistor.io "b9e4e682-2bee-4cec-b4ad-ee228e3603dd" not found" namespace=hwameistor

[docs] refine the "What is HwameiStor"

The content of "What is HwameiStor" should be refined as below:

move out the sections of "Functional modules" and "Terminology";
add the introduction for the HwameiStor's functions.

删除ldc 后，ld 状态还显示为Claimed

ldc 删除后，ld 还显示为Claimed状态，是否类似pv的处理显示未一个特殊状态Released之类的？

Output of kubectl version:

v1.23.1

Output of kubectl get lsn -o yaml:

apiVersion: v1
items:
- apiVersion: hwameistor.io/v1alpha1
  kind: LocalStorageNode
  metadata:
    creationTimestamp: "2022-04-17T08:14:05Z"
    generation: 1
    name: kind-k8s-1-18-worker
    resourceVersion: "279502"
    selfLink: /apis/hwameistor.io/v1alpha1/localstoragenodes/kind-k8s-1-18-worker
    uid: 0a68078b-8a7c-4621-bf14-fc21d5c60c20
  spec:
    hostname: kind-k8s-1-18-worker
    storageIP: 172.18.0.9
    topogoly:
      region: default
      zone: default
  status:
    pools:
      LocalStorage_PoolHDD:
        class: HDD
        freeCapacityBytes: 1178960134144
        freeVolumeCount: 998
        name: LocalStorage_PoolHDD
        totalCapacityBytes: 1181107617792
        totalVolumeCount: 1000
        type: REGULAR
        usedCapacityBytes: 2147483648
        usedVolumeCount: 2
        volumeCapacityBytesLimit: 1181107617792
        volumes:
        - pvc-33dbf764-168f-43df-abcd-88f2c8bac963
        - pvc-675e03fa-ec15-4f38-8f07-e984f61085c8
      LocalStorage_PoolSSD:
        class: SSD
        disks:
        - capacityBytes: 1073737629696
          devPath: /dev/loop1
          state: Available
          type: SSD
        freeCapacityBytes: 1073737629696
        freeVolumeCount: 1000
        name: LocalStorage_PoolSSD
        totalCapacityBytes: 1073737629696
        totalVolumeCount: 1000
        type: REGULAR
        usedCapacityBytes: 0
        usedVolumeCount: 0
        volumeCapacityBytesLimit: 1073737629696
    state: Ready
- apiVersion: hwameistor.io/v1alpha1
  kind: LocalStorageNode
  metadata:
    creationTimestamp: "2022-04-17T08:14:05Z"
    generation: 1
    name: kind-k8s-1-18-worker2
    resourceVersion: "3743"
    selfLink: /apis/hwameistor.io/v1alpha1/localstoragenodes/kind-k8s-1-18-worker2
    uid: 58f151a9-b4de-4d9f-85e5-1fbfa0b6090d
  spec:
    hostname: kind-k8s-1-18-worker2
    storageIP: 172.18.0.7
    topogoly:
      region: default
      zone: default
  status:
    state: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Cloud Provider/Platform (AKS, GKE, Minikube etc.):

kind

lsn status is 'offline'

promotion with awesome list

https://github.com/sindresorhus/awesome
or/and
https://github.com/rootsongjc/awesome-cloud-native

Applications cannot be scheduled normally in a namespace other than hwameistor

[docs] Add the introduction of "admission control" module

A new module of "admission control" is added into the HwameiStor, it's request to keep the document up to date.

Add the brief intro of "admission control" into https://hwameistor.io/docs/intro
Add a new item of the Components section to describe the "admission control"
Add the info about "admission control" into the "Quick Deployment"

Incorrect volume data remains in lsn, resulting in usedVolumeCount inconsistent with reality

After repeatedly installing and uninstalling hwameistor,Incorrect volume data remains in lsn, resulting in usedVolumeCount inconsistent with reality

apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
creationTimestamp: "2022-06-22T09:26:35Z"
generation: 1
managedFields:

apiVersion: hwameistor.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:spec:
.: {}
f:hostname: {}
f:storageIP: {}
f:topogoly:
.: {}
f:region: {}
f:zone: {}
f:status:
.: {}
f:pools:
.: {}
f:LocalStorage_PoolHDD:
.: {}
f:class: {}
f:disks: {}
f:freeCapacityBytes: {}
f:freeVolumeCount: {}
f:name: {}
f:totalCapacityBytes: {}
f:totalVolumeCount: {}
f:type: {}
f:usedCapacityBytes: {}
f:usedVolumeCount: {}
f:volumeCapacityBytesLimit: {}
f:volumes: {}
f:state: {}
manager: local-storage
operation: Update
time: "2022-06-22T11:37:42Z"
name: demo-dev-worker-01
resourceVersion: "88947404"
uid: 09a643a3-229a-4ea6-a227-5bb2a15659f8
spec:
hostname: demo-dev-worker-01
storageIP: 10.6.182.104
topogoly:
region: default
zone: default
status:
pools:
LocalStorage_PoolHDD:
class: HDD
disks:
- capacityBytes: 21470642176
  devPath: /dev/sdc
  state: InUse
  type: HDD
- capacityBytes: 10733223936
  devPath: /dev/sdd
  state: InUse
  type: HDD
- capacityBytes: 214744170496
  devPath: /dev/sde
  state: InUse
  type: HDD
  freeCapacityBytes: 235136876544
  freeVolumeCount: 991
  name: LocalStorage_PoolHDD
  totalCapacityBytes: 246948036608
  totalVolumeCount: 1000
  type: REGULAR
  usedCapacityBytes: 11811160064
  usedVolumeCount: 9
  volumeCapacityBytesLimit: 246948036608
  volumes:
- pvc-b0ddc5e4-eaf7-4579-a102-22ab89adc40c
- pvc-b1821a9b-c0d3-4436-85d1-50f903cf3056
- pvc-0888ad0e-0846-495a-83ed-8ed719a52133
- pvc-19b330e8-56b8-4a54-9330-0a4b53ed70ca
- pvc-25005fb3-18c1-4b43-968b-a1cf164c40b3
- pvc-2a570cad-a440-470b-a124-df46e4622151
- pvc-6831a655-7f75-4e2f-a106-db18cd2ef1ff
- pvc-6f972b09-1394-4a4e-b9ff-4ed4b5b11e9a
- pvc-b0cf7e2f-9e0a-4210-b39a-a96c96d89265
  state: Ready

kubectl get lv
NAME POOL REPLICAS CAPACITY ACCESSIBILITY STATE RESOURCE PUBLISHED AGE
pvc-2a570cad-a440-470b-a124-df46e4622151 LocalStorage_PoolHDD 3 1073741824 Ready 1 demo-dev-worker-01 19h
pvc-6f972b09-1394-4a4e-b9ff-4ed4b5b11e9a LocalStorage_PoolHDD 1 1073741824 Ready -1 demo-dev-worker-01 43h
pvc-b0cf7e2f-9e0a-4210-b39a-a96c96d89265 LocalStorage_PoolHDD 2 1073741824 Ready 3 demo-dev-master-03 15h

helm install error

Hi：
I use helm to deploy hwameister

error log

local-disk-manager:v0.0.2

  Warning  Failed     3m11s (x5 over 4m36s)  kubelet            Error: failed to start container "local-disk-manager": Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/local-disk-manager": stat /local-disk-manager: no such file or directory: unknown

[Doc] Enhance the document's structure

Suggested the new structure of document as below:

What's the HwameiStor
Quick Start
2.1 Installations
2.2 Start a Stateful Application
2.2.1 Basic Operations
2.2.2 Advanced Operations
2.3 Advanced Features
2.3.1 Node Expansion
2.3.2 Disk Expansion
2.3.3 Upgrade
2.4 Uninstall
Architecture of HwameiStor
3.1 Overview
3.2 Components
3.3 Resources
3.4 APIs
Development Designs (Optional)
Applications
5.1 TiDB
5.2 MinIO
...
Terminologies
Community
FAQs

Namespace consistency in HwameiStor Deployment

During HwameiStor deployment, first we need to create HwameiStor system namespace. With Helm Charts deployment the namespace created is 'hwameistor':

https://github.com/hwameistor/helm-charts/blob/main/README.md#step-1-install-hwameistor

While in the code https://github.com/hwameistor/local-storage/blob/main/deploy/01_namespace.yaml, the namespace is 'local-storage-system'.

Suggest to make these 2 in consistent.

Create LDC first and then add disk, but LDC will not be bound

First create the LDC of node1 and then add the disk sdc but the LDC has no bound

The tag of the image in the helm file does not take effect normally

The replicaNumber of LocalVolumeConvert cannot be set to 3

Disks that meet the conditions will be fully reserved

k8s-node1-sdb & k8s-node2-sdd Meet the capacity requirements of PVC

k8s-node2-sdd is not used but reserved by the scheduler

[docs] Prerequisites is too simple

The Prerequisites is too simple.

Should we make it more detail?
Like recommend Disk Size and OS for DRBD, I think it's very important information.

How to clean up the storage volume whose reclaimPolicy is Retain？

sc yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-disk-hdd-disk
provisioner: disk.hwameistor.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
reclaimPolicy: Retain
parameters:
diskType: "HDD"

pvc yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-disk-pvc
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
storageClassName: local-disk-hdd-disk
resources:
requests:
storage: 5Gi

I created a sc whose reclaimPolicy is Retain, and created the corresponding pvc mount deploy.
Then I deleted the deploy, pvc, but because the reclaimPolicy is Retain, the pv is not deleted.
At this time, how can I delete the used storage and release the corresponding local disk？

[bug] setting storageNodes after installation cause all hwameistor pods deleted

Kubernetes version: 1.24.3

After deploying hwameistor with storageNodes left empty, set storageNodes by helm -n hwameistor upgrade hwameistor ./helm/hwameistor --set "storageNodes={k8s-worker-1,k8s-worker-2,k8s-worker-3}".

The command succeeds but then all pods under hwameistor namespace get terminated.

【New Feature Support】Add Support the Observability Feature to Expose external storage service status and Hard disk throughput or Other indicators

Output of lsblk:

# paste output here

Output of blkid:

# paste output here

Output of kubectl version:

# paste output here

Output of kubectl get lsn -o yaml:

# paste output here

Cloud Provider/Platform (AKS, GKE, Minikube etc.):

# paste output here

Show we add Notes in helm like bitnami (https://github.com/bitnami/charts/blob/master/bitnami/nginx/templates/NOTES.txt)
Give the use some suggestion to check the install.

hwameistor / hwameistor Goto Github PK

hwameistor's Introduction

HwameiStor

Current Status

Build Status

Release Status

Running Environments

Kubernetes compatibility

Modules and Code

local-disk-manager

local-storage

Scheduler

admission-controller

Evictor

HA module installer

Exporter

Volume Snapshot

Volume Auto Resize

Volume IO Throtting

App Failover

Audit

UI

Documentation

Build and Preview Website Locally

Change Navigation

Roadmap

Community

Blog

Slack

WeChat

Discussion

Pull Requests and Issues

License

hwameistor's People

Stargazers

Watchers

Forkers

hwameistor's Issues

error log

local-disk-manager:v0.0.2

Recommend Projects

Recommend Topics

Recommend Org