kadalu support for IPv6 host about kadalu HOT 10 OPEN

@vatsa287 please check this issue.

from kadalu.

@Kanaga13579

• Please let me know if Kadalu type is native or external? If it is external, check if gluster volume is up and running & also accessible from kadalu operator pod.
• Please also provide storage-config file used if possible.
• Yes the current socket only supported IPV4 addresses, after using INET6 check locally by logging into csi-pod and calling is_server_pod_reachable(hosts, 24007, 20) with IPV6 host(s).
• If hostname resolution is possible, can use hostnames from kubectl get nodes and try.

In the meantime I'll try to produce locally and update.

from kadalu.

[2023-05-18 08:23:04.762861 +0000] E [MSGID: 101073] [name.c:254:gf_resolve_ip6] 0-resolver: error in getaddrinfo [{family=2}, {ret=Name or service not known}]
[2023-05-18 08:23:04.762874 +0000] E [name.c:383:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 2001:420:284:2004:4:110:122
[2023-05-18 08:23:04.763128 +0000] I [glusterfsd-mgmt.c:2771:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: 2001:420:284:2004:4:110:122

This log is coming from glusterfs, so from my best knowledge the missing part of IPV6 is also from its parsing.
You can also try setting the same with standalone glusterfs with same IPV6 address to double check for it.

The hostname/IP mentioned in storage-config file are seen in kadalu config file and are further used for reaching them.

$ kubectl get configmap -nkadalu kadalu-info -ojson 

Also another way of checking if Kadalu is using the right hostname or not, is by logging into a server-pod,

ps ax | grep "glusterfs"

Check if service details has the right hostname.

from kadalu.

Hi Vatsa,

1. Kadalu type is External. The gluster volume is up and running and is accessible from kadalu operator pod.

# ping 2001:420:284:2004:4:110:122:a131
PING 2001:420:284:2004:4:110:122:a131(2001:420:284:2004:4:110:122:a131) 56 data bytes
64 bytes from 2001:420:284:2004:4:110:122:a131: icmp_seq=1 ttl=63 time=0.384 ms
64 bytes from 2001:420:284:2004:4:110:122:a131: icmp_seq=2 ttl=63 time=0.373 ms
64 bytes from 2001:420:284:2004:4:110:122:a131: icmp_seq=3 ttl=63 time=0.433 ms
64 bytes from 2001:420:284:2004:4:110:122:a131: icmp_seq=4 ttl=63 time=0.460 ms
64 bytes from 2001:420:284:2004:4:110:122:a131: icmp_seq=5 ttl=63 time=0.335 ms
^C
--- 2001:420:284:2004:4:110:122:a131 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4082ms
rtt min/avg/max/mdev = 0.335/0.397/0.460/0.044 ms

2. storage config file used

root@2001-420-284-2004-4-110-122-a131-hybrid:/home/cw-admin# kubectl get kadalu storage-vol1 -oyaml
apiVersion: kadalu-operator.storage/v1alpha1
kind: KadaluStorage
metadata:
  creationTimestamp: "2023-05-18T19:48:50Z"
  generation: 1
  name: storage-vol1
  namespace: default
  resourceVersion: "1759"
  uid: 07484544-8ce7-4679-b130-3e1c6b58b1ed
spec:
  details:
    gluster_hosts:
    - 2001:420:284:2004:4:110:122:a131
    - 2001:420:284:2004:4:110:122:a132
    - 2001:420:284:2004:4:110:122:a133
    gluster_options: log-level=DEBUG
    gluster_volname: glustervol1
  kadalu_format: non-native
  pvReclaimPolicy: retain
  single_pv_per_pool: true
  type: External

3)After adding AF_INET6, IPv6 hosts are reachable.

from kadalu.

[2023-05-18 08:23:04.762861 +0000] E [MSGID: 101073] [name.c:254:gf_resolve_ip6] 0-resolver: error in getaddrinfo [{family=2}, {ret=Name or service not known}]
[2023-05-18 08:23:04.762874 +0000] E [name.c:383:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 2001:420:284:2004:4:110:122
[2023-05-18 08:23:04.763128 +0000] I [glusterfsd-mgmt.c:2771:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: 2001:420:284:2004:4:110:122

This log is coming from glusterfs, so from my best knowledge the missing part of IPV6 is also from its parsing. You can also try setting the same with standalone glusterfs with same IPV6 address to double check for it.

The hostname/IP mentioned in storage-config file are seen in kadalu config file and are further used for reaching them.

$ kubectl get configmap -nkadalu kadalu-info -ojson 

Also another way of checking if Kadalu is using the right hostname or not, is by logging into a server-pod,

ps ax | grep "glusterfs"

Check if service details has the right hostname.

In my case, I dont see any server pods in kadalu namespace.

root@2001-420-284-2004-4-110-122-a131-hybrid:/home/cw-admin# kubectl get po -n kadalu
NAME                          READY   STATUS    RESTARTS   AGE
kadalu-csi-nodeplugin-2769n   3/3     Running   0          11h
kadalu-csi-nodeplugin-9ngp5   3/3     Running   0          11h
kadalu-csi-nodeplugin-ggbm4   3/3     Running   0          11h
kadalu-csi-nodeplugin-k7bll   3/3     Running   0          11h
kadalu-csi-provisioner-0      5/5     Running   0          5h11m
operator-67df4b9df9-v26m6     1/1     Running   0          11h

In this case, how to check this command
ps ax | grep "glusterfs"

Would really appreciate your help on this.

from kadalu.

root@2001-420-284-2004-4-110-122-a131-hybrid:/home/kubectl exec -it kadalu-csi-provisioner-0 -c kadalu-provisioner -n kadalu -- sh
# cd opt/etc
# ls
bash_completion.d  ganesha  glusterfs
# cd glusterfs
# ls
eventsconfig.json         glusterd.vol                group-db-workload       group-metadata-cache  group-virt.example   thin-arbiter.vol
gluster-rsyslog-5.8.conf  glusterfs-georep-logrotate  group-distributed-virt  group-nl-cache        gsyncd.conf
gluster-rsyslog-7.2.conf  glusterfs-logrotate         group-gluster-block     group-samba           logger.conf.example
# cat glusterd.vol
volume management
    type mgmt/glusterd
    option working-directory /opt/var/lib/glusterd
    option transport-type socket
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
    option transport.socket.listen-port 24007
    option ping-timeout 0
    option event-threads 1
#   option lock-timer 180
    option transport.address-family inet6
#   option base-port 49152
    option max-port  60999
end-volume

we have edited this file to accept inet6(option transport.address-family inet6), but looks like glusterfs is not taking this configuration.
Any idea how to restart glusterfs service here to pickup the changes?

from kadalu.

If Kadalu Storage Pool is created with external volume, server-pods will not be created.
Can check the service details in the node where brick process is running.

@aravindavk Maybe we can update Glusterfs if there are new fixes for IPV6 mounts.

from kadalu.

Hi @vatsa287 ,
The reported issue got fixed by adding xlator-option=transport.address-family=inet6 in gluster_options in kadalu storage yaml.

Example:

---
apiVersion: kadalu-operator.storage/v1alpha1
kind: KadaluStorage
metadata:
  name: kadalu-external-storage
spec:
  type: External
  details:
    gluster_hosts:
      - gluster1.kadalu.io
    gluster_volname: kadalu
    gluster_options: log-level=DEBUG,xlator-option=transport.address-family=inet6

So the following command gets executed at gluster end.
/opt/sbin/glusterfs --process-name fuse -l /var/log/gluster/gluster.log --volfile-id glustervol2 --volfile-server 2001:420:284:2004:4:110:122:a131:24007 --volfile-server 2001:420:284:2004:4:110:122:a132:24007 --volfile-server 2001:420:284:2004:4:110:122:a133:24007 --log-level=DEBUG /mnt/storage-vol2 --xlator-option=transport.address-family=inet6

Thanks kadalu team for your timely support!

from kadalu.

Above will address the issue as long as the other code changes where it does check the connectivity use the INET6, as have to enable the glusterconfig in the base image to support v6. Is it possible to provide a release for V6 ?

from kadalu.

@Kanaga13579 cool!!
@rajtupakula sure we will do that!

from kadalu.