[K3S] SMF/UPF Containers crash about towards5gs-helm HOT 18 CLOSED

Okay thank you,

And do not forget to make the n6netowk parameters (subnetIP, cidr, gateway) compliant with the ones of enp6s0 interface

from towards5gs-helm.

@EmanueleGallone, please consider using microk8s which is super easy to configure, including multus, and was verified for this set of helm charts.

from towards5gs-helm.

Hi @rajewluk thank you but my idea is to integrate this deployment within an existing infrastracture which uses K3S. At least, with the kind help of @raoufkh we've narrowed down the problem to Multus-CNI, which I hope to find a fix soon.

from towards5gs-helm.

I managed to make the Multus-CNI work within a subset of the K3S cluster's nodes by following this comment
k8snetworkplumbingwg/multus-cni#564 (comment)

The towards5gs-helm chart works like a charm!

Closing the issue!

from towards5gs-helm.

Hi @EmanueleGallone

We are currently using the MACVLAN plugin for all interfaces added by using multus. Each MACVLAN interface will enslave an interface on the host as described here.

By default, the n6 interface on the UPF POD will expect the existence of an interface called eth1 on the host to be created correctly. If you have only one interface on the worker node that hosts the UPF, you should see this section.

from towards5gs-helm.

Hi @raoufkh

Thanks for the clarification. I'm actually using a kubernetes cluster (K3S instance) running on multiple physical machines. I was able to retrieve the interface that provides internet connection (enp6s0 in my case) on the worker node in which the UPF pod is being deployed. I edited the /charts/free5gc/values.yml, editing the masterIf:

  n6network:
    enabled: true
    name: n6network
    masterIf: enp6s0
    subnetIP: 10.100.100.0
    cidr: 24
    gatewayIP: 10.100.100.1
    excludeIP: 10.100.100.254

and deployed the chart through helm, to no avail.
The same errors occur. The UPF and SMF pods keep on crashing.

Edit:
The AMF pod which uses the N2 interface runs perfectly, without editing the values.yml file, even though it's deployed on the same worker node as the UPF

from towards5gs-helm.

Hello again

• You say the AMF pod runs perfectly, so I assume that the corresponding k8s node has an interface called eth0 because this is the one which is enslaved by default by the AMF's n2 interface. Is the N3IWF running too? Can you provide the status of AMF and N3IWF pods using the kubectl describe pod <pod_name> command please?
• For the SMF and UPF, can you provide more information about the status of these pods by running these commands (for both SMF and UPF):

1. kubectl describe pod <pod name>
2. kubectl get pod -o yaml <pod name>

from towards5gs-helm.

Hi,

Yes, the AMF and N3IWF are running just fine.
The following list of files contains all the requested output. Let me know if you need anything else. Much obliged for your support.

AMF

1. amf.txt
2. amf.yml.txt

N3IWF

1. n3iwf.txt
2. n3iwf.yml.txt

SMF

1. smf.txt
2. smf.yml.txt

UPF

1. upf.txt
2. upf.yml.txt

from towards5gs-helm.

Can you kill the UPF pod by kubectl delete pod and then resend the results for the new pod (another UPF pod will be created to replace the deleted one). Because it seems that your pod is to old and I can only see this event which has been repreduced many times.
Warning BackOff 14s (x534 over 115m) kubelet Back-off restarting failed container

In addition, can you provide the logs from the multus pod that runs on the UPF node (you should do this after the first step such as we can see the new events).

Another k8s resource type to check is network-attachment-definition.

1. kubectl get network-attachment-definition ==> you should have n6network and n4network in the list (tell me only if it exists or not)
2. kubectl describe network-attachment-definition n6network, kubectl describe network-attachment-definition n4network and kubectl describe network-attachment-definition n4network ==> results in files please

from towards5gs-helm.

N6network is present within the network-attachment-definition list.
N6: n6network.txt
N4: n4network.txt

UPF:

1. upf.txt
2. upf.yml.txt

I think that multus is the culprit. I'm detecting a lot of restart events, even though the multus pods keep running
Multus logs: multus-pod.txt

from towards5gs-helm.

On the n6network.txt file, I'm seeing that eth2 is used as a master interface. Does this interface exist on the node? I assume that the response would be no
You should use this configuration when installing the helm chart because enp6s0 is the interface that enables to reach the data network (internet):

...
  n6network:
    enabled: true
    name: n6network
    masterIf: enp6s0
    subnetIP: 10.100.100.0
    cidr: 24
    gatewayIP: 10.100.100.1
...

with further customizartion:

• subnetIP and cidr should be the same as the subnet enp6s0 interface belongs to
• gatewayIP should be the IP address of the gateway on the subnet enp6s0 belongs to (not the IP address of enp6s0 but its gateway)

from towards5gs-helm.

Sorry about the eth2; it was a typo from an older trial; I've already tried the configuration

  n6network:
    enabled: true
    name: n6network
    masterIf: enp6s0
    subnetIP: 10.100.100.0
    cidr: 24
    gatewayIP: 10.100.100.1

I just re-installed the chart through helm with the right interface but the same errors occur

from towards5gs-helm.

Can you provide the new logs of multus, upf and smf please?
And the description of n4netowk and n2network to compare them (because for n2network there is no problem)

from towards5gs-helm.

Multus logs: multus.txt
N6network: n6network.txt

kubectl get logs free5gc-upf: upf-logs.txt
kubectl describe pod free5gc-upf : upf-describe.txt

kubectl get logs free5gc-smf: smf-logs.txt
kubectl describe pod free5gc-smf: smf-describe.txt

from towards5gs-helm.

the description of n4netowk and n2network please as for n2network there is no problem (I'll try to compare)

from towards5gs-helm.

N2net: n2network.txt

N3net: n3network.txt

N4net: n4network.txt

Edit: if you're wondering, yes, I already tried the eth0 in N6network, but it didn't work

from towards5gs-helm.

It seems like the configuration of networks is good. It may be an issue f using multus with k3s.
Please see this issue on the multus repository. There may be useful comments for you

from towards5gs-helm.

I'll take a look on that issue. Thank you. I'll keep this issue open until I find a way to solve it and post the solution

from towards5gs-helm.