l3af-project / ebpf-package-repository Goto Github PK

@dthaler on 12 Jan
Not sure this is a good thing.

@jniesz jniesz 26 days ago Member
I think it probably makes sense to investigate different chaining semantics based on underlying platform support / requirements in the future

          why the inconsistent types?  i.e. why not struct in_addr in line 41?

Originally posted by @dthaler in #65 (comment)

Address the build warnings and eliminate any unused functions.

• IP header length is variable (20 to 60 bytes) due to the options field which is optional.
• So L4 offset is not always fixed at the default offset of 20 bytes (struct iphdr) from the start of IP header.
• Actual L4 offset is derived from IHL field (4 times IHL) in the IP header, which includes the options as well.

error
make -C /usr/src/linux/samples/bpf/../../tools/lib/bpf/ RM='rm -rf' LDFLAGS= srctree=/usr/src/linux/samples/bpf/../../ O=
make[1]: Entering directory '/usr/src/linux/tools/lib/bpf'
HOSTCC fixdep.o
HOSTLD fixdep-in.o
LINK fixdep
make[1]: Leaving directory '/usr/src/linux/tools/lib/bpf'
make -C /usr/src/linux/ /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/ BPF_SAMPLES_PATH=/usr/src/linux/samples/bpf
make[1]: Entering directory '/usr/src/linux'
CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
make -C /usr/src/linux/samples/bpf/../../tools/lib/bpf/ RM='rm -rf' LDFLAGS= srctree=/usr/src/linux/samples/bpf/../../ O=
HOSTCC fixdep.o
HOSTLD fixdep-in.o
LINK fixdep
HOSTCC /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.o
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c: In function ‘rtattr_add’:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:87:28: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
87 | struct rtattr *d = data;
| ^~~~
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c: In function ‘update_ports’:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:729:9: warning: variable ‘temp’ set but not used [-Wunused-but-set-variable]
729 | int temp = -555;
| ^~~~
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c: In function ‘update_proto’:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:760:9: warning: variable ‘temp’ set but not used [-Wunused-but-set-variable]
760 | int temp = -555;
| ^~~~
At top level:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:712:13: warning: ‘bin’ defined but not used [-Wunused-function]
712 | static void bin(unsigned n)
| ^~~
In function ‘rtattr_add’,
inlined from ‘do_route’ at /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:122:5:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:89:20: warning: ‘__builtin_memcpy’ forming offset [4, 15] is out of the bounds [0, 4] of object ‘nic_id’ with type ‘int’ [-Warray-bounds]
89 | *ptr++ = *d++;
| ~~~~~~~^~~~~~
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c: In function ‘do_route’:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:98:42: note: ‘nic_id’ declared here
98 | _inet_addr *gateway_ip, int nic_id)
| ~~~~^~~~~~
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c: In function ‘log_timestamp’:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:351:41: warning: ‘%06ld’ directive output may be truncated writing between 6 and 20 bytes into a region of size between 0 and 63 [-Wformat-truncation=]
351 | snprintf(log_ts, TIMESTAMP_LEN, "%s.%06ld", tmbuf, tv.tv_usec);
| ^~~~~
In file included from /usr/include/stdio.h:894,
from /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:18:
/usr/include/aarch64-linux-gnu/bits/stdio2.h:71:10: note: ‘__builtin___snprintf_chk’ output between 8 and 85 bytes into a destination of size 64
71 | return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
72 | __glibc_objsize (__s), __fmt,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
73 | __va_arg_pack ());
| ~~~~~~~~~~~~~~~~~
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c: In function ‘nl_update_route’:
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:225:16: warning: ‘nl_flags’ may be used uninitialized in this function [-Wmaybe-uninitialized]
225 | send_ret = do_route(socket, nl_command, nl_flags, &remote_ip, &gateway_ip,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
226 | r.nic_id);
| ~~~~~~~~~
/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring.c:225:16: warning: ‘nl_command’ may be used uninitialized in this function [-Wmaybe-uninitialized]
HOSTCC /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/../../bpf_load.o
HOSTLD /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring
CLANG-bpf /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring_ingress_kern.o
Error: Unsupport signed division for DAG: 0xfc5efe8: i64 = sdiv 0xfc63c48, Constant:i64<2>Please convert to unsigned div/mod.
LLVM ERROR: Cannot select: 0xfc5efe8: i64 = sdiv 0xfc63c48, Constant:i64<2>
0xfc63c48: i64 = sra 0xfc63eb8, Constant:i64<32>
0xfc63eb8: i64 = shl 0xfc5ee48, Constant:i64<32>
0xfc5ee48: i64,ch = load<(load (s32) from %ir.68, !tbaa !10), anyext from i32> 0xfbea648, 0xfc5f328, undef:i64
0xfc5f328: i64,ch = CopyFromReg 0xfbea648, Register:i64 %4
0xfc5f390: i64 = Register %4
0xfc5f8d8: i64 = undef
0xfc63e50: i64 = Constant<32>
0xfc63e50: i64 = Constant<32>
0xfc63a40: i64 = Constant<2>
In function: _ingress_redirect
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0. Program arguments: llc -march=bpf -filetype=obj -o /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring_ingress_kern.o

1. Running pass 'Function Pass Manager' on module ''.
2. Running pass 'BPF DAG->DAG Pattern Instruction Selection' on function '@_ingress_redirect'
   #0 0x0000ffff8ea7ac40 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xe3ac40)
   #1 0x0000ffff8ea78c40 llvm::sys::RunSignalHandlers() (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xe38c40)
   #2 0x0000ffff8ea7b200 (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xe3b200)
   #3 0x0000ffff93e7e7dc (linux-vdso.so.1+0x7dc)
   #4 0x0000ffff8d80f200 (/lib/aarch64-linux-gnu/libc.so.6+0x7f200)
   #5 0x0000ffff8d7ca67c gsignal (/lib/aarch64-linux-gnu/libc.so.6+0x3a67c)
   #6 0x0000ffff8d7b7130 abort (/lib/aarch64-linux-gnu/libc.so.6+0x27130)
   #7 0x0000ffff8e9baf50 llvm::report_fatal_error(llvm::StringRef, bool) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xd7af50)
   #8 0x0000ffff8f1d0ac0 llvm::SelectionDAGISel::isOrEquivalentToAdd(llvm::SDNode const*) const (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x1590ac0)
   #9 0x0000ffff8f1d0034 llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*, unsigned int) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x1590034)
   #10 0x0000ffff908faef0 (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x2cbaef0)
   #11 0x0000ffff8f1c9cf4 llvm::SelectionDAGISel::DoInstructionSelection() (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x1589cf4)
   #12 0x0000ffff8f1c94e4 llvm::SelectionDAGISel::CodeGenAndEmitDAG() (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x15894e4)
   #13 0x0000ffff8f1c8b20 llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x1588b20)
   #14 0x0000ffff8f1c6c58 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x1586c58)
   #15 0x0000ffff8edcbf28 llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0x118bf28)
   #16 0x0000ffff8ebb0710 llvm::FPPassManager::runOnFunction(llvm::Function&) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xf70710)
   #17 0x0000ffff8ebb72f0 llvm::FPPassManager::runOnModule(llvm::Module&) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xf772f0)
   #18 0x0000ffff8ebb1118 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib/aarch64-linux-gnu/libLLVM-14.so.1+0xf71118)
   #19 0x000000000040f93c main (/usr/lib/llvm-14/bin/llc+0x40f93c)
   #20 0x0000ffff8d7b73fc (/lib/aarch64-linux-gnu/libc.so.6+0x273fc)
   #21 0x0000ffff8d7b74cc __libc_start_main (/lib/aarch64-linux-gnu/libc.so.6+0x274cc)
   #22 0x000000000040aff0 _start (/usr/lib/llvm-14/bin/llc+0x40aff0)
   Aborted (core dumped)
   make[2]: *** [/usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/Makefile:100: /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/mirroring_ingress_kern.o] Error 134
   make[1]: *** [Makefile:1737: /usr/src/linux/samples/bpf/eBPF-Package-Repository/traffic-mirroring/] Error 2
   make[1]: Leaving directory '/usr/src/linux'
   make: *** [Makefile:64: build] Error 2
   11

It is necessary to upload artifacts, and as such, upload-artifact actions should be added to each ebpf program CI pipeline.

Remove hard coding of map paths in eBPF programs

The current Traffic Mirroring implementation only allows for GUE tunnels, but there is a need to support other linux interface tunnels, such as GRE. The user should be allowed to configure any Linux tunnels that are suitable for their specific requirements.

Modify existing eBPF programs (tc-root, xdp-root, ratelimiting, connection limiting) for l3afd v2.0.0 native go chaining

@dthaler 16 days ago
Per discussion on line 119, maybe add the word "Public" here, since the L3AFD daemon can work with other package repositories too. Indeed, if there is any actual "code" or scripts for such a repository, then we should separate implementation from policy. I.e., implementation would be a repo that could be used by anyone when creating their own package repository, and policy is anything specifically for the L3AF Public eBPF Package Repository that may not apply to local repositories (e.g., the fact that you're proposing having to contribute source code instead of byte code is in my opinion, something that varies by repository).

@jniesz jniesz 12 days ago Member
I think adding public makes sense here and for the l3af public eBPF package repository IMO it makes sense to require source code

@dthaler dthaler 11 days ago
Here’s my issues with requiring source code for all packages in the L3AF repo:

If the authoritative version of ebpf source is already in some other existing repo, such as it is for Cilium, Calico, etc. today, then adding a copy under L3AF has both efficiency issues (why copy? Which one is authoritative? How do you submit a patch?), as well as possible governance issues (is the licensing model compatible? Is the CLA/DCO process compatible? Is the security vulnerability reporting process compatible? Etc.) since it’s in two different open source governance organizations.
If the tooling requires source code, rather than byte code, then there can be cases where the tooling won’t work for use in enterprise-private package repos if enterprises want to use their repo to contain both OSS packages and commercial packages they may not have source code for, so that they can use common L3AF tooling and program lifecycle management for all their ebpf needs. This would result in multiple package repo implementations instead of reuse.
Using byte code in the repository, with recommending a pointer to where source can be found, would avoid both types of issue above. This is why I think the initial version should do that instead of source.

@jniesz jniesz 11 days ago Member
My comment was really around the initial version and the limited set of (seed) programs added to this repo. I don't think it is feasible to review / accept eBPF packages by l3af maintainers without the source code.

Once there are more checks and gates in place, I don't think we should require source code and byte code alone would be sufficient.

@dthaler dthaler 11 days ago
I don't think it is feasible to review / accept eBPF packages by l3af maintainers without the source code.

My opinion is that manual review is not scalable. If the "public" (or whatever) repo is limited to only what can be done via manual review, then I would expect it will always only contain a small number of small programs, and additions will take a long time to get reviewed, and the reviewers will have a long backlog of requests, and people will get displeased with the length of time it takes to get anywhere, and go elsewhere to a different "public" repo that is automated and does not attempt to make promises about security vetting, but leaves that responsibility to those pulling from the repo, and reinforce the point that one should never point l3afd at the "public" repo only at one's own vetted repo (which would contain a small number of programs by comparison). Personally I would rather have the initial version do the latter rather than doing something that requires much more effort and still doesn't solve the security concerns.

@jniesz jniesz 11 days ago Member
I agree that manual review is not scalable and not something we should offer to the public until we have the automation in place. What if we keep the name "L3AF eBPF Package Repository" repository for now and then change it to "Public L3AF eBPF Package Repository" when we are ready with the automation? I think this is a good initial place to store the build artifacts (byte code) for several of the eBPF programs contributed already under L3AF (xdp-root, ratelimiting, connection-limit). In my opinion it seems cleaner to store these here vs. in the l3afd repository. When we are ready to open this to the public they will already be added and can be reference examples for other folks to contribute their own eBPF packages to this repo.

@dalalkaran dalalkaran 10 days ago Member
I think it is important to distinguish between the initial and future version spec of the "eBPF Package Repository".

In the initial version spec, the proposal is to simply have a GitHub repository to store eBPF program source code. This is to make sure that all the contributed eBPF programs can be made available in a single repository. The proposal acknowledges that the initial version won't have automation in place, and so all submissions will be manually reviewed and published in this repo. The review process will also ensure that the code submissions conform to L3AF's eBPF program chaining mechanics. In this context, it may not be possible to review and accept eBPF packages without source code.

The future version builds on top of the initial version and aims to include a fully automated build-release system that can perform reviews and build/upload the package to a storage repo. The proposal for future version discusses the option of allowing contributors to upload the eBPF package without sharing the source code. It also aims to address various other challenges such as security (vulnerability detection, signing, semantic code analysis) and portability. The eBPF Package Repository Committee will discuss the future spec in detail to draw agreements and can choose to add the word "public" to "eBPF Package Repository".

Hope this helps clarify!

xdp_root.c currently calls the set_link_xdp_fd() libbpf api.

However, per https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L864 that API has been deprecated since libbpf 0.8 and bpf_xdp_attach() should be used instead.

When we run the make command to compile the binary, we encounter a "multiple declaration" error.

Define detailed approach to

1. Signing of the kernel eBPF program (i.e., byte code),
2. Signing of the eBPF programs package (i.e., an archive containing the built eBPF program and any associated user space programs), and
3. Signing for Hypervisors (i.e., Virtualization-based Security).

Add new workflow to enable automated reviews.

Currently, tc-root can be attached to the interface whose name contains only alphabets or digits. However, in case of the hypervisors, tap interface generally contains a hyphen. Hence, we need to fix traffic mirroring code and allow interface names with hyphens as well.

This issue to track the storage requirements like blob or object storage to upload eBPF packages.

A priority for future versions of the repository would be to improve the user experience.

Some examples:

• Frontend website
• Searchable
• Rating system
• Reviews
• Buying and selling
• TLS
• Oauth

Hi Team,
This is little different from this repo, but I thought since you are also working on ebpf hence might've seen something similar.
I'm trying to achieve port routing(route packet simply from 8000 -> 8001) for tcp packets at xdp hook. Below is the logic i used:

---- filtered the packet and reached at tcp header  then:
  __u64 des =tcp->dest;
          __u64 clientport =tcp->source;
          

          if (tcp->dest == ntohs(8000) && tcp->source == ntohs(8001)) {

            bpf_trace_printk("tcp port becomes CLIENT port\n");
                        tcp->dest = clientport;
                         
          }
          else if (tcp->dest == ntohs(8000) && tcp->source == clientport) {
            bpf_trace_printk("tcp port 8001\n");
            
           
            tcp->dest = ntohs(8001);
          }
          tcp->source = ntohs(8000);
         
          tcp->check = tcp_csum(tcp);
// then on return statement:
    
         return XDP_PASS;

When I run this code before connection establishing, i get below output in wireshark:

Then i stopped the code and ran after establishing the connection, sent some packet and then:

What am I doing wrong? If someone can help here.

EPR has workflows for each folders (i.e. eBPF Programs). Combine all these workflows in a single yaml file.

As l3afd attempts to chain the ebpf program to the connection limit, it is unable to read the next program's FD from the xdp_cl_ingress_next_prog map.

console output -
2023-05-12T17:09:33Z WRN failed to verify program ID /sys/fs/bpf/xdp_cl_ingress_next_prog error="get program by id: no such file or directory" 2023-05-12T17:09:33Z WRN failed to fetch the program ID, retrying after a second ... 2023-05-12T17:09:34Z WRN failed to verify program ID /sys/fs/bpf/xdp_cl_ingress_next_prog error="get program by id: no such file or directory" 2023-05-12T17:09:34Z WRN failed to fetch the program ID, retrying after a second ...

The traffic-mirroring program does not currently support IPv6. This issue will be used to track this limitation and possible future work to add IPv6 support.

Fix for Leaking File Descriptors In eBPF user Programs.

Support BPF CO-RE to improve eBPF compatibility on Linux

In order to maintain a timestamp and the total number of connections received in a given period of time, ratelimiting uses the rl_window_map of type BPF_MAP_TYPE_HASH. The user program deletes stale map entries based on the timestamp at which the map element was created. This method can be found in the user program here.
If the type of map has changed from BPF_MAP_TYPE_HASH to BPF_MAP_TYPE_LRU_HASH, the kernel will handle the deletion, and the user program will not be required to perform it.

XDP support for Windows

The eBPF programs are built with kernel source version 5.1, which must be modified to build with version 5.15.

• xdp and tc root programs
• Ratelimiting
• Conetion limit
• Ipfix
• Traffic Mirroring
• examples

1. Avoiding ntohs() operation for every incoming packet

• We read the source and destination port from the packet and change the byte order before doing look-up. We can do this operation while updating the port in the MAP, so that we can avoid it doing for every packet. ntohs() is done twice for every packet.

2. Performing Bitwise operation instead of Modulo

• Even or odd can be identified by doing bitwise & with 1, instead of modulo operation which involves division and getting the reminder i.e requires more operations. (((*ifany) >> (0)) % 2 == 1) is done thrice for every packet if the variable 'ifany' is configured.

• Several files have at the top:

// SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)

but then at the bottom have:

char _license[] SEC("license") = "GPL";

which is a contradiction. From https://github.com/iovisor/bpf-docs/blob/master/bpf_helpers.rst#license I believe it should be

char _license[] SEC("license") = "Dual BSD/GPL";

for dual-licensed code.

• In addition, xdp-root\xdp_pass1.c lines 3-13 contain a copyright/license notice that only mentions GPL. I just want to make sure that the entire file is indeed dual-licensed, rather than that being an error. Is it?

• Finally, the ip-flow-exporter programs are GPL-2.0 where all the others are dual licensed. Can ip-flow-exporter be dual-licensed as well?

Currently XDP programs are only built for Linux, but ebpf-for-wIndows supports XDP programs as well.
Since we want L3AF to be cross-platform, this issue tracks adding the ability to build for Windows.
The repository might have some Linux-only programs, some cross-platform programs, and theoretically even some Windows-only programs.
So we need a way to know for each program, which platforms are supported, and then build those programs on each platform.

1. Normal mode - All the verification will be done outside the l3afd
2. Trusted mode - L3afd will verify the signature.

This option should be configurable in l3afd (i.e., l3afd.cfg).

On connection limiting:
i want to close the connection once it exceeds a limit instead of dropping. I tried this by sending FIN packet from server side to initiate connection close.
If i set FIN flag on the packet and do XDP_TX if does as below:

See the info part in screenshot, Here 172.17.0.4 is Client and 172.17.0.1 is LB.
When at first the Syn packet came from 4 -> 1 , the server got it and replied to LB with Ack and then LB routes that to the client with fin flag set(so see the Syn, Ack and Fin flag set in next line).

Then further the client send the Ack back for which server send RST flag. So this happens many times both ways.
Connection does close but not gracefully it seems.

What is a good way to close few connections on reaching certain limit?

Also check the __license C variable if it exists.