Code Monkey home page Code Monkey logo

libbpf-rs's Introduction

libbpf-rs's People

Contributors

alexgartrell avatar anakryiko avatar arilou avatar brianc118 avatar chengshuyi avatar chenhengqi avatar d-e-s-o avatar danielocfb avatar danobi avatar davide125 avatar dependabot[bot] avatar germancoding avatar heyrutvik avatar insearchoflosttime avatar jfernandez avatar kckeiks avatar loshz avatar mendess avatar michel-slm avatar mimullin-bbry avatar rbartlensky avatar tchebb avatar thisseanzhang avatar ueno avatar weizhang555 avatar willfindlay avatar yan-ace62 avatar yunbo-xufeng avatar yunwei37 avatar yuval-k avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libbpf-rs's Issues

libbpf-cargo tests do not work outside of the git repo

Looks like the tests assume they're being run in the git repo, where the libpf-rs sources are one level up in path. This is not the case when building from the crate (e.g. when packaging this in Fedora).

     Running `/usr/bin/rustc --crate-name cargo_libbpf --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no --test -C metadata=0eca454eb3e06fda -C extra-filename=-0eca454eb3e06fda --out-dir /builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps -L dependency=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps --extern anyhow=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libanyhow-f52d6e2d8b872bc9.rlib --extern cargo_metadata=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libcargo_metadata-201286cef2c0f393.rlib --extern goblin=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libgoblin-0ee0c613fd9ed025.rlib --extern libbpf_cargo=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/liblibbpf_cargo-555dbf843feae080.rlib --extern libbpf_sys=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/liblibbpf_sys-03751eec0e5e8d63.rlib --extern memmap=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libmemmap-6b80da01316dd249.rlib --extern num_enum=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libnum_enum-92ef6d87fed9e797.rlib --extern regex=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libregex-e5e411dbb90962d5.rlib --extern scroll=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libscroll-ae68cbd995482dd6.rlib --extern scroll_derive=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libscroll_derive-58cac732bd0f41b2.so --extern semver=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libsemver-e24cc60083843439.rlib --extern serde=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libserde-a2f925a616df0a1a.rlib --extern serde_json=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libserde_json-f64fb15ed2340581.rlib --extern structopt=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libstructopt-0605240d071edd38.rlib --extern tempfile=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libtempfile-b2db7dd4d4b67661.rlib --extern thiserror=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libthiserror-7848cb14d8adb2c2.rlib -Copt-level=3 -Cdebuginfo=2 -Clink-arg=-Wl,-z,relro,-z,now -Ccodegen-units=1 --cap-lints=warn -L native=/usr/lib64 -L native=/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/build/libbpf-sys-96b26f105dc6b24f/out`
    Finished release [optimized] target(s) in 35.15s
     Running `/builddir/build/BUILD/libbpf-cargo-0.7.0/target/release/deps/libbpf_cargo-9b34da430701d3e1`

running 32 tests
test build::test_extract_version ... ok
test test::test_btf_dump_definition_anon_enum ... ok
test test::test_btf_dump_definition_datasec_multiple ... ok
test test::test_btf_dump_definition_datasec ... ok
test test::test_btf_dump_basic_long_array ... ok
test test::test_btf_dump_basic ... ok
test test::test_btf_dump_definition_datasec_long_array ... ok
test test::test_btf_dump_definition_bitfield_struct_fails ... ok
test test::test_btf_dump_definition_datasec_multiple_long_array ... ok
test test::test_btf_dump_definition_packed_struct ... ok
test test::test_btf_dump_definition_shared_dependent_types ... ok
test test::test_btf_dump_definition_packed_struct_long_array ... ok
test test::test_btf_dump_definition_enum ... ok
test test::test_btf_dump_definition_struct_inner_anon_union ... ok
test test::test_btf_dump_definition_struct_inner_anon_struct ... ok
test test::test_btf_dump_definition_struct_inner_anon_struct_and_union ... ok
test test::test_btf_dump_definition_union ... ok
test test::test_build_workspace_collision ... ok
test test::test_build_custom ... ok
test test::test_build_invalid_prog ... ok
test test::test_build_workspace ... ok
test test::test_build_default ... ok
test test::test_enforce_file_extension ... ok
test test::test_btf_dump_struct_definition ... ok
test test::test_skeleton_builder_clang_opts ... ok
test test::test_btf_dump_struct_definition_long_array ... ok
libbpf: elf: skipping unrecognized data section(13) .eh_frame
libbpf: elf: skipping relo section(14) .rel.eh_frame for section(13) .eh_frame
libbpf: elf: skipping unrecognized data section(13) .eh_frame
libbpf: elf: skipping relo section(14) .rel.eh_frame for section(13) .eh_frame
libbpf: elf: skipping unrecognized data section(13) .eh_frame
libbpf: elf: skipping relo section(14) .rel.eh_frame for section(13) .eh_frame
libbpf: elf: skipping unrecognized data section(13) .eh_frame
libbpf: elf: skipping relo section(14) .rel.eh_frame for section(13) .eh_frame
test test::test_skeleton_builder_basic ... FAILED
libbpf: elf: skipping unrecognized data section(14) .eh_frame
libbpf: elf: skipping relo section(15) .rel.eh_frame for section(14) .eh_frame
libbpf: elf: skipping unrecognized data section(14) .eh_frame
libbpf: elf: skipping relo section(15) .rel.eh_frame for section(14) .eh_frame
test test::test_make_basic has been running for over 60 seconds
test test::test_make_workspace has been running for over 60 seconds
test test::test_skeleton_basic has been running for over 60 seconds
test test::test_skeleton_datasec has been running for over 60 seconds
test test::test_skeleton_empty_source has been running for over 60 seconds
test test::test_make_basic ... ok
test test::test_make_workspace ... ok
test test::test_skeleton_empty_source ... FAILED
test test::test_skeleton_basic ... FAILED
test test::test_skeleton_datasec ... FAILED

failures:

---- test::test_skeleton_builder_basic stdout ----
clang is version 12.0.0
Building /tmp/.tmpZwVfSx/proj/src/bpf/prog.bpf.c
Warning: unrecognized map: .maps
thread 'test::test_skeleton_builder_basic' panicked at 'failed to canonicalize libbpf-rs: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/test.rs:112:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- test::test_skeleton_empty_source stdout ----
Metadata for package=proj
	null
Found bpf progs to compile:
	UnprocessedObj { package: "proj", path: "/tmp/.tmp3rhryw/proj/src/bpf/prog.bpf.c", out: "/tmp/.tmp3rhryw/proj/target/bpf", name: "prog" }
Building /tmp/.tmp3rhryw/proj/src/bpf/prog.bpf.c
Metadata for package=proj
	null
Found bpf objs to gen skel:
	UnprocessedObj { package: "proj", path: "/tmp/.tmp3rhryw/proj/src/bpf/prog.bpf.c", out: "/tmp/.tmp3rhryw/proj/target/bpf", name: "prog" }
thread 'test::test_skeleton_empty_source' panicked at 'failed to canonicalize libbpf-rs: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/test.rs:112:10

---- test::test_skeleton_basic stdout ----
Metadata for package=proj
	null
Found bpf progs to compile:
	UnprocessedObj { package: "proj", path: "/tmp/.tmpf9kfDw/proj/src/bpf/prog.bpf.c", out: "/tmp/.tmpf9kfDw/proj/target/bpf", name: "prog" }
Building /tmp/.tmpf9kfDw/proj/src/bpf/prog.bpf.c
Metadata for package=proj
	null
Found bpf objs to gen skel:
	UnprocessedObj { package: "proj", path: "/tmp/.tmpf9kfDw/proj/src/bpf/prog.bpf.c", out: "/tmp/.tmpf9kfDw/proj/target/bpf", name: "prog" }
Warning: unrecognized map: .maps
thread 'test::test_skeleton_basic' panicked at 'failed to canonicalize libbpf-rs: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/test.rs:112:10

---- test::test_skeleton_datasec stdout ----
Metadata for package=proj
	null
Found bpf progs to compile:
	UnprocessedObj { package: "proj", path: "/tmp/.tmpcwUAUo/proj/src/bpf/prog.bpf.c", out: "/tmp/.tmpcwUAUo/proj/target/bpf", name: "prog" }
Building /tmp/.tmpcwUAUo/proj/src/bpf/prog.bpf.c
Metadata for package=proj
	null
Found bpf objs to gen skel:
	UnprocessedObj { package: "proj", path: "/tmp/.tmpcwUAUo/proj/src/bpf/prog.bpf.c", out: "/tmp/.tmpcwUAUo/proj/target/bpf", name: "prog" }
thread 'test::test_skeleton_datasec' panicked at 'failed to canonicalize libbpf-rs: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/test.rs:112:10


failures:
    test::test_skeleton_basic
    test::test_skeleton_builder_basic
    test::test_skeleton_datasec
    test::test_skeleton_empty_source

test result: FAILED. 28 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 126.61s

Skeleton: No default implementation for generated enum types

Using an enum as a member of a struct results in a compilation error due to the generated skeleton not implementing default for the enum.

I'll use the same minimal example as last time, but modified slightly:

#include "vmlinux.h"

#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

enum test {
    TEST1 = 0x1,
    TEST2 = 0x2,
    TEST3 = 0x4,
};

struct foo {
    enum test test;
};

struct foo foo = {};

SEC("tp_btf/execve")
int BPF_PROG(trace_execve, char *filename, pid_t pid, pid_t old_pid)
{
    return 0;
}

This produces the following skeleton:

// ....
pub mod enum_bss_types {
    #[derive(Debug, Copy, Clone)]
    #[repr(C)]
    pub struct bss {
        pub foo: foo,
    }
    #[derive(Debug, Default, Copy, Clone)]
    #[repr(C)]
    pub struct foo {
        pub test: test,
    }
    #[derive(Debug, Copy, Clone, PartialEq)]
    #[repr(u32)]
    pub enum test {
        TEST1 = 1,
        TEST2 = 2,
        TEST3 = 4,
    }
}
// ....

As you can see, test does not implement Default and therefore #[derive(Default)] on the foo causes a compilation error.

Improper dropping of libbpf_rs::perf_buffer::CbStruct? Segfaulting

I've been messing with the library for a while. I have a few thoughts on things that could potentially be improved, but I'll probably just submit PRs for those (like looping over programs and maps in a BPF object using bpf_map__next and bpf_program__next so you don't need to know the program names, I'm not using any of the CO-RE functionality). But I've hit a segfaulting bug that I'm not completely sure how to fix so I thought I'd submit an issue.

Here's my code that opens the object, loads it, grabs the perf buffer, and then tries to create and populate an Array map with create_syscall_filter_map.

        let file = Path::new("syscall_snoop.bpf.o");

        let perf_name = "syscallsnoop_events";

        let open_obj = ObjectBuilder::default().open_file(file)?;
        let mut object = open_obj.load()?;

        let perf_map = object.map(perf_name)?
                       .ok_or_else(|| anyhow!("Can't fetch perf map '{}'", perf_name))?;

        let perf_buf = PerfBufferBuilder::new(perf_map)
                        .sample_cb(handle_syscall_event)
                        .lost_cb(handle_lost_events)
                        .build()?;

        create_syscall_filter_map(&mut object, syscalls_to_trace)?;

And here's the filter map function.

fn create_syscall_filter_map(object: &mut libbpf_rs::Object, syscalls: Vec<&str>) -> Result<()> {

    let sys_filter_map =  object.map("sys_filter_map")?
                        .ok_or_else(|| anyhow!("[!] Can't get pid map"))?;

    for syscall in syscalls {

        let pre_key = get_sys_id(syscall)?;
        let key = pre_key.to_le_bytes();

        let val = 1u32.to_le_bytes();

        sys_filter_map.update(&key, &val, MapFlags::ANY)?;
    }

    Ok(())
}

In this function I access a HashMap. The segfault occurs when I try to access an invalid key in the HashMap. In that case I've set my errors to back-propagate all the way to main, which then exits with an error.
Well, that's what I expected. What actually happens is a segfault. Here's the gdb trace of the segfault

Program received signal SIGSEGV, Segmentation fault.
__GI___libc_free (mem=0x1) at malloc.c:3102
3102	malloc.c: No such file or directory.

Instead of diving deeper into the gdb output, I decided to run valgrind since it was a memory issue. Here's the output

==43153== Invalid free() / delete / delete[] / realloc()
==43153==    at 0x483CA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==43153==    by 0x19942D: alloc::alloc::dealloc (alloc.rs:104)
==43153==    by 0x19981C: <alloc::alloc::Global as core::alloc::Allocator>::deallocate (alloc.rs:239)
==43153==    by 0x19952B: alloc::alloc::box_free (alloc.rs:334)
==43153==    by 0x19ABA0: core::ptr::drop_in_place (mod.rs:179)
==43153==    by 0x19B917: <libbpf_rs::perf_buffer::CbStruct as core::ops::drop::Drop>::drop (/home/vagrant/.cargo/registry/src/github.com-1ecc6299db9ec823/libbpf-rs-0.7.0/src/perf_buffer.rs:22)
==43153==    by 0x1703AE: core::ptr::drop_in_place (mod.rs:179)
==43153==    by 0x1702FB: core::ptr::drop_in_place (mod.rs:179)
==43153==    by 0x1709A0: core::ptr::drop_in_place (mod.rs:179)
==43153==    by 0x1814F9: syscalls::sys_snoop::SyscallSnooper::new (src/sys_snoop.rs:66)
==43153==    by 0x139AFE: syscalls::main (src/main.rs:43)
==43153==    by 0x13848A: core::ops::function::FnOnce::call_once (function.rs:227)
==43153==  Address 0x1 is not stack'd, malloc'd or (recently) free'd
==43153==

This points to libbpf_rs::perf_buffer::CbStruct::drop not functioning correctly in this case.
If I change the order of my code, then I can avoid the segfault.

        let file = Path::new("syscall_snoop.bpf.o");

        let perf_name = "syscallsnoop_events";

        let open_obj = ObjectBuilder::default().open_file(file)?;
        let mut object = open_obj.load()?;

        // now call this prior to getting the perf buffer
        create_syscall_filter_map(&mut object, syscalls_to_trace)?;

        let perf_map = object.map(perf_name)?
                       .ok_or_else(|| anyhow!("Can't fetch perf map '{}'", perf_name))?;

        let perf_buf = PerfBufferBuilder::new(perf_map)
                        .sample_cb(handle_syscall_event)
                        .lost_cb(handle_lost_events)
                        .build()?;

I took a look at how Drop is implemented for CbStruct and it's just casting the raw pointers to Box with the expectation that it will then be cleaned up properly I believe. I'm not completely sure why this segfault is occurring, but perhaps there is a safer way to deal with dropping/freeing the CbStruct such that it can contain the unsafe code better? I wish I had a solution to this to share, but in my own libbpf Rust library I've worked on I actually stored the functions in a Box initially and then coerced into c_void pointers when needed and that's how I avoided this issue.

[Bug: libbpf-cargo] Structures with Inner unions not correctly constructed

The following Test Case wil fail


#[test]
fn test_btf_dump_definition_struct_inner_union() {
    let (_dir, proj_dir, cargo_toml) = setup_temp_project();

    // Add prog dir
    create_dir(proj_dir.join("src/bpf")).expect("failed to create prog dir");

    // Add a prog
    let mut prog = OpenOptions::new()
        .write(true)
        .create(true)
        .open(proj_dir.join("src/bpf/prog.bpf.c"))
        .expect("failed to open prog.bpf.c");

    write!(
        prog,
        r#"
        #include "vmlinux.h"
        #include "bpf_helpers.h"

        struct Foo {{
            int x;
            union Bar {{
                u8 y[10];
                u16 z[16];
            }};
        }};

        struct Foo foo;
        "#,
    )
    .expect("failed to write prog.bpf.c");

    // Lay down the necessary header files
    add_bpf_headers(&proj_dir);

    // Build the .bpf.o
    assert_eq!(
        build(true, Some(&cargo_toml), Path::new("/bin/clang"), true),
        0
    );

    let obj = OpenOptions::new()
        .read(true)
        .open(proj_dir.as_path().join("target/bpf/prog.bpf.o").as_path())
        .expect("failed to open object file");
    let mmap = unsafe { Mmap::map(&obj) }.expect("Failed to mmap object file");
    let btf = Btf::new("prog", &*mmap)
        .expect("Failed to initialize Btf")
        .expect("Did not find .BTF section");

    assert!(btf.types().len() > 0);

    // Find our struct
    let mut struct_foo: Option<u32> = None;
    for (idx, ty) in btf.types().iter().enumerate() {
        match ty {
            btf::BtfType::Struct(t) => {
                if t.name == "Foo" {
                    assert!(struct_foo.is_none()); // No duplicates
                    struct_foo = Some(idx.try_into().unwrap());
                }
            }
            _ => (),
        }
    }

    assert!(struct_foo.is_some());

    let foo_defn = r#"#[derive(Debug, Default, Copy, Clone)]
#[repr(C)]
pub struct Foo {
    pub x: i32,
}
"#;
    assert_ne!(
        foo_defn,
        btf.type_definition(struct_foo.unwrap())
            .expect("Failed to generate union Foo defn")
    );
}

I am not sure what the output should be, but it should definitely not be the value contained in foo_defn

Add skeleton support

The goal of libbpf-rs skeleton is exactly the same as libbpf skeleton: to reduce programming errors and streamline boilerplate.

To that end, there are multiple pieces that need to fit together. In order, they are:

  • object file bundling
    • remove need to ship around .o files
    • add a cargo libbpf make subcommand to run: cargo libbpf build && cargo libbpf gen && cargo build
      • reduce chance that user forgets to rebuild object and regen skeleton
  • map and prog bindings
    • decide skeleton accessor API
    • obj.maps().name_of_my_map()
    • obj.progs().my_prog()
  • use libbpf skeleton API for loading object
    • don't want to duplicate the mmap'ing code
    • punting on that for now b/c
      • requires generating a lot of unsafe code in skeleton
      • prevents us from storing extra state in libbpf map/prog/link objects (b/c the ptr -> struct constructor wouldn't take extra state)
      • doesn't save us that much duplication
  • automatic prog attachment and link storage
    • give user an API to have skeleton attach all progs and store the links inside the skeleton
      • that way user doesn't need to hold onto links and links will simply be detached when skeleton is dropped
    • use libbpf skeleton load API
  • mmap'd datasec fields
    • support .rodata/.data/.bss section global variables as field accesses
    • generate a #[repr(C)] struct that's allocated behind a Box and provide a &mut T accessor to the struct
      • don't give user access to the Box so they can't invalidate the mmap
    • obj.bss().my_var = 123

Feature request: looping over programs in loaded Object

I think it would be really handy for non-CO-RE programs to have something like the libbpf function and macro that allows for looping over all the available programs in a BPF object. Currently I don't think the progs Vec is populated when an Object is created, so you have to search for the program(s) by name. Because you have to know the program names ahead of time, you can't just load and attach programs generically.

I would guess that you could implement an iterator such that you could do the following code

for program in bpf_object.programs {
    // do something
 }

The iterator would just call bpf_program__next instead of trying to grab the next from the progs vec.

There might be some naming issues that would need to be addressed if you assume that the progs vec is empty by default though.

I thought about just implementing this and making a PR, but I want to see the receptiveness (since this library is "opinionated") and if any of the actual maintainers want to do this themselves.

bpf_printk: breaking skeleton generation

The following patch breaks compilation of examples

diff --git a/examples/runqslower/src/bpf/runqslower.bpf.c b/examples/runqslower/src/bpf/runqslower.bpf.c
index cf821da..eca67ff 100644
--- a/examples/runqslower/src/bpf/runqslower.bpf.c
+++ b/examples/runqslower/src/bpf/runqslower.bpf.c
@@ -56,6 +56,7 @@ int handle__sched_wakeup(u64 *ctx)
        /* TP_PROTO(struct task_struct *p) */
        struct task_struct *p = (void *)ctx[0];

+       bpf_printk("this causes an error, but shouldn't");
        return trace_enqueue(p->tgid, p->pid);
 }

The following is observed during a cargo build


   Compiling capable v0.1.0 (/code/libbpf-rs-mainline/examples/capable)
error: failed to run custom build command for `runqslower v0.1.0 (/code/libbpf-rs-mainline/examples/runqslower)`

Caused by:
  process didn't exit successfully: `/code/libbpf-rs-mainline/target/debug/build/runqslower-b81ac9d4cca6e1bc/build-script-build` (exit status: 101)
  --- stderr
  Warning: unrecognized map: .maps
  Warning: unrecognized map: license
  error: expected `:`, found `.`
     --> <stdin>:167:29
      |
  167 |     pub handle__sched_wakeup.____fmt: [i8; 36],
      |                             ^ expected `:`

  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Generate("Failed to generate skeleton for /tmp/.tmpizMHT9/runqslower.o: Failed to rustfmt: exit status: 1")', examples/runqslower/build.rs:19:47
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

Feature Request: Expose libbpf-cargo BPF object and skeleton generation code as a library

Crates like bindgen often expose code generation functionality via a builder pattern which can then be easily integrated into a project's build.rs file. I have found myself wanting to do something similar with libbpf-rs in one of my large projects, so it would be nice if this crate offered something similar.

Currently, as a workaround, I am just invoking the libbpf-cargo CLI from my build.rs, but obviously this is not ideal.

Non-deterministic output starting in libbpf-cargo 0.8+

I tried the following:

$ cargo libbpf --version
cargo-libbpf-libbpf 0.7.2
$ cargo libbpf make
$ git add src
$ cargo libbpf make
$ git diff --stat # no output

$ cargo install libbpf-cargo
$ cargo libbpf --version
cargo-libbpf-libbpf 0.8.0
$ cargo libbpf make
$ git add src
$ cargo libbpf make
$ git diff
diff --git a/src/my_file.skel.rs b/src/my_file.skel.rs
index 9bd1e5d..43e3718 100644
--- a/src/my_file.skel.rs
+++ b/src/my_file.skel.rs
@@ -487,7 +487,7 @@ const DATA: &[u8] = &[
     0, 0, 0, 0, 0, 0, 208, 1, 0, 0, 0, 0, 0, 0, 64, 1, 0, 0, 4, 0, 155, 0, 0, 0, 8, 1, 1, 251, 14,
     13, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 115, 114, 99, 47, 98, 112, 102, 0, 47, 117, 115, 114,
     47, 105, 110, 99, 108, 117, 100, 101, 47, 97, 115, 109, 45, 103, 101, 110, 101, 114, 105, 99,
-    0, 47, 116, 109, 112, 47, 46, 116, 109, 112, 70, 80, 118, 116, 74, 84, 47, 98, 112, 102, 0, 47,
+    0, 47, 116, 109, 112, 47, 46, 116, 109, 112, 57, 98, 80, 117, 48, 105, 47, 98, 112, 102, 0, 47,
     117, 115, 114, 47, 105, 110, 99, 108, 117, 100, 101, 47, 108, 105, 110, 117, 120, 0, 0, 117,
     100, 112, 95, 114, 111, 117, 116, 101, 46, 98, 112, 102, 46, 99, 0, 1, 0, 0, 105, 110, 116, 45,
     108, 108, 54, 52, 46, 104, 0, 2, 0, 0, 98, 112, 102, 95, 104, 101, 108, 112, 101, 114, 95, 100,

This makes it annoying to commit the generated file to git, since it will always be marked as out of date.

Development Enviroment

I'm sorry if this is not the right place to ask this, but my question revolves around me trying to tweak certain things in the runqslower example within this repo as part of my foray into ebpfs.

I was able to clone this repo, build and successfully run runqslower. However, when I opened this up in an IDE (I use CLion) ,code navigations into the bpf helper functions in C code didn't work as it didn't know where to find it due to the fact that it cannot find the bpf_helpers header file and suggests me to make a CMake file.

I believe libbpf-rs itself gets them from libbpf_sys which packs libbpf. Any ideas on how I can point IDE resolution towards this libbpf?

Cannot reuse fds for BPF map without pinning it globally.

Currently, reuse_pinned_map does not allow for multiple BPF programs created by the same userspace program to share one BPF map.

/// Reuse an already-pinned map for `self`.
    pub fn reuse_pinned_map<P: AsRef<Path>>(&mut self, path: P) -> Result<()> {
        let cstring = util::path_to_cstring(path)?;

        let fd = unsafe { libbpf_sys::bpf_obj_get(cstring.as_ptr()) };
        if fd < 0 {
            return Err(Error::System(errno::errno()));
        }

        let ret = unsafe { libbpf_sys::bpf_map__reuse_fd(self.ptr, fd) };

        // Always close `fd` regardless of if `bpf_map__reuse_fd` succeeded or failed
        //
        // Ignore errors b/c can't really recover from failure
        let _ = unistd::close(fd);

        if ret != 0 {
            return Err(Error::System(-ret));
        }

        Ok(())
    }
}

setting XDP modes

Hello,

I see that bpf_program__attach_xdp is used to attach programs to XDP but how do we specify the mode (generic, native, offloaded)? Seems like generic is the default, is that right?

I found this set of patches and it looks like the bpf_link-based API was added to attach BPF XDP programs through
some low level (BPF_LINK_CREATE) command. Does this mean that libbpf does not provide a way to specify the mode when using the bpf link based attachment?
I'm aware that there are other means to attach (using netlink I think) and specify the mode but they wont leverage the bpf_link infrastructure.

I'm new to BPF so any guidance would be great. I loving this project btw.

cargo libbpf build command failed

symptoms

Fail to build if clang --version returns a string which unexpected format.

expected

clang version 10.0.0
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin

the string when the command fail.

Ubuntu clang version 11.0.1-++20201121072624+973b95e0a84-1~exp1~20201121063303.19
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin

And cargo libbpf build command returns following:

$ cargo libbpf build
/bin/clang is invalid: Error parsing major identifier

clang

installed from https://apt.llvm.org/groovy/pool/main/l/llvm-toolchain-11/

kernel version

Linux shun159.localhost 5.9.8 #1 SMP Wed Nov 18 14:56:48 JST 2020 x86_64 x86_64 x86_64 GNU/Linux

lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
DISTRIB_DESCRIPTION="Ubuntu 20.10"

if you need further information, please let me know.

Regards

support for manipulating btf

I'm currently trying to symbolize my kernel stack trace. It seems that /system/kernel/btf/vmlinux needs to be parsed and that libbpf and libbpf-cargo have support for it, but not libbpf-rs.

libbpf-rs: Add contributing guidelines

Let's add contributing guidelines (https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/setting-guidelines-for-repository-contributors#about-contributing-guidelines).

We should mention few common things:

  1. squashing commits that were added to address feedback (commit should be a self-contained change, not spread out across multiple commits and two separate changes should be in two separate commits).
  2. Signed-off-by: First Last [email protected] (to follow libbpf and kernel guidelines, it's nice to be able to attribute commits to real people).
  3. Full and descriptive commits. No need to write poems, but simplistic "add abc" is a bit too terse. Again, take a page from kernel guidelines: 1) succinct single-line subject 2) detailed enough description of (high-level) what and why.

Not able to build the project

I have installed this library using git and wanted to run some scripts

 error[E0063]: missing fields `__bindgen_padding_0` and `__bindgen_padding_1` in initializer of `bpf_object_open_opts`
  --> /home/Desktop//libbpf-rs/libbpf-rs/src/object.rs:62:9
   |
62 |         libbpf_sys::bpf_object_open_opts {
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `__bindgen_padding_0` and `__bindgen_padding_1`

For more information about this error, try `rustc --explain E0063`.
error: could not compile `libbpf-rs` due to previous error

i am getting this following error when i try to run the command cargo build

my Cargo.toml looks like this

package]
name = "my_project"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
anyhow = "1.0"
chrono = "0.4"
libbpf-rs = { path = "../libbpf-rs/libbpf-rs" }
libc = "0.2"
plain = "0.2"
structopt = "0.3"

[build-dependencies]
libbpf-cargo = { path = "../libbpf-rs/libbpf-cargo" }


Reduce boiler plate in libbpf-cargo tests

Right now there's a bunch of integration tests in libbpf-cargo that have a huge amount of boiler plate. I think that could easily be reduced. Just scroll through the libbpf-cargo/src/test.rs and you'll see it.

Cannot build examples

Try cargo libbpf make --manifest-path examples/runqslower/Cargo.toml

Compiling BPF objects
Failed to compile progs: Failed to compile obj=runqslower.bpf.o with status=exit code: 1
 stdout=
 
 stderr=
 /var/home/vladislav/workspace/vlad9486/libbpf-rs/examples/runqslower/src/bpf/runqslower.bpf.c:4:10: fatal error: 'bpf/bpf_helpers.h' file not found
#include <bpf/bpf_helpers.h>
         ^~~~~~~~~~~~~~~~~~~
1 error generated.

I suppose here need to add -I option, but my system has no such header. I see out dir of libbpf-sys containing required header. But I cannot make clang see it.

Support for ringbuf?

I noticed this crate seems to support the older perf buffer technique for passing per-event data to userspace but not the newer ringbuf map. Do you have plans to include this / would you be interested in a PR?

Thanks! Loving the crate so far.

how to push the data from kernel space to user space using this library

I have a char array of size 128 in the kernel space code that is my_example.bpf.c

how to view this data to user space that is to main.rs

can someone help me with some code snippets

This is my example.bpf.c

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

struct cached_file{
char path[128];
};
struct {
  __uint(type, BPF_MAP_TYPE_INODE_STORAGE);
  __uint(map_flags, BPF_F_NO_PREALLOC);
  __type(key, int);
  __type(value, struct cached_file);
} files SEC(".maps");

struct cached_file *get_cached_file(struct inode *inode) {
  return bpf_inode_storage_get(&files, inode, 0, 0);
}


SEC("lsm/inode_unlink") 
int BPF_PROG(inode_unlink, struct inode *dir,struct dentry *victim)
{
  struct cached_file *cached = get_cached_file(victim->d_inode);
if (cached) 
 {
	bpf_printk("From inode_unlink %s",cached);
}

This is my main.rs file

use anyhow::{bail,Result};

mod bpf;
use bpf::*;
fn bump_memlock_rlimit() -> Result<()> {
    let rlimit = libc::rlimit {
        rlim_cur: 128 << 20,
        rlim_max: 128 << 20,
    };

    if unsafe { libc::setrlimit(libc::RLIMIT_MEMLOCK, &rlimit) } != 0 {
        bail!("Failed to increase rlimit");
    }

    Ok(())
}

fn main() -> Result<()> {
    println!("entering the main function");
    let mut example_builder = ExampleSkelBuilder::default();
    println!("entering the skilbuilder open section");
    bump_memlock_rlimit()?;

    let mut example = example_builder.open()?;
    println!("entering the loading stage");
    let mut trace = example.load()?;
    trace.attach()?;
    println!("Hello, world!");
    let delay =std::time::Duration::from_secs(30);
    std::thread::sleep(delay);
    Ok(())
}

how to push this variable cached to user space programme to main.rs code

Hard error if a `package.metadata.*` section exists that isn't `.libbpf`

$ cargo libbpf make
Compiling BPF objects
Error: Failed to compile BPF objects

Caused by:
    Failed to process package=bpf-graceful-restart, error=unknown variant `deb`, expected `libbpf`

The short-term workaround is to comment out the [package.metadata.deb] section, but it would be nice for it to just work.

Getting code to run cross OS(host) through containers

I have got it working , but I think only partially. @danobi I would like your thoughts on this.

My host is Ubuntu, so is container. But apparently the container comes with BTF info, even though it actually doesn't contain a kernel? I am mounting, right now, just sys/kernel/debug for the capable example. And its working, but its probably using the btf from within the container and I'm getting lucky. Should I be trying to mount the sys/kernel/btf directory as well?

All Examples should use the same vmlinux.h file

No need for every example to have it's own vmlinux.h file.

Put one vmlinux_505.h inside of a ./examples/header/ directory and have all examples' vmlinux.h files soft link to that. (or something similar)

rustfmt binary dependency not explicit

Since the inclusion of the generation of the skeleton, I was not able to use cargo libbpf make, even in the example folder of a plain git clone of the project.

My problem was that rustfmt was not part of the default install on Fedora, and so the generation of the skeleton failed. The problem is the error message is not very explicit.

For a reproducer and the output, see the following:

$> podman run --rm -ti fedora:33
[root@434ed7ebb708 /]# cd
[root@434ed7ebb708 ~]# dnf install -y cargo git clang llvm zlib-devel libbpf-devel
[root@434ed7ebb708 ~]# git clone https://github.com/libbpf/libbpf-rs.git
[root@434ed7ebb708 ~]# cd libbpf-rs/examples/runqslower/
[root@434ed7ebb708 runqslower]# cargo install libbpf-cargo
[root@434ed7ebb708 runqslower]# export PATH=/root/.cargo/bin:$PATH
[root@434ed7ebb708 runqslower]# cargo libbpf make
Compiling BPF objects
Generating skeletons
libbpf: elf: skipping unrecognized data section(25) .eh_frame
libbpf: elf: skipping relo section(26) .rel.eh_frame for section(25) .eh_frame
libbpf: elf: skipping unrecognized data section(25) .eh_frame
libbpf: elf: skipping relo section(26) .rel.eh_frame for section(25) .eh_frame
Warning: unrecognized map: .maps
Warning: unrecognized map: license
Failed to generate skeleton for /root/libbpf-rs/examples/runqslower/src/bpf/runqslower.bpf.c: No such file or directory (os error 2)
Failed to generate skeletons

So today, I took rust-gdb and traced down why it wouldn't open /root/libbpf-rs/examples/runqslower/src/bpf/runqslower.bpf.c while it is obviously there. And it turns out that the os error 2 is not related to this file, but because of https://github.com/libbpf/libbpf-rs/blob/master/libbpf-cargo/src/gen.rs#L767:

let skel = rustfmt(&gen_skel_contents(debug, name, obj)?, rustfmt_path)?;

I am very much a novice in rust, so it was not immediate to me that this was trying to access rustfmt, and external binary.

It would be good if we could either have a detection of this problem earlier, or even a dependency on rustfmt when installing libbpf-cargo.

Thanks!

declaring a map requires having a field named type

This can not be accomplished when compiling from rust. type_ as emitted by bindgen or r#type generate wrong identifier names in the debug info making it impossible to get btf to acknowledge a type field. I'd suggest also accepting type_.

Ups wrong repo should be the libbpf one.

Skeleton generation issues with short filename + global variables

This is something I noticed while I was working on my personal project based on libbpf-rs. When I have a very short BPF source file name and global variables, the resulting skeleton generates incorrectly, such that there is a name mismatch between the generated map name and the generated skeleton name.

The above may be a bit confusing, so I'll illustrate with an example:

src/bpf/ebph.bpf.c:

/* ... */
volatile const int ebph_logging;
/* ... */

src/bpf/ebph_skel.rs:

fn build_skel_config() -> libbpf_rs::Result<libbpf_rs::skeleton::ObjectSkeletonConfig<'static>> {
    let mut builder = libbpf_rs::skeleton::ObjectSkeletonConfigBuilder::new(DATA);
    builder
        .name("ebph_bpf")
        .map("__profile_key_t__alloc", false)
        .map("__profile_key_t__temp", false)
        .map("profiles", false)
        .map("ebph.rodata", true)
        .prog("bprm_check_security");

    builder.build()
}

Note that name is ebph_bpf but we have ebph.rodata. In the libbpf debug logs, we can see the problem:

libbpf: map 'ebph_bpf.rodata' (global data): at sec_idx 6, offset 0, flags 480.
libbpf: map 3 is "ebph_bpf.rodata"
(...)
libbpf: failed to find skeleton map 'ebph.rodata'

If I go into the generated skeleton and manually update to either .name("ebph") or .map("ebph_bpf.rodata"), we get the expected result. With longer program names, the _bpf part gets cut off and we get the expected result as well. You can see this in runqslower for example:

fn build_skel_config() -> libbpf_rs::Result<libbpf_rs::skeleton::ObjectSkeletonConfig<'static>> {
    let mut builder = libbpf_rs::skeleton::ObjectSkeletonConfigBuilder::new(DATA);
    builder
        .name("runqslower_bpf")
        .map("start", false)
        .map("events", false)
        .map("runqslow.rodata", true)
        .prog("handle__sched_wakeup")
        .prog("handle__sched_wakeup_new")
        .prog("handle__sched_switch");

    builder.build()
}

I haven't had a chance to look at the skeleton generation code yet, but I suspect there is a trivial fix.

[libbpf-cargo] Add ability for user defined Default

Arrays cannot be #[derive(Default)] if they are > 32 in length.

As such, the line https://github.com/libbpf/libbpf-rs/blob/master/libbpf-cargo/src/btf/btf.rs#L348 will fail compilation for larger arrays. Because the struct is defined as default per this line of code, the user cannot impl Default themselves.

Arrays of longer than 32 can have user defined Default so long as the generated skeleton does not #[derive(Default)].

Request: Add a flag so that *_bss_types do not automatically add the Default Derive.

Request++: automatically detect if an array in a structure is > 32 length, and create an impl Default in the skeleton

Workaround: cargo libbpf build && cargo libbpf gen && EDIT THE *.skel.rs FILE MANUALLY && add user defined impl Default for types

[libbpf-cargo] Rust Compiler complains about snake-case enums

The Rust compiler complains about snake cased enums

eg

// THIS IS C CODE
enum file_event_t {
    CREATE,
    CLOSE,
    DELETE,
    LINK,
};

// THIS IS RUST TRANSLATION
#[derive(Debug, Copy, Clone, PartialEq)]
#[repr(u32)]
pub enum file_event_t {
    CREATE = 0,
    CLOSE = 1,
    DELETE = 2,
    LINK = 3,
}
impl Default for file_event_t {
    fn default() -> Self {
        file_event_t::CREATE
    }
}

THIS IS WARNING DURING COMPILE
|     pub enum file_event_t {
    |              ^^^^^^^^^^^^ help: convert the identifier to upper camel case: `FileEventT`
    |
    = note: `#[warn(non_camel_case_types)]` on by default

[Libbpf-rs] Support context in sample_cb and lost_cb

Right now, sample_cb and lost_cb receive two parameters. (cpu: i32, data: &[u8]) and (cpu: i32, count: u64) respectively.

It would be useful to allow a user defined type as context such as a *mut, or a such that
sample_cb(ctx: *mut, cpu: i32, data: &[u8])
or
sample_cb(ctx: T, cpu: i32, data: &[u8])

bool in bpf code becomes u8 in generated skeleton code

Hi,

The current skel generation does this. I was looking up how to convert the bool properly to u8 when I found this . Any thoughts on why the skel generation represents bool in C as u8 in Rust? Also, how would you convert rust bool to rust u8 so that the bool value does get passed to bpf code?

Thanks in advance

Don't parse ELF in libbf-cargo gen

Once libbbpf_sys is updated to libbpf v0.2, we can use the btf__set_endianness() API to force native endianness and not need to worry about endianness.

issue with bpf_d_path when used along with inode_getattr

This is the Programme that i am trying to execute

SEC("lsm/inode_getattr")
int BPF_PROG(inode_getattr,const struct path *path)
{char *buf; 
bpf_d_path(path,buf,100);
bpf_printk("%s",buf);
bpf_printk("enterred the parameters part");
}

but getting this error

entering the main function
entering the skilbuilder open section
libbpf: elf: skipping unrecognized data section(7) .rodata.str1.1
entering the loading stage
libbpf: load bpf program failed: Permission denied
libbpf: -- BEGIN DUMP LOG ---
libbpf: 
R1 type=ctx expected=fp
; int BPF_PROG(inode_getattr,const struct path *path)
0: (79) r1 = *(u64 *)(r1 +0)
func 'bpf_lsm_inode_getattr' arg0 has btf_id 726 type STRUCT 'path'
; bpf_d_path(path,buf,100);
1: (b7) r3 = 100
2: (85) call bpf_d_path#147
R2 !read_ok
processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

libbpf: -- END LOG --
libbpf: failed to load program 'inode_getattr'
libbpf: failed to load object 'example_bpf'
libbpf: failed to load BPF skeleton 'example_bpf': -4007
Error: System error, errno: 4007



Is this error more related to permission to use bpf_d_path along with inode_getattr or something else please let me know how to fix this

Add the ability to iterate over links

As of #90 we now have the ability to iterate over maps and programs using the underlying Object and OpenObject. It would be nice to be able to do the same thing for Links. That way the user could, for example, easily disconnect every single link once #104 lands.

Unfortunately, the path forward is not so clear, since Links are not exposed by Object but rather by the generated skeleton itself. The obvious way is to refactor the way the generated skeleton manages links such that it stores them all in a HashMap for example, but then it feels kind of weird to expose Program and Map iterators through Object but the Link iterator through the skeleton. Any thoughts would be appreciated :)

Interacting with existing maps?

Long story short: I want to use libbf-rs to interact with globally pinned maps that are not created by loading an object with libbpf-rs, but I can not find a way to do that.

Some context: I am working on a user space program which communicates with a bpf object loaded as a tc direct-action via bpf maps. My bpf object in kernel creates the maps, and pins them globally. So, the task for the user-space part is just to manipulate these existing maps. I came up with this design since I have not find any option to load my bpf object as a direct-action w/o using tc.

Libbpf has a straightforward way to interact with existing maps: bpf_obj_get() returns the fd of the map, then the returned fd can be used for map operations. Is there any way to do this with libbpf-rs?

Thanks!

Skeleton generation silently fails when using a typedefed type as a libbpf global

The following minimal example will cause skeleton generation to silently fail, producing an empty mod.rs file:

#include "vmlinux.h"

#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

typedef enum {
    TEST1 = 0x1,
    TEST2 = 0x2,
    TEST3 = 0x4,
} test_t;

const volatile test_t test;

SEC("tp_btf/execve")
int BPF_PROG(trace_execve, char *filename, pid_t pid, pid_t old_pid)
{
    return 0;
}

I'm using the latest libbpf-rs and libbpf-cargo from the master tree.

The following diff currently works as a workaround, but it would be nice if we could support using typedefed types directly. Is this technically feasible? At the very least this should be a hard error during skeleton generation if we cannot support it.

*** enum.bpf.c  2021-05-18 10:06:13.760039580 -0400
--- enum.bpf.working.c  2021-05-18 10:06:05.433372871 -0400
***************
*** 4,16 ****
  #include <bpf/bpf_helpers.h>
  #include <bpf/bpf_tracing.h>

! typedef enum {
      TEST1 = 0x1,
      TEST2 = 0x2,
      TEST3 = 0x4,
  } test_t;

! const volatile test_t test;

  SEC("tp_btf/execve")
  int BPF_PROG(trace_execve, char *filename, pid_t pid, pid_t old_pid)
--- 4,16 ----
  #include <bpf/bpf_helpers.h>
  #include <bpf/bpf_tracing.h>

! typedef enum test {
      TEST1 = 0x1,
      TEST2 = 0x2,
      TEST3 = 0x4,
  } test_t;

! const volatile enum test test;

  SEC("tp_btf/execve")
  int BPF_PROG(trace_execve, char *filename, pid_t pid, pid_t old_pid)

Parsing pinned maps

I am trying to parse a pinned map using your crate. I have found the function reuse_pinned_map from the OpenMap struct, but I am unable to initiate an OpenMap structure. I don't know how I can create a variable with the type*mut libbpf_sys::bpf_map. I am pretty new to rust. Any help is welcome.

XDP Support Planned?

Is there any plan to support the XDP portions of libbpf? If so, are there thoughts on how you would like that accomplished so people can submit PRs or contributions to that end?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.