opensourcerouting / c-capnproto Goto Github PK
View Code? Open in Web Editor NEWThis project forked from jmckaskill/c-capnproto
C library/compiler for the Cap'n Proto serialization/RPC protocol
License: MIT License
This project forked from jmckaskill/c-capnproto
C library/compiler for the Cap'n Proto serialization/RPC protocol
License: MIT License
capn_write_mem()
and capn_init_mem()
take a packed
parameter, but this parameter is not tested in the tests and when I tried to use it, it does not work correctly (actually writing works fine, reading does not).
The problem is in init_fp()
the code needs to first read in the packed data, unpack it, then use that unpacked data as input. Enabling the packed option in capn-stream-test.cpp will demonstrate my point.
I have a Cap'n Proto schema that looks like:
struct UI {
union {
thing1 @0 :Text;
thing2 @1 :AnEnumType;
thing3 @2 :UInt16;
}
}
The enum type is represented as a C enum. But the generated code treats it as interchangeable with the UInt16
type:
void read_UI(struct UI *s capnp_unused, UI_ptr p) {
capn_resolve(&p.p);
capnp_use(s);
s->which = (enum UI_which)(int) capn_read16(p.p, 0);
switch (s->which) {
// ...
case UI_thing2:
case UI_thing3:
s->thing3 = capn_read16(p.p, 2);
The size of a C enum is not specified by the standard; compilers can choose whatever is appropriate. On my platform this is 4 octets (32 bits), so the LSBs of the enum are wiped. This changes with the positioning of the enum in the Cap'n Proto definition; in an even numbered position, it's fine.
Meson is a very popular build system for C-only projects nowadays. It's used by many big projects like Mesa, ffmpeg, etc.
There are two implementations of it:
Not sure if this is a bug in the library, or user error (but I fail to see how, do tell if you find an issue) I have created
enum MessageCategory
{
none @0;
system @1;
log @2;
debug @3;
file @4;
cmd @5;
max @6;
}
struct Message
{
category @0 :MessageCategory;
type @1 :UInt32;
containedMessage @2 :Data;
}
Built fine with c-capnproto compiler, but upon serialization/de-serialization something breaks.
int main()
{
uint8_t data[] = { 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41 };
uint8_t buf[0x1000];
int32_t bufDataLen = 0;
{
struct capn c;
capn_init_malloc(&c);
capn_ptr cr = capn_root(&c);
struct capn_segment* cs = cr.seg;
const char helloWorld[] = "Hello fucking world";
capn_list8 list8 = capn_new_list8(cs, sizeof(helloWorld));
capn_setv8(list8, 0, helloWorld, sizeof(helloWorld));
struct Message m;
m.category = MessageCategory_file;
m.type = 2;
m.containedMessage.p = list8.p;
memset(buf, 0xFE, sizeof(buf));
bufDataLen = capn_write_mem(&c, buf, sizeof(buf), false);
capn_free(&c);
}
FILE* file = fopen("dump.bin", "wb");
fwrite(&bufDataLen, sizeof(bufDataLen), 1, file);
fwrite(buf, sizeof(uint8_t), bufDataLen, file);
fclose(file);
file = NULL;
{
struct capn rc;
int init_mem_ret = capn_init_mem(&rc, buf, sizeof(buf), false);
if (init_mem_ret != 0)
{
return -1;
}
Message_ptr root;
struct Message msg;
root.p = capn_getp(capn_root(&rc), 0, true);
read_Message(&msg, root);
capn_free(&rc);
printf("got msg (%d) (%d)\n", msg.category, msg.type);
}
return 0;
}
dump.bin contains some tags, and my string "Hello fucking world" just fine. The de-serialization code runs also with no issue. The Message msg; is full of uninitialized values, and after the read_Message call, it's all zeroed out (instead of intended values). Not what they intended to be from the buffer.
PRs aren't being reacted upon, last commit two years ago. Would be nice to make it work on modern systems and make a new release.
$ cat infra.capnp
@0xab01f6ae1ef59a57;
struct S1 {
s1 @0 :Int64;
}
struct S2 {
s2 @0 :Int64;
}
$ cat msg.capnp
@0xed2b3b98b3eb515b;
using import "/infra.capnp".S1;
struct M {
m1 @0: S1;
}
$ capnp -I ~/ws/capn compile -o /home/assafb/ws/capn/capnpc-c:/home/assafb/ws/capn/ ~/ws/capn/msg.capnp
cant find node with id 0x85f0eda94e40ad1a
/home/assafb/iguazio/workspace/engine/zeek/build/vendor/c-capnproto/capnpc-c: plugin failed: exit code 2
I somehow can't leave it uninitialized, or I get,
*** ERROR DECODING PREVIOUS MESSAGE ***
The following error occurred while decoding the message above.
This probably means the input data is invalid/corrupted.
Exception description: expected boundsCheck(segment, ptr, ref->structRef.wordSize()); Message contained out-of-bounds struct pointer.
Code location: capnp/layout.c++:2159
*** END ERROR ***
From the capnp FAQ:
Pointer fields are a bit different. They start out “null”, and you can check for nullness using the hasFoo() accessor. You could use a null pointer to mean “not present”. Note, though, that calling getFoo() on a null pointer returns the default value, which is indistinguishable from a legitimate value, so checking hasFoo() is in fact the only way to detect nullness.
What do I need to do to not set a pointer?
Specifically, I sometimes do not set the worker
field here: https://gitlab.com/dxpb/dxpb/blob/1b8775bd089ca3c2df1c53030ddb5adec7fa4313/src/log.capnp#L45-48
[ RUN ] Examples.RoundTripPerson
../tests/example-test.cpp:89: Failure
Expected: (f) != ((void*)0), actual: NULL vs NULL
[ FAILED ] Examples.RoundTripPerson (0 ms)
[ RUN ] Examples.PersonWithAccessors
[ OK ] Examples.PersonWithAccessors (0 ms)
[----------] 2 tests from Examples (0 ms total)
[----------] 1 test from Schema
[ RUN ] Schema.ReadSimple
[ OK ] Schema.ReadSimple (0 ms)
[----------] 1 test from Schema (0 ms total)
[----------] Global test environment tear-down
[==========] 24 tests from 4 test suites ran. (0 ms total)
[ PASSED ] 23 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] Examples.RoundTripPerson
1 FAILED TEST
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
Summary of Failures:
1/1 capn-test FAIL 0.40s exit status 1
Ok: 0
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
It's Clang on macOS ARM64 (M1):
ℤ clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
ℤ sw_vers
ProductName: macOS
ProductVersion: 13.3.1
BuildVersion: 22E261
Hi ,
I was trying to use the plugin and I encountered what I think is a bug, please let me know if my understanding is wrong about how the generated API works.
The schema definitions that I used is the following:
struct Leaf{
value @0 : Int32;
}
struct Inner {
left @0 :TreeNode;
right @1 : TreeNode;
}
struct TreeNode {
nodeType @0 :Int32;
leaf @1 : Leaf;
inner @2 :Inner;
}
For the following code I expect 1 to be printed but 0 is printed instead ;It looks like there is something wrong in the method Tree_Node_set_leaf(...) unless I misunderstand how it should work!
struct capn c;
struct capn_segment *cs;
capn_ptr cr;
capn_init_malloc(&c);
cr = capn_root(&c);
cs = cr.seg;
TreeNode_ptr n = new_TreeNode(cs);
TreeNode_set_nodeType(n, 1);
Leaf_ptr l = new_Leaf(cs);
Leaf_set_value(l , 1);
TreeNode_set_leaf(n, l);
cout<<Leaf_get_value(TreeNode_get_leaf(n))<<endl;
In the function capn_write_mem_packed in capn-malloc.c the uint32_t pointer "header" is calculated.
The function gets a 32 bit aligend pointer to the payload buffer passed.
"header" has an offset to the payload buffer pointer of header size + 2 bytes. Depending on header size it can happen that "header" is not aligned to 32 bit.
Subsequently header_render() performs 32 bit write access to the unaligned "header" pointer.
Processor which can't perform unaligned 32 bit access went to hard fault/memory fault.
As a walk around it is possible to write the data to be written in a variable and copy the variable to the pointer with memcpy.
I have a struct like this:
struct Event {
args @0 :List(Text);
}
Which generates a struct like this:
struct Event {
capn_ptr args;
};
Which seems wrong because if I try to get the length of the list using capn_len I cannot compile my program because capn_len
tries to access a member named p
and struct capn_ptr
doesn't have a member named p
:
In file included from bufs/main.capnp.h:4:0,
from event.c:3:
event.c: In function 'event_tocapn':
../deps/build/usr/local/include/capnp_c.h:182:31: error: 'capn_ptr {aka struct capn_ptr}' has no member named 'p'
#define capn_len(list) ((list).p.type == CAPN_FAR_POINTER ? (capn_resolve(&(list).p), (list).p.len) : (list).p.len)
^
event.c:173:27: note: in expansion of macro 'capn_len'
printf("Len is %d\n", capn_len(ie->args));
^~~~~~~~
...
If I use a custom defined type as the list members (such as a struct containing only a Text field) everything is fine.
From looking at other examples, it looks like this field should be a capn_data
struct or one of the capn_list
structs?
I see this included even in the file tests/addressbook.capnp.h
https://github.com/opensourcerouting/c-capnproto/blob/master/tests/addressbook.capnp.h#L10
#include "c.capnp.h"
This header file is not installed by make install
Why is this file included in the resulting schema header? If it is necessary shouldn't c.capnp.h
be installed along with other headers?
According to our experiments, if the size of the raw message is less than 200 bytes, cap'proto gets slower than protobuf. The reason could be the overhead taken by the malloc process invovled in the deserializing each message?
It would be great if you could show in the docs how the Example Usage in https://capnproto.org/cxx.html translates into C using this plugin.
I could be mis-reading this, but it looks like https://github.com/opensourcerouting/c-capnproto/blob/master/lib/capn-malloc.c#L38 should instead be:
s->user = u;
From what I see, in each example or project using c-capnproto, each time capn_write_mem
is used, the target buffer is mostly everytime allocated on stack with small length (typically 4k or 8k).
If my object is (let assume) random amount of megabytes, how can I allocate a buffer large enough ?
I don't see any function able to tell me the expected size of the output.
Is there any way possible to expect the final size to allocate it ?
If capn_setp
not been called. data serialized shell not contain it. or an assert shell emit at least.
It's really hard to debug it.
eg.
file a.capnp
@0xb5c2cbd0a1ee9f74;
struct Person{
name @0 :Text;
}
#include <unistd.h>
#include<sys/types.h>
#include<fcntl.h>
#include <capnp_c.h>
#include "a.capnp.h"
void x(){
};
int main() {
struct capn capn;
struct capn_segment *seg;
capn_init_malloc(&capn);
capn_ptr root = capn_root(&capn);
Person_ptr personobj=new_Person(root.seg);
printf("root->type %d\n",root.type);
struct capn_text ustr={
3,"aaa", NULL
};
Person_set_name(personobj, ustr);
int fd=open("aaa.data",O_CREAT|O_RDWR);
capn_write_fd(&capn, write , fd, 0);
close(fd);
}
this will generate file aaa.data
, content:
[root@7e6235426e5d bbb]# od -x aaa.data
0000000 0000 0000 0003 0000 0000 0000 0000 0000
0000020 0001 0000 0022 0000 6161 0061 0000 0000
0000040
[root@7e6235426e5d bbb]# capnp decode a.capnp Person < aaa.data
()
[root@7e6235426e5d test-go-capnp]# od -x good.data
0000000 0000 0000 0003 0000 0000 0000 0000 0001
0000020 0001 0000 0022 0000 6161 0061 0000 0000
0000040
[root@7e6235426e5d test-go-capnp]# capnp decode a.capnp Person < good.data
(name = "aaa")
configure.ac:18: error: required file 'ac/ltmain.sh' not found
configure.ac:14: error: required file 'config.h.in' not found
capn_write_mem()'s return value is int.
../lib/capn.c:484:17: runtime error: left shift of negative value -1
[ OK ] WireFormat.StructRoundTrip_OneSegment (6 ms)
../lib/capn.c:704:19: runtime error: member access within null pointer of type 'struct copy'
../lib/capn-malloc.c:219:12: runtime error: store to misaligned address 0x7ffcfc967a0a for type 'uint32_t', which requires 4 byte alignment
0x7ffcfc967a0a: note: pointer points here
00 00 f8 ff ff ff ff ff ff ff 90 7a 96 fc fc 7f 00 00 1e 00 00 00 00 00 00 00 f8 00 00 00 00 00
^
../lib/capn-malloc.c:220:22: runtime error: store to misaligned address 0x7ffcfc967a0e for type 'uint32_t', which requires 4 byte alignment
0x7ffcfc967a0e: note: pointer points here
00 00 00 00 ff ff 90 7a 96 fc fc 7f 00 00 1e 00 00 00 00 00 00 00 f8 00 00 00 00 00 00 00 00 00
^
../lib/capn-malloc.c:225:17: runtime error: store to misaligned address 0x7ffcfc967a0e for type 'uint32_t', which requires 4 byte alignment
0x7ffcfc967a0e: note: pointer points here
00 00 00 00 00 00 00 00 96 fc fc 7f 00 00 1e 00 00 00 00 00 00 00 f8 00 00 00 00 00 00 00 00 00
^
When I try to build capnp file to c source, I always get an error. Any suggestions would be greatly appreciated.
sudo capnp compile -o c addressbook.capnp
addressbook.capnp:27:18-28: error: Import failed: /c.capnp
addressbook.capnp:28:2-3: error: Not defined: C
The generated .c file for the below capnp does not compile due to undeclared capn_buf:
$ cat Test.capnp.c
#include "Test.capnp.h"
/* AUTO GENERATED - DO NOT EDIT */
#ifdef __GNUC__
# define capnp_unused __attribute__((unused))
# define capnp_use(x) (void) x;
#else
# define capnp_unused
# define capnp_use(x)
#endif
static const capn_ptr capn_null = {CAPN_NULL};
static capn_data capn_val1 = {{2,0,0,0,1,0,0,(char*)&capn_buf[0],(struct capn_segment*)&capn_seg}};
...
File Test.capnp:
@0xa4ba9dce9642ee58;
struct Record {
info @0: Data = "";
}
struct RootStruct {
myList @0: List(Record);
}
Hello!
I want to perform element-wise reading on a capn_list64
. I have a problem with capn_get64()
, which always returns a zero value, whereas capn_getv64()
works fine. What could be the reason?
I test as follows:
uint64_t sample_get = capn_get64(samples, sample_index);
uint64_t sample_getv;
capn_getv64(samples, sample_index, &sample_getv, 1);
In the above example, sample_get = 0
, but sample_getv = 4625979026566705573
, which is the number that I stored in the first place.
Thanks in advance!
I have problem with reading build-in list types. This code simply creates new Message structure, calls capn_setv, writes message to capn segment and reads it. After this operations calling capn_getv on body fails.
d.capnp:
@0xec7cea5be31ae639;
struct Message {
body @0 :List(UInt8);
}
d.c:
#include <capnp_c.h>
#include <string.h>
#include "d.capnp.h"
struct msg_capn_t {
uint8_t segment_bytes[128];
struct capn_segment s;
struct capn c;
struct Message msg;
Message_ptr ptr;
};
int main(void){
struct msg_capn_t ms;
memset(&ms, 0, sizeof(ms));
ms.s.data = (char*)ms.segment_bytes;
ms.s.cap = sizeof(ms.segment_bytes);
capn_append_segment(&ms.c, &ms.s);
capn_root(&ms.c);
// Begin capnp root
ms.ptr = new_Message(&ms.s);
capn_setp(capn_root(&ms.c), 0, ms.ptr.p);
{ // Begin capnp child
uint8_t buffer[2] = {99, 76};
ms.msg.body = capn_new_list8(&ms.s, 2);
capn_setv8(ms.msg.body, 0, buffer, 2);
}
write_Message(&ms.msg, ms.ptr);
read_Message(&ms.msg, ms.ptr);
{ // Begin capnp child
uint8_t buffer[2] = {0, 0};
int c = capn_getv8(ms.msg.body,0, buffer, 2);
if (c == 2){
printf("success\n");
return 0;
} else {
printf("fail capn_getv returned %d\n",c);
return 1;
}
}
}
I have found one solution - manually changing generated file d.capnp.c from:
void read_Message(struct Message *s, Message_ptr p) {
capn_resolve(&p.p);
s->body.p = capn_getp(p.p, 0, 0);
}
to
void read_Message(struct Message *s, Message_ptr p) {
capn_resolve(&p.p);
s->body.p = capn_getp(p.p, 0, 1);
}
This can also be changed in c-capnproto/compiler/capnpc-c.c:604
str_addf(func, "%s = capn_getp(%s, %d, 1);\n", pvar, ptr, f->f.slot.offset);
but I am not sure if it is correct solution, or bug is somewhere else (for example in my code).
Hi,
Thanks for the C implementation of capnproto.
I have hit a wall with regard to using a "Data" type for serializing. De-serialization works well but I am not able to serialize and send data using any method.
Can you please give a simple example to serialize the following structure
struct DataMessage {
packetdata @0 :Data;
}
Thank you very much in advance
I am currently working on a project where I will be using this on Solaris/Sparc 32-bit and I wanted to understand why there is a requirement for 8-byte alignment on the data, len, and cap fields in the struct capn_segment
. As of right now, the call to ALIGNED in struct ALIGNED_(8) capn_segment
fails. Thank you in advance!
For this schema:
struct A {
union {
zero @0 :Void;
one :union {
first @1 :Void;
second @2 :UInt16;
}
}
}
I get this C code:
struct A {
enum A_which which;
capnp_nowarn union {
enum A_one_which one_which;
capnp_nowarn union {
uint16_t second;
} one;
};
};
The one_which
member is in a union alongside one
, which means assigning to one.second
will overwrite one_which
.
Per #47 - the expression headersz + datasz
could conceivably require int64_t
.
[19/44] Compiling C object test.capnp.c.p/compiler_test.capnp.c.o
../compiler/test.capnp.c:3159:42: warning: bitwise comparison always evaluates to true [-Wtautological-bitwise-compare]
s->secondBit = (capn_read8(p.p, 0) & 2) != 1;
~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
../compiler/test.capnp.c:3190:39: warning: bitwise comparison always evaluates to true [-Wtautological-bitwise-compare]
secondBit = (capn_read8(p.p, 0) & 2) != 1;
~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
Document Required. digging code makes me crezy.
I strongly believe that there should be a security warning in the README regarding malicious input.
There is currently no code to verify buffers/structures, unless I missed something completely?
I think a small notice like #29 should suffice.
Any program will crash horrendously should it encounter malicious/corrupted input. If anyone is interested, I adjusted the included tests/examples to be used with american fuzzy lop here. You can test with make fuzz-mem
and make fuzz-fp
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.