commaai / cereal Goto Github PK

We currently have tests around the high level messaging abstractions like PubMaster and SubMaster which use some of the lower level messaging functions, but each function in messaging/__init__.py should be individually tested.

Would probably require introducing mutexes around signaling readers, but would fix a rare lost wakeup in msgq_poll

Thank you so much for sharing this piece of code, very interesting!

Could you please explain the reasons why ZMQ pub/sub (or maybe nng) ins't fast enough? Or how faster is msgq (shm) compared to ZMQ?

Track average receive frequency over the last 10 seconds. With this we can add a second check on frequencies that can be a lot stricter than on instantaneous frequency.

This should also include coverage of the cython extensions.

• python (a91e43f)
• cython
• C++

I hit commaai/openpilot#1358 and did scons -j4 as suggested there,
but I get 7 errors generated from deprecation warnings from OpenCL that look like this:

clang++ -o cereal/messaging/bridge cereal/messaging/bridge.o -Lphonelibs/libyuv/mac/lib -L/usr/local/lib -L/opt/homebrew/lib -L/usr/local/opt/openssl/lib -L/opt/homebrew/opt/openssl/lib -L/System/Library/Frameworks/OpenGL.framework/Libraries -Lcereal -Lphonelibs -Lopendbc/can -Lselfdrive/boardd -Lselfdrive/common cereal/libmessaging.a -lzmq -framework OpenCL
cereal/visionipc/visionbuf_cl.cc:48:18: error: 'clCreateCommandQueue' is deprecated: first deprecated in macOS 10.14 - (Define
      CL_SILENCE_DEPRECATION to hide this warning) [-Werror,-Wdeprecated-declarations]
  this->copy_q = clCreateCommandQueue(ctx, device_id, 0, &err);

I see a similar things was fixed on commaai/openpilot#1394, does it just need porting here?

See #299

Current msgq relies on sending a signal to the receiver thread for the poll function to work correctly. On linux we use syscall(SYS_tkill, tid, SIGUSR2) for this. Does a MacOS equivalent exist?

I used logreader tools to parse rlog.bz2 file but get "Compressed data ended before the end-of-stream marker was reached " error?
How to deal with this.
Thanks.

E.g. use capnproto/pycapnp#260

MSGQ will currently just segfault

Once bridge only republishes services which have been subscribed to, so openpilot can always run it.

eventfd is only available on Linux, so building openpilot and its tools (specifically I'm trying to run cabana) fails on macOS due to the way IPC synchronization was implemented in #439.

Here is the full error I see, after freshly cloning master and running scons a second time:

$ scons -j8 -u -k
scons: Entering directory `/Users/ebrown1/PycharmProjects/openpilot3'
scons: Reading SConscript files ...
Git commit hash for gitversion.h: e1805f65
scons: done reading SConscript files.
scons: Building targets ...
clang++ -o cereal/messaging/event.os -c -std=c++1z -DGL_SILENCE_DEPRECATION -DSWAGLOG="\"common/swaglog.h\"" -g -fPIC -O2 -Wunused -Werror -Wshadow -Wno-unknown-warning-option -Wno-deprecated-register -Wno-register -Wno-inconsistent-missing-override -Wno-c99-designator -Wno-reorder-init-list -Wno-error=unused-but-set-variable -DGL_SILENCE_DEPRECATION -DSWAGLOG="\"common/swaglog.h\"" -fPIC -I/opt/homebrew/include -I/opt/homebrew/opt/[email protected]/include -I. -Ithird_party/acados/include -Ithird_party/acados/include/blasfeo/include -Ithird_party/acados/include/hpipm/include -Ithird_party/catch2/include -Ithird_party/libyuv/include -Ithird_party/json11 -Ithird_party/curl/include -Ithird_party/linux/include -Ithird_party/snpe/include -Ithird_party/mapbox-gl-native-qt/include -Ithird_party/qrcode -Ithird_party -Icereal -Iopendbc/can -Ithird_party/json11 cereal/messaging/event.cc
generate_dbc_json(["tools/cabana/generate_dbc_json"], [])
tools/cabana/dbc/generate_dbc_json.py --out tools/cabana/dbc/car_fingerprint_to_dbc.json
Traceback (most recent call last):
  File "tools/cabana/dbc/generate_dbc_json.py", line 5, in <module>
    from selfdrive.car.car_helpers import get_interface_attr
  File "/Users/ebrown1/PycharmProjects/openpilot3/selfdrive/car/car_helpers.py", line 5, in <module>
    from common.params import Params
  File "/Users/ebrown1/PycharmProjects/openpilot3/common/params.py", line 1, in <module>
    from common.params_pyx import Params, ParamKeyType, UnknownKeyName, put_nonblocking, put_bool_nonblocking # pylint: disable=no-name-in-module, import-error
ModuleNotFoundError: No module named 'common.params_pyx'
scons: *** Error 1
cereal/messaging/event.cc:9:10: fatal error: 'sys/eventfd.h' file not found
#include <sys/eventfd.h>
         ^~~~~~~~~~~~~~~
1 error generated.
scons: *** [cereal/messaging/event.os] Error 1
scons: done building targets (errors occurred during build).

As a temporary workaround, I was able to build cabana by checking out d0a8b3780c in the openpilot repo.

Isn't it interesting to set snd/rcv watermark and linger ZeroMQ options to avoid overflooding the in memory pub/sub queues?

Here is an example for setting it once for all instance sockets, but it can also be done per socket instance.

zsys_set_linger(0);
size_t hwm = 30;
zsys_set_sndhwm(hwm);
zsys_set_rcvhwm(hwm);

On msgq implementation is there any pub/sub queue size limit?

When going through the code, the visionipc_client is binded to "127.0.0.1"
sock = SubSocket::create(msg_ctx, get_endpoint_name(name, type), "127.0.0.1", conflate, false);

But when I hard-coded it with some ip (192.168.1.10), it still doesn't work.

I tried it with the following commands:

# in host machine: ip: 192.168.1.10
export ZMQ=1
replay --demo

# in client machine: ip: 192.168.1.123
export ZMQ=1
python ui.py 192.168.1.10

msgq uses a fixed default size for all queues. Since our struct are all different sizes, this leaves some services with only 10s of buffer and others with more than an hour; if left running for long enough, the queues will use up ~400MB. At build time, we know the struct size (except lists) and frequency for all services, so we can set each service's queue size based on buffer time, which should save quite a bit of memory.

one writer send a img A, can multiple clients recv the img A at the same time?

some read/write operation should be protected with semaphore for mutual exclusion. otherwise it will cause unexpected behaviors, such as lost messages

• setup pyproject.toml
• publish to PyPI
• test on all supported Python versions (example: commaai/bodyjim#10)

When I try to run the stress.py,
I get an error saying
terminate called after throwing an instance of 'YAML::BadFile'
what(): bad file
Aborted (core dumped)

When I used breakpoints I could see that the error is caused in this line pub_socket.connect(c,"controlsState")

Any help ??

I am not able to run the debug_control.py. when I run , I get this error

(openpilot) cyngn@vivekgr:~/roadtrain/openpilot/tools/carcontrols$ python debug_controls.py
make: Nothing to be done for 'all'.
make: Nothing to be done for 'all'.
[ CC ] can_list_to_can_capnp.o
clang++ -std=c++11 -g -fPIC -I../ -I../../ -O2 -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Werror=int-conversion -Werror=return-type -Werror=format-extra-args -I/usr/include/libusb-1.0 -MMD
-Iinclude -I.. -I../..
-I../../phonelibs/capnp-cpp/include
-I../../phonelibs/zmq/aarch64/include
-I../../selfdrive/messaging
-c -o 'can_list_to_can_capnp.o' 'can_list_to_can_capnp.cc'
can_list_to_can_capnp.cc:6:10: fatal error: 'cereal/gen/cpp/log.capnp.h' file not found
#include "cereal/gen/cpp/log.capnp.h"

Would be great if someone can help me.