pybind / pybind11_protobuf Goto Github PK

Hi!

Thanks for the great work! In my project I need to pass a protobuf message between python and a C++ function installed as a module using pybind. I am just curious as to, for such a scenario why would someone resort to using this instead of just using string arguments?

I am guessing speed could be one factor but how big do the protobuf messages need to be to really make a difference in speed?

Thanks,
Indraneel

I am very interested in getting this to run in a CMake project and possibly contributing to make that happen.

I have this repo cloned as a submodule inside a small test project, but cannot get this to compile / run without linker errors.

This is my current setup:

Files

extern/pybind11_protobuf
CMakeLists.txt
fastproto.cpp
Person.proto

Person.proto:

syntax = "proto2";

message Person {
  required int32 id = 2;
  required string name = 1;
  optional string email = 3;
}

fastproto.cpp

#include <pybind11/pybind11.h>
#include "Person.pb.h"
#include <string>
#include "pybind11_protobuf/native_proto_caster.h"

Person get_person() {
    Person person;
    person.set_id(1);
    person.set_name(std::string{"Maximilian Nöthe"});
    person.set_email(std::string{"[email protected]"});
    return person;
}

PYBIND11_MODULE(fastproto, m) {
    pybind11_protobuf::ImportNativeProtoCasters();
    m.def("get_person", &get_person);
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.17..3.22)

project(Quirc++ VERSION 0.1.0 LANGUAGES C CXX)

include(GNUInstallDirs)

find_package(Python3 REQUIRED COMPONENTS Development Interpreter)
find_package(pybind11 REQUIRED)
find_package(Protobuf REQUIRED)

protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS Person.proto)

pybind11_add_module(
    fastproto
    fastproto.cpp
    ${PROTO_SRCS}
    extern/pybind11_protobuf/pybind11_protobuf/native_proto_caster.cc
    extern/pybind11_protobuf/pybind11_protobuf/proto_cast_util.cc
    extern/pybind11_protobuf/pybind11_protobuf/proto_utils.cc
)
target_include_directories(fastproto PRIVATE ${CMAKE_CURRENT_BINARY_DIR} extern/pybind11_protobuf)
target_link_libraries(fastproto PRIVATE protobuf::libprotobuf)
target_compile_features(fastproto PRIVATE cxx_std_14)
set_target_properties(fastproto PROPERTIES
    CXX_EXTENSIONS OFF
    CXX_STANDARD_REQUIRED ON
)

Or use the repo here: https://github.com/maxnoe/pybind_protobuf_test

Experimenting with this, I had several issues. However now in this version including the sources my main problem is that a protobuf header is not found:

❯ cmake --build build
-- Found pybind11: /usr/include (found version "2.9.0")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/maxnoe/Projects/protobuf_pybind/build
Consolidate compiler generated dependencies of target fastproto
[ 14%] Building CXX object CMakeFiles/fastproto.dir/extern/pybind11_protobuf/pybind11_protobuf/proto_cast_util.cc.o
/home/maxnoe/Projects/protobuf_pybind/extern/pybind11_protobuf/pybind11_protobuf/proto_cast_util.cc:19:10: fatal error: python/google/protobuf/proto_api.h: No such file or directory
   19 | #include "python/google/protobuf/proto_api.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/fastproto.dir/build.make:126: CMakeFiles/fastproto.dir/extern/pybind11_protobuf/pybind11_protobuf/proto_cast_util.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/fastproto.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

It seems to be this one:
https://github.com/protocolbuffers/protobuf/blob/master/python/google/protobuf/proto_api.h

But that seems not to be part of an installed protobuf.

Any help on getting this to run would be appreciated.

I have a simple example that works perfectly when protoc generates native Python for protobufs but is not working correctly when using the C++ implementation (i.e. cleaning, setting export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp, then rebuilding)

The full example is here https://github.com/srmainwaring/pybind11_protobuf/tree/feature/extras.

System: macOS Big Sur 11.6.1
Python: 3.9.9 installed with brew
protoc: 3.19.1 installed with brew

The relevant portion of the stack trace is:

    msg = m.set_vector3d(msg)
TypeError: set_vector3d(): incompatible function arguments. The following argument types are supported:
    1. (msg: extras::msgs::Vector3d) -> None

Invoked with: x: 1.0
y: 2.0
z: 3.0

where the example is a variation of the basic example from the README comprising a pair of set and get functions returning and accepting a simple message.

The isinstance behaviour of returned types is also not working as expected.

Uninstalling the brew version of protobuf does not change the outcome. I expect this may be a user / configuration issue but can't spot where I'm going wrong. Any help appreciated.

Hi,
Could you please add a complete set of instructions on how to include and use with Bazel?

I added the following stanza to my WORKSPACE file:

git_repository(
    name = "pybind11_protobuf",
    branch = "main",
    remote = "https://github.com/pybind/pybind11_protobuf.git",
    strip_prefix = "pybind11_protobuf",
)

However, I cannot get the requirement for a com_google_protobuf dependency to work.
I believe the current practice would be to add this dep to MODULE.bazel:
https://registry.bazel.build/modules/protobuf/21.7

and then add this to the BUILD rule:
"@protobuf//:protos_python",

This however gives me this error:
no such package '@com_google_protobuf//': The repository '@com_google_protobuf' could not be resolved: Repository '@com_google_protobuf' is not defined and referenced by '@pybind11_protobuf//:native_proto_caster'

What's the correct way to get this to work?
(for future compatibility, I have filed a BCR request here bazelbuild/bazel-central-registry#1121 )

For this test I pulled the example code from
https://github.com/davidtwomey/pybind11_protobuf_example.git and updated it to use native_proto_caster + latest versions of libs
pybind_proto_example.zip
.

The issue is that the proto created in python has the type <class 'example_pb2.TestMessage'>
but the proto created in C++ has the type <class 'TestMessage'>

The module name is not preserved despite package being specified in example.proto. This breaks isinstance().

We see even more weirdness in our code with nested protos. Sometimes the nested proto includes the full module name (python-style) and sometimes it doesn't (C++ style).

Questions:

1. Why is it happening?
2. Is it safe?

To test:
unzip the attached code and run bazel run example

Has pybind11_protobuf been tested with Python 3.11 and forward?
I believe PyFrameObject has been removed from the public API and curious to know someone has tested it working with this version or later.

This looks like it's from google3.
Consider renaming?
I'll fork from here.

Hello, I am trying to integrate the latest pybind11_protobuf (commit a3d93a93387af7fa57d72d56cfc0a4ba7f4a60e4) into my project which uses pybind 2.4.3 and bazel.

I am running into a compiler error on what seems to be an incompatibility between pybind11 and pybind11_protobuf. Do you have any advice for how to resolve this?

In file included from external/com_github_pybind_pybind11_protobuf/pybind11_protobuf/native_proto_caster.cc:1:
external/com_github_pybind_pybind11_protobuf/pybind11_protobuf/native_proto_caster.h:102:8: error: too many template arguments for class template 'move_only_holder_caster'
struct move_only_holder_caster<
       ^
external/pybind11/include/pybind11/cast.h:1518:8: note: template is declared here
struct move_only_holder_caster {
       ^
In file included from external/com_github_pybind_pybind11_protobuf/pybind11_protobuf/native_proto_caster.cc:1:
external/com_github_pybind_pybind11_protobuf/pybind11_protobuf/native_proto_caster.h:154:8: error: too many template arguments for class template 'copyable_holder_caster'
struct copyable_holder_caster<
       ^
external/pybind11/include/pybind11/cast.h:1438:8: note: template is declared here
struct copyable_holder_caster : public type_caster_base<type> {
       ^
2 errors generated.

Here is my bazel rule for pybind11_protobuf:

cc_library(
    name = "pybind11_protobuf",
    hdrs = glob([
        "pybind11_protobuf/*.h",
    ]),
    deps = [
        "@pybind11//:pybind11",
        "@com_google_protobuf//:protobuf",
        "@com_google_protobuf//:proto_api",
        "@com_google_absl//absl/container:flat_hash_map",
        "@com_google_absl//absl/strings",
        "@com_google_absl//absl/types:optional"
    ],
    visibility = ["//visibility:public"],
    copts=[
        "-Iexternal/pybind11/include/include",
        "-Iexternal/pybind11/include",
        "-Iexternal/com_google_protobuf/include",
        "-Iexternal/com_google_absl/include"
    ]
)

I wanted to use this in an environment where the pyext for proto_api wasn't build/installed and as enable_pyproto_api_setting is disabled by default I assumed this would work.

However the build failed already due to an unconditional dependency on @com_google_protobuf//:proto_api and include of python/google/protobuf/proto_api.h

I made a couple changes such that it builds with the version TF 2.13.0 uses: Flamefire@f49bc41

However on current main it seems to be much harder as now check_unknown_fields depends on that too which makes it look like it may not work that easily anymore.

Is there interest in getting this fixed/done? Any feedback on the feasibility of the above change/commit?

The usecase was to compile TensorFlow with a pre-installed protobuf to avoid conflicts when using potentially different versions in one environment.

Thanks for the great work!

I was using protobuf 3.15.3 and encountered the following error message.
'const struct google::protobuf::python::PyProto_API' has no member named 'DescriptorPool_FromPool'

I checked https://github.com/protocolbuffers/protobuf/blob/v3.15.3/python/google/protobuf/proto_api.h and found there is no such an API in protobuf 3.15.3. Is is possible to integrate the project with the given version? Thanks.

proto_cast_util has a dependency on "@com_google_protobuf//:python/google/protobuf/pyext/_message.so". However, it is not visible.

Building with --check_visibility=false succeeds however.

A Bazel BUILD file would be nice... or should this be included in https://github.com/pybind/pybind11_bazel instead?

Add minimal GHA workflow to run on Ubuntu latest that does a build and test using the CMake build (#73).

I am encountering different behaviour in trying to run tests depending on whether protobuf is already installed in the default python executable (in my case /usr/bin/python3.8). OS = Ubuntu 20

Any help would be much appreciated! Perhaps I am missing something obvious here.

Without pre-existing protobuf py installation 🆗

When running tests without any python protobuf library the bazel run command works fine.
(Although I cannot import the extension outside of bazel run.)

# Check no protobuf py installation
pip show protobuf
$-> WARNING: Package(s) not found: protobuf

git clone https://github.com/pybind/pybind11_protobuf
cd pybind11_protobuf
bazel run pybind11_protobuf/tests:wrapped_proto_module_test

Tests run and pass ok

Running tests under Python 3.8.10: /usr/bin/python3...`
Ran 20 tests in 0.011s
... 
OK

(NOTE however that you cannot run the generated protobuf test_pb2.py file
python bazel-bin/pybind11_protobuf/tests/test_pb2.py

Traceback (most recent call last):
 File "bazel-bin/pybind11_protobuf/tests/test_pb2.py", line 21, in <module>
   create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

With protobuf pre-installed (default) ❌

When I install protobuf via default pip installation before running the tests, I encounter a segmentation fault.

# Install protobuf
pip install protobuf
bazel clean
bazel run pybind11_protobuf/tests:wrapped_proto_module_test

Tests now fail with a Segmentation fault

Running tests under Python 3.8.10: /usr/bin/python3
[ RUN      ] FastProtoTest.test_call_with_none
[       OK ] FastProtoTest.test_call_with_none
[ RUN      ] FastProtoTest.test_call_with_str
[       OK ] FastProtoTest.test_call_with_str
[ RUN      ] FastProtoTest.test_equality
Fatal Python error: Segmentation fault

Current thread 0x00007fb44f774740 (most recent call first):
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_pybind11_protobuf/pybind11_protobuf/tests/wrapped_proto_module_test.py", line 76 in test_equality
  File "/usr/lib/python3.8/unittest/case.py", line 633 in _callTestMethod
  File "/usr/lib/python3.8/unittest/case.py", line 676 in run
  File "/usr/lib/python3.8/unittest/case.py", line 736 in __call__
  File "/usr/lib/python3.8/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.8/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.8/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.8/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.8/unittest/runner.py", line 176 in run
  File "/usr/lib/python3.8/unittest/main.py", line 271 in runTests
  File "/usr/lib/python3.8/unittest/main.py", line 101 in __init__
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/testing/absltest.py", line 2409 in _run_and_get_tests_result
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/testing/absltest.py", line 2438 in run_tests
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/testing/absltest.py", line 2122 in main_function
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/app.py", line 251 in _run_main
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/app.py", line 303 in run
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/testing/absltest.py", line 2124 in _run_in_app
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_absl_py/absl/testing/absltest.py", line 2007 in main
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_pybind11_protobuf/pybind11_protobuf/tests/wrapped_proto_module_test.py", line 88 in <module>

With protobuf pre-installed (cpp_implementation build flag) ❌

Another variant of protobuf installation where I specify a pip install option which forces it to fall back on a setup.py build.

# Remove previous version
pip uninstall protobuf
# Install protobuf
pip install protobuf==3.18 --install-option="--cpp_implementation"
# Check protobuf implementation
python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"
$ -> cpp

bazel clean
bazel run pybind11_protobuf/tests:wrapped_proto_module_test

Tests fail

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pybind11_protobuf/tests:wrapped_proto_module_test
-----------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_pybind11_protobuf/pybind11_protobuf/tests/wrapped_proto_module_test.py", line 14, in <module>
    from pybind11_protobuf.tests import compare
  File "/home/dtwomey/.cache/bazel/_bazel_dtwomey/866346c0fc431ca03a19bd6a2c5a1c06/execroot/com_google_pybind11_protobuf/bazel-out/k8-fastbuild/bin/pybind11_protobuf/tests/wrapped_proto_module_test.runfiles/com_google_pybind11_protobuf/pybind11_protobuf/tests/compare.py", line 50, in <module>
    from google.protobuf import descriptor
  File "/home/dtwomey/.local/lib/python3.8/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: /home/dtwomey/.local/lib/python3.8/site-packages/google/protobuf/pyext/_message.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZNK6google8protobuf10TextFormat21FastFieldValuePrinter19PrintMessageContentERKNS0_7MessageEiibPNS1_17BaseTextGeneratorE