Serialization/deserialization test of 'bytes' type is not implemeted

Make automatic linking of generated code static libraries "whole-archive"

Move assigment doesn't emit field changed signal for repeated/map types.

For message types it's only required internal fields update.

How hard are the version requirements on protobuf (3.6.0) and grpc (1.15.0)?
I ask because I'm trying to make qprotobuf work on the NVIDIA Jetson Nano, and there's no good way (that I've found so far) to upgrade it beyond Ubuntu 18.04, which includes protobuf 3.0.0 and grpc 1.3.2 . My attempts to compile them from source have been frustrated -- in the compilation of grpc there's a compile error similar to what's described in facebook/folly#1098 but without the option of exercising that OP's fix.
I've also tried making gcc8 the default -- that didn't help either.
I think that gprotobuf 3.6 is not going to be a problem; I think it's just grpc.

Are there specific grpc-1.15.0 features/fixes that are needed for qtprotobuf?

Access to gRPC client calls should be thread safe

Currently 2 styles of cmake are variables defined in project QTPROTOBUF_ and QT_PROTOBUF

Using "transparent" type to define fields of output C++ Qt classes introduces bugs and potential pitfalls - since it allows a value to be assigned to a getter function call. Here is some code to illustrate:

transparent<int> m;

transparent<int> GetM() { return m; }

int n;

int GetN() { return n; }

int main()
{
    std::cout << "Hello World!\n";

    GetM() = 25; // No warning or compiler error
    GetN() = 26; // error C2106: '=': left operand must be l-value 
}

Assigning a value to the transparent value getter appears legal and does not produce a warning or an error. This leads to logic errors since usage of setters is no longer enforced - and the assignment does not produce an expected result.

For us, this is a big issue. We are trying to replace a bunch of plain C structs with proto-generated C++ Qt classes. Simply replacing struct field references with class getters results in hard-to-detect errors - since assigning a value to a getter is legal.

The types using "transparent" wrapper break a lot of QML functionality: To demonstrate, I created this proto:

syntax = "proto3";

package test;

message Test1 {
	int32 count = 1; 
}

The generated class defines the property as follows:

I also created another class Test2 manually, using "int" as the property type:

I then registered both types with QML:

    // Register Qt Protobuf-generated types
    QtProtobuf::qRegisterProtobufTypes();

    qmlRegisterType<Test2>("test", 1, 0, "Test2");

Problem # 1: Property assignment is broken

    property Test1 test1: Test1 {
        count: 0
    }

    property Test2 test2: Test2 {
        count: 0
    }

The above results in error:

qrc:/main.qml:13 Invalid property assignment: unsupported type "QtProtobuf::int32"

So, the assignment is ok for Test2 but not for Test1.

Problem # 2: Implicit boolean conversion is broken

I comment out the broken assignment so that both test1 and test2 have count = 0 and display the following:

    Row {
        Label {
            text: test1.count ? "TRUE" : "FALSE"
        }

        Label {
            text: "   "
        }

        Label {
            text: test2.count ? "TRUE" : "FALSE"
        }
    }

The result is:

It was expected that test1.count (being an integer equal to 0) would convert to "false" - but the opposite is the case. I guess that the implicit conversion in this case is the "null" test instead.

Problem # 3: Locale formatting is broken

The transparent types are not recognized as "Number" by QML. So the following code also results in unexpected output:

    Row {
        Label {
            text: test1.count.toLocaleString(Qt.locale())
        }

        Label {
            text: "   "
        }

        Label {
            text: test2.count.toLocaleString(Qt.locale())
        }
    }

Results in:

Is there any way to make "transparent" types compatible with expected QML behavior, or to get rid of these wrappers?

In building qtprotobuf, I had to explicitly append a "-lpthread" to some of the build lines (like for test_protobuf) in order to get them to successfully link. It think that's a gcc 8 thing.

Currently generate_qtprotobuf put target library name to QtProtobuf_GENERATED variable in parent scope.

Suggest to add argument to generate_qtprotobuf function, that will specify variable name that will be used to store generated target name.

At the moment Qt Protobuf output provides a convenient way to register Qt types through qRegisterProtobufTypes. However, to make the types available to QML, qmlRegisterType has to be still manually called for each new type.

Could you add QML type registration to qRegisterProtobufTypes as well?

Thanks,

Michael

I'm now trying to include google.protobuf.Timestamp in a message, and I haven't been able to find anything in the documentation regarding special steps that [may] need to be taken in order for this to work. Could you please elaborate a little on how that should be done?

Current gRPC API doesn't fully supported by QML. Need to design and implement additional QML helper classes and make deeper integraion.

serializationplugin installs artifacts by make install rule, but should not

It seems that only the class output of only the last compiled .proto file is included in "qtprotobuf_global.qpb.h". For example, if the project contains Test1.proto and Test2.proto, during build I see the following:

Compiling Protobuf schema Test1.proto (Qt)
Compiling Protobuf schema Test2.proto (Qt)

So, protoc is executed twice - once for each file. The output ""qtprotobuf_global.qpb.h" file contains the following (note that Test1 is missing):

/* This file is autogenerated. DO NOT CHANGE. All changes will be lost */

#pragma once

#include
#include
#include
#include
#include

#include
#include <unordered_map>

#include "Test2.qpb.h"

namespace nvram {
inline void qRegisterProtobufTypes() {
qRegisterProtobufType<Test2>();
}
}

Add code annotations/comments/doxygen to generated header files.

For repeated elements, was there a recent change to implement those using QList instead of QVector? See https://marcmutz.wordpress.com/effective-qt/containers/ scroll down to the QList description.

When I compile the project according to your instructions I get the following error

simpletest.cpp:(.text._ZN19qtprotobufnamespace5tests20StepChildEnumMessageD2Ev[_ZN19qtprotobufnamespace5tests20StepChildEnumMessageD5Ev]+0xf): undefined reference to `vtable for qtprotobufnamespace::tests::StepChildEnumMessage'
collect2: error: ld returned 1 exit status
gmake[2]: *** [tests/test_protobuf/CMakeFiles/qtprotobuf_test.dir/build.make:167: tests/test_protobuf/qtprotobuf_test] Fehler 1
gmake[1]: *** [CMakeFiles/Makefile2:583: tests/test_protobuf/CMakeFiles/qtprotobuf_test.dir/all] Fehler 2
gmake: *** [Makefile:141: all] Fehler 2

Do you have an idea what I'm doing wrong?

Thanks for your help

Global enums currently have lots of limitations and should be replaced with enum-based classes.

3rd party modules like protobuf, grpc, should be excluded out of installation rules.

There is no standard UUID message distributed with protobuf. However, it would be really nice to be able to automatically convert a message member to a QUuid, don't you agree?

This may entail distributing a set of .proto files with qtprotobuf, that, if used as part of the user's .proto-defined messages, would get automatically translated that way.

UPD by @semlanik:
Following types will be added as less-well-known types:

• QUrl

• QChar

• QUuid

• QLocale - Not unified

• QColor

• QDate

• QTime

• QDateTime

• QSize

• QSizeF

• QRect

• QRectF

• QPolygon

• QPolygonF

• QPoint

• QPointF

• QImage

• QPixmap - is internal Qt representation of image. Not sure if applicable to transfer it using protocol buffers

• QIcon - not something that has own data.

• QBitmap - inherited of QPixmap

• QQuaternion

• QTransform

• QMatrix - Deprecated by Qt.

• QMatrix4x4

• QVector2D

• QVector3D

• QVector4D

• QEasingCurve - skip for now. Will add by request. It seems this field should be of google.protobuf.Any type

Any other protoc generator that outputs C++/C# code uses name of the source file for the output file name, producing only one output source file for C# and a pair of H/CPP for C++.

For example, "security.proto" defining Account and Login messages would result in "security.pb.h" and "security.pb.cc" output files, containing both Account and Login classes.

qtprotobuf generates a pair of H/CPP for each message and uses class name for naming each pair.

For example, the same "security.proto" from above results in "Account.cpp", "Account.h", "Login.cpp", "Login.h".

Here are the problems with this approach:

1. It deviates from convention, producing confusion. It is no longer apparent which proto file produces a certain generated file - unless you go into code and search by name.

2. It breaks custom build step used by Visual Studio, since for proper cleanup/rebuild the step needs to know each output file name - which becomes a tedious manual work when there are multiple messages in a proto file. Adding a new message to existing proto file would involve going into custom build step setting for that proto file and manually adding names of the output files for the new message:

For a single pair of "pb.h" and "pb.cc" per proto file, custom build step output files can be generically specified as:
$(InputDir)\$(InputName).pb.cc;$(InputDir)\$(InputName).pb.h

It looks to me that the static data member of QProtobufSerializerPrivate, "SerializerRegistry handlers" needs to be protected by a mutex. My program frequently crashes when two separate threads both instantiate a QProtobufSerializer at the same time. Better yet, have you considered using Q_GLOBAL_STATIC? You might still need a mutex (I would use Q_GLOBAL_STATIC_WITH_ARGS to give it a QMutex::Recursive behavior) depending on whether you decide to change how "handlers" gets populated.

I think that the primary advantage of Q_GLOBAL_STATIC lies in its ability to preserve the uniqueness of "handlers" across shared library (DLL) boundaries.

Add .pri file for qtprotobuf to use it from qmake-based projects

Not all maps variants are covered by tests yet

Add compilation of static version of QtProtobuf and QtGrpc libraries

Implement JSON serializer/deserializer according to protobuf rules.

add qmake related prl files to import QtProtobuf libraries and generation using qmake

A while ago, I reported issues with statically registering Qt protobuf types due to "Static Initialization Order Fiasco." In response to that report, you removed automatic registration and replaced it with the "qRegisterProtobufTypes" function in qtprotobuf_global.qpb.h that can be called manually by the developer. The main problem with this approach for Windows developers (who often do not use CMAKE for VS projects) is that only the last processed type shows up in qRegisterProtobufTypes if the .proto files are processed manually with "protoc." So, such developers (including me) are forced to manually add qRegisterProtobufType<...>() call for each new type. Here is the better way which avoids both issues:

1. Define static namespace to hold type registration function lambdas for the types:

a) QtProtobufRegister.h

namespace QtProtobufRegister
{
	typedef std::function<void(void)> RegisterFunction;

	void Init();
	QList<RegisterFunction>& GetList();

	template <class T>
	struct Type {
		Type(RegisterFunction initializer)
		{
			GetList().append(initializer);
		}
	};
};

b) QtProtobufRegister.cpp

#include "QtProtobufRegister.h"

namespace QtProtobufRegister
{
	QList<RegisterFunction>& GetList()
	{
		static QList<std::function<void(void)>> registrationList;
		return registrationList;
	}

	void Init()
	{
		for (auto registerFunc : GetList())
		{
			registerFunc();
		}
	}
}

2. For generated code, add the following static variable to the class CPP file (inside namespace but before the class definition):

    static QtProtobufRegister::Type<{CLASS_TYPE}> s_protobufRegister([](){
        qRegisterProtobufType<{CLASS_TYPE}>();
    });

3. When it is finally safe to register the Qt types, a developer can simply call the following to register all Qt Protobuf types all at once:

QtProtobufRegister::Init();

The way this works is for every class its registration call is not performed immediately when the class is loaded statically - but rather is stored in the list. When 'QtProtobufRegister::Init()' is caller - all registration function from that list are executed, avoiding the Static Initialization fiasco.

I've successfully used a similar approach to register over 100 QML types in our application.

Another advantage is that you can drop "qtprotobuf_global.qpb.h" file as redundant.

Implement server-side ZeroMQ channel support

Add ci procedures for static QtProtobuf

Need to pass correct include path for protoc generation function of well-known types.

The generator produces this "helpful" piece of code in the output:

static QtProtobuf::QProtobufRegistrationHelper helper(registerTypes);

This static variable often leads to crash on startup when running VS-compiled executables, due to well-known "static order initialization fiasco":

https://stackoverflow.com/questions/41938297/calling-qmlregistertype-in-the-registered-class-on-debug-crashes-application

Basically, at the time QProtobufRegistrationHelper tries to register the type, the type's static metadata has not been created yet - causing the crash (since the order is indeterminate).

Suggestion - do not attempt to call registerTypes() statically. Instead, provide a static method and leave the decision up to a developer to execute when appropriate

Better suggestion - store pointer to each registerTypes() in a static list. Provide static function to call all functions on the list at the time of developer's choosing.

Example async servers memory management is awful and badly designed. Need to fix leaks.

Regarding the doxygen generated content -- it would make my life easier if there weren't any references to https://fonts.googleapis.com/ as it relates to stylesheets. When that host can't be reached, it causes the page to load very slowly.

Implement generation of gRPC server stubs. Define basic server interface and work with channels

Add QML_PLUGIN option for generator

Add QML plugin generation for each qtprotobuf target, that enabled QML_PLUGIN option

Generation of protobuf classes should be folder-based.

Each generated message should be located in package specific folder.

The Jenkins build fails on Windows with these lines:

Generating generated/moc_simplefixed32sint32mapmessage.cpp
E:\Jenkins\workspace\QtProtobuf\3rdparty\grpc\third_party\boringssl\tool\transport_common.cc(19): error C2220: warning treated as error - no 'object' file generated [E:\Jenkins\workspace\QtProtobuf\build\3rdparty\grpc\third_party\boringssl\tool\bssl.vcxproj]
E:\Jenkins\workspace\QtProtobuf\3rdparty\grpc\third_party\boringssl\tool\transport_common.cc(19): warning C4005: '_SCL_SECURE_NO_WARNINGS': macro redefinition [E:\Jenkins\workspace\QtProtobuf\build\3rdparty\grpc\third_party\boringssl\tool\bssl.vcxproj]
E:\Jenkins\workspace\QtProtobuf\3rdparty\grpc\third_party\boringssl\tool\transport_common.cc: note: see previous definition of '_SCL_SECURE_NO_WARNINGS'

The build is performed as follows:

1. Fresh code is checked out of "https://github.com/semlanik/qtprotobuf"
2. Path is set as "Path=$Path;C:\Qt\5.13.2\msvc2017\bin"
3. call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build\vcvars32.bat"
4. git submodule update --init --recursive
5. mkdir build
6. mkdir install
7. cd build
8. "C:\Program Files\CMake\bin\cmake.exe" .. -DCMAKE_PREFIX_PATH="C:\Qt\5.13.2\msvc2017\lib\cmake" -DCMAKE_INSTALL_PREFIX="..\install" -DCMAKE_DEBUG_POSTFIX=D -G "Visual Studio 15 2017"
9. "C:\Program Files\CMake\bin\cmake.exe" --build . --config Debug --target INSTALL -- /m:4
10. "C:\Program Files\CMake\bin\cmake.exe" --build . --config RelWithDebInfo --target INSTALL -- /m:4

Please find the full log below:

log.txt

The generator does not handle field names with "_" correctly and generates different names in ".h' and ".cpp" output files - causing compiler failure.

For example, consider this proto:

syntax = "proto3";

package test;

message Test
{
	string some_field = 1;
}

Output contains these lines in ".h":

private:
    QString m_someField;

While ".cpp" contains:

Test::Test(const QString &some_field, QObject *parent) : QObject(parent)
    , m_some_field(some_field)
{
}

Note the difference in field naming (m_someField vs m_some_field).

P.S. It would be great to be able to select the naming convention of the output somehow - so that the output will match VS formatting used by the developer.

Hello,
I have a freshly installed Ubuntu 19.10 VM and can't seem to get cmake to be happy ... see attached.
Glen

cmake_stdout.txt
CMakeFiles.tar.gz

Create ubuntu deb package artifacts.

Could you please briefly articulate your vision for supporting Timestamp? I might be able to take a shot at implementing it.

Create rpm package artifacts for yum/zypper

Design ZeroMQ based communication protocol for QtProtobuf

Implement client-side ZeroMQ channel