kxsystems / protobufkdb Goto Github PK

To reflect split example q scripts
Also add before and after compares to the example q scripts

If a protobuf schema only contains fields of the same type, e.g.:

message Basic {
    int32 first = 1;
    int32 second = 2;
}

you cannot create the necessary two -KI atom mixed list in q to satifisy the message type checking since q will convert it to a KI simple list:

q)type (1i;2i)
6h

Solution is to allow general nulls (::) to be present in the message's mixed list to prevent the conversion:

q)type (1i;2i;::)
0h

Any fields set to :: should be ignored. Also the type checking on incorrect number of message fields need to be loosened, only throwing an error is the mixed list length is less that the number of fields in the message.

In attempt to run the package in rhel6 I am getting an error
usr/lib64/libstdc++.so.6 version GLIBCXX_3.4.18 not found required by protobufkdb.so
The above version of package isnt available for rhel6 machines.

I'm trying to use protobufkdb to parse the following schema:

.protobufkdb.addProtoImportPath["proto/mark/v1"];
.protobufkdb.importProtoFile["mark.proto"];
.protobufkdb.displayMessageSchema[`model.mark.v1.Mark]
message Mark {
  double latpoint = 1;
  double longpoint = 2;
}
.protobufkdb.serializeArrayFromDict[`model.mark.v1.Mark;`latpoint`longpoint!(100f;100f)]

evaluation error:

Invalid message field values, descriptor: 'model.mark.v1.Mark', expected: 0, received: 9

[0] .protobufkdb.serializeArrayFromDict[model.mark.v1.Mark;latpoint`longpoint!(100f;100f)]

   ^

Furthermore, I recieve the same result from a list:
.protobufkdb.serializeArrayFromList[model.mark.v1.Mark;(100f;100f)]`

Invalid message field values, descriptor: 'model.mark.v1.Mark', expected: 0, received: 9

[0] .protobufkdb.serializeArrayFromDict[`model.mark.v1.Mark;(100f;100f)]

   ^

using:

• ubuntu:18.04
• libprotoc 3.0.0
• built from source with cmake as detailed in docs
• KDB 64bit edition
• protobufkdb version 3014000i

Could someone advise me on how to fix this?
Thanks

Once repo is public and travis builds are running. Should use similar test runner as hdf5

Would there be anything impeding the use of protobufkdb with the 32bit version of kdb?

Currently protobufkdb represents a proto message in kdb as a mixed lists of field values in field positional order. This is for performance reasons and to more closely tie in with how protobuf serialises messages.

However, some users may prefer messages to use a dictionary from symbol field names to a mixed list of field values, more similar to how JSON is represented. In this case each field would be looked up by its name so the positional order is no longer required.

Both styles should be supported by separate APIs. So for each of the serialisation and parsing functions both the mixed list and dictionary versions are exposed, e.g.:

serializeArray
becomes
serializeArrayFromList
serializeArrayFromDict

and:

parseArray
becomes
parseArrayToList
parseArrayToDict

Note the dictionary versions need to work recursively on sub-messages or maps to messages. To help with using the dictionary version, it would be helpful to have a function which returns a symbol list of the field names for the specified proto message schema.

• importProtoFile does not clear the errors list between calls
• Errors returned by importProtoFile are formatted differently to the same errors when generated by protoc
• When installing the packaged version of protobufkdb, not all of Google's .proto files are available

Currently these are all represented as symbols. This should be changed on the parsing path to use kdb char array and byte array as follows:

On the serialisation path is should be possible to use any of -11|4|10h interchangeably and protobufkdb will convert it under the covers. However, it is recognised that one disadvantage of this flexible approach is that you lose the round trip reflective behaviour – so while for serialising a proto string you could use any of -11|4|10h, on parsing you'll always get a 10h back.

New docs folder wont currently be added to a release build - should be included in the future while docs are there.
.travis.yml example area
e.g.
elif [[ $TRAVIS_OS_NAME == "windows" ]]; then
7z a -tzip $FILE_NAME README.md install.bat LICENSE q examples;
elif [[ $TRAVIS_OS_NAME == "linux" || $TRAVIS_OS_NAME == "osx" ]]; then
tar -zcvf $FILE_NAME README.md install.sh LICENSE q examples;