jangko / msgpack4nim Goto Github PK

Hi, I've been trying to create a simple proc that uses MSGPACK_OBJ_TO_MAP as the default, much like the current implementation of pack. However, it seems that creating this helper proc and importing it from somewhere else makes it fail with undeclared identifier: 'pack_type'.

Here's a minimal example:

file:util.nim

import msgpack4nim

proc mapPack*[T](val: T): string =
  var s = MsgStream.init(sizeof(T), MSGPACK_OBJ_TO_MAP)
  s.pack(val)
  return s.data

when isMainModule:
  echo mapPack("s")

file: main.nim

import util

echo mapPack("s")

If run nim c -r util.nim, it works. But when running nim c -r main.nim, it complains that pack_type is undeclared when used in undistinct_pack.

I tried replicating it with simple template, but my skills with macro/template is still too green to understand why this is happening, so I wasn't able to create a minimal examples without msgpack.

Any insights?

Unlike the regular Stream type, MsgStream does not have a public writeData method, so I/O functions like pack_type using a generic [ByteStream] will fail to compile when ByteStream is MsgStream.

There’s no efficient workaround for writing a buffer of binary data. The alternatives are to copy the buffer to a string first and call write(string), or to write the individual bytes as write(byte). The first allocates memory and the second is quite slow.

The generic write(MsgStream, T) seemed like it would work, but if I call it with a seq[byte] it ends up writing the pointer as 8 or 4 bytes, which is of course garbage.

Simply making the writeData method (at msgpack4nim.nim:69) public would resolve the problem.

(I’m not sure why MsgStream is an independent type and not an object of Stream? Inheriting from Stream would fix API incompatibilities and make your code smaller.)

When deserializing a map that has an extra field to an obj, deserialization stops with ObjectConversionError when no field has the same name as key because it skips unpacking of value, and continues trying to unpack value to name (string).

proc unpack_type*[ByteStream; T: tuple|object](s: ByteStream, val: var T) =
  template dry_and_wet(): untyped =
    when defined(msgpack_obj_to_map):
      let len = s.unpack_map()
      var name: string
      for i in 0..len-1:
        unpack_proxy(name)  # Tries to unpack map value here if skipped value unpacking
        for field, value in fieldPairs(val):
          if field == name:
            unpack_proxy(value) # Skips this line if no match

Simply forcibly unpacking a value (and catching ObjectConversionError) is enough to make deserialization work, but it's an ugly hack (with exception raising). There should be a way to skip a value, no matter its type, and allow the map deserialization continue normally.

Hey there,

Love the library, been using a heap!

In our embedded firmware, we've got some {.gcsafe.} procs that call out to unpack -- with v0.3.1 this worked no worries, but as of v0.4.0 (though the tag says v0.4.2 here on Github? I can't install it via Nimble though, only v0.4.0):

/root/.nimble/pkgs/msgpack4nim-0.4.0/msgpack4nim.nim(987, 6) Warning: 'skip_msg' is not GC-safe as it calls 'unpack_ext' [GcUnsafe2]
/root/.nimble/pkgs/msgpack4nim-0.4.0/msgpack4nim.nim(1022, 6) Warning: 'unpack_type' is not GC-safe as it calls 'skip_msg' [GcUnsafe2]
/root/.nimble/pkgs/msgpack4nim-0.4.0/msgpack4nim.nim(1167, 6) Warning: 'unpack' is not GC-safe as it calls 'unpack_type' [GcUnsafe2]
/root/.nimble/pkgs/msgpack4nim-0.4.0/msgpack4nim.nim(1174, 6) Warning: 'unpack' is not GC-safe as it calls 'unpack' [GcUnsafe2]
/workspaces/firmware/main/kernel/comms_thread.nim(85, 6) Error: 'auth' is not GC-safe as it calls 'unpack'

Any ideas with this one?

If I want to do the following :
proc pack_type*[ByteStream](s: ByteStream, x: Tensor[int64])= #...
proc pack_type*[ByteStream](s: ByteStream, x: Tensor[int64])= #...

I would like to be able to do the following :
proc pack_type*[ByteStream, T: SomeFloat|SomeInteger](s: ByteStream, x: Tensor[T])= #...

But the specialized version I define is not called when I use pack(...). Same issue with unpack

Hi! Was about to play around with this but I've run into this bump in the road:

nimble install msgpack4nim

Downloading https://github.com/jangko/msgpack4nim using git
Tip: 1 messages have been suppressed, use --verbose to show them.
Error: Traceback (most recent call last)
... nimble.nim(1073) nimble
... nimble.nim(999) doAction
... nimble.nim(606) install
... nimble.nim(570) downloadPkg
... download.nim(199) doDownload
... download.nim(165) verifyClone
... packageparser.nim(325) getPkgInfo
... packageparser.nim(313) getPkgInfoFromFile
... packageparser.nim(307) readPackageInfo
... packageparser.nim(125) validatePackageInfo
... packageparser.nim(28) raiseNewValidationError
... The .nimble file name must match name specified inside /tmp/nimble_53350/githubcom_jangkomsgpack4nim/msgpack.nimble

I have some data stored as msgpack file in python. The file contains more than 250columns.

In order to read the file, do I need to create a type that holds the data? Is there a workaround for that? I mean, something that creates that type definition from the file structure.

Unpacking a sequence of ref object in unpack_items_imp writes to same address.

The following code should echo 2 3 5 but echos 5 5 5:

import msgpack4nim, streams
import msgpack4nim / msgpack4collection
import lists

type Element = ref object
  value: int

proc newElement(value: int): Element =
  result = new Element
  result.value = value

var elementList = initDoublyLinkedList[Element]()
elementList.add(newElement(2))
elementList.add(newElement(3))
elementList.add(newElement(5))

var strm = MsgStream.init()
strm.pack(elementList)

var sstrm = MsgStream.init(strm.data)
var unpackedList: DoublyLinkedList[Element]
sstrm.unpack(unpackedList)

for e in unpackedList:
  echo e.value

Moving var x: T into the for loop in the unpack_items_imp template fixed the problem for me.

I see no tests for pack_type/unpack_type.

I need to see those so I can understand what I'm doing wrong. I have this:

proc unpack_type*(ss: MsgStream, x: var str9) =
  var str: string
  ss.unpack(str)
  copyMem(addr x[0], addr str[0], 9)

And I get this:

falcon/rr_hctg_track.nim(65, 5) Error: type mismatch: got <MsgStream, string>
but expected one of:
proc unpack[T](data: string; val: var T)
  first type mismatch at position: 1
  required type: string
  but expression 'ss' is of type: MsgStream

It worked with older msgpack and nim-0.18.0. I have tried this:

-proc unpack_type*(ss: streams.Stream, x: var str9) =
+proc unpack_type*(ss: MsgStream, x: var str9) =
   var str: string
-  msgpack4nim.unpack(ss, str)
+  ss.unpack(str)
   copyMem(addr x[0], addr str[0], 9)

I tried to send a command to neovim and receive a response using msgpack-rpc.
I added * to proc stringify[ByteStream](s: ByteStream, zz: ByteStream) in msgpack4nim.nim and tried to output stringified response from neovim to stdout.
But I got runtime error as Stream returned from proc outputStream(p: Process): Stream doesn't have peekDataImpl on windows.

Should peekDataImpl be implemented for proc outputStream(p: Process): Stream?
Or msgpack4nim should not use peek?

import osproc, streams
import msgpack4nim, macros

macro writeRPCReq[ByteStream](
                              s: ByteStream;
                              methodName: string;
                              id: int;
                              args: varargs[typed]) =
  let
    argsLen = args.len.newLit
  result = quote do:
    `s`.pack_array 4
    `s`.pack_type 0
    `s`.pack_type `id`
    `s`.pack_type `methodName`
    `s`.pack_array `argsLen`
 
  for a in args:
    result.add newCall(bindSym"pack", s, a)

proc main =
  var
    p = startProcess("nvim", args = ["--embed"], options = {poUsePath})
    inStrm = p.inputStream
    outStrm = p.outputStream
    stdoutStrm = stdout.newFileStream
  inStrm.writeRPCReq("nvim_command", 1, "echo $MYVIMRC")
  inStrm.flush()
  outStrm.stringify stdoutStrm.Stream
  p.close()

main()

I don't know this code sends request message to neovim correctly.

After I update to the latest msgpack4nim (720f8bb):

falcon/rr_hctg_track.nim(414, 19) template/generic instantiation from here
../repos/msgpack4nim/msgpack4nim.nim(1069, 53) template/generic instantiation from here
../repos/msgpack4nim/msgpack4collection.nim(38, 4) template/generic instantiation from here
../repos/msgpack4nim/msgpack4nim.nim(681, 6) template/generic instantiation from here
../repos/msgpack4nim/msgpack4nim.nim(727, 32) Error: undeclared field: 'data'
          s.pack_type undistinct(field)
                                 ^

  3 import "../msgpack4nim/msgpack4nim.nim"
  4 import "../msgpack4nim/msgpack4collection.nim"
...
413   var msgss = msgpack4nim.MsgStream.init() #you can pass some buffer capacity here https://github.com/jangko/msgpack4nim/is    sues/18
414   msgpack4nim.pack(msgss, rtn)
415   var ss = streams.newFileStream(fn_rtn, system.fmWrite)
416   defer: ss.close()
417   ss.write(msgss.data)

What do I need to change?

Hi,
I am trying the compilation with Mingw64, the command
nim c -r test.nim
results the following error:
msgpack.nim(764, 24) template/generic instantiation from here
msgpack.nim(751, 12) Error: type mismatch: got (int64) but expected 'int'

Could you please check it.

Thanks,
Louis

Hello,
packing from and unpacking to JsonNode would be really handy.

Hi,
I'm tring to run "make" command in /nim-falcon/repos/msgpack4nim/
But I get this error:

nim --threads:on --threadAnalysis:off --verbosity:2 -d:release  --tlsemulation:on --out:fc_rr_hctg_track2.exe c fc_rr_hctg_track2.nim
config/nim.cfg(42, 3) Hint: added path: '/rhome/fxie/.nimble/pkgs/' [Path]
config/nim.cfg(44, 5) Hint: added path: '/opt/nimble/pkgs/' [Path]
Hint: used config file '/bigdata/baolab/fxie/new2/nim/Nim/config/nim.cfg' [Conf]
Hint: system [Processing]
Hint: fc_rr_hctg_track2 [Processing]
Hint: rr_hctg_track [Processing]
Hint: msgpack4nim [Processing]
Hint: streams [Processing]
Hint: endians [Processing]
Hint: strutils [Processing]
Hint: parseutils [Processing]
Hint: math [Processing]
Hint: algorithm [Processing]
Hint: sequtils [Processing]
Hint: hashes [Processing]
Hint: tables [Processing]
Hint: intsets [Processing]
Hint: lists [Processing]
Hint: queues [Processing]
Hint: sets [Processing]
Hint: strtabs [Processing]
Hint: os [Processing]
Hint: times [Processing]
Hint: posix [Processing]
Hint: ospaths [Processing]
Hint: critbits [Processing]
Hint: macros [Processing]
/bigdata/baolab/fxie/new2/Falcon/FALCON-integrate/nim-falcon/repos/msgpack4nim/msgpack4nim.nim(49, 50) Error: type mismatch: got (uint16) but expected 'uint8'
make: *** [fc_rr_hctg_track2.exe] Error 1

My python version is python/2.7.13-venv.

What should I do to solve this issue?
Thanks a lot!

msgpack.nim(505, 6) Error: attempting to call undeclared routine: 'pack_imp_int8'

nim 0.11.3 nimble 0.6.0
...

type
NodeKind = enum # the different node types
nkInt, #a leaf with an integer value
nkFloat, #a leaf with a float value
nkString, #a leaf with a string value
nkAdd, #an addition
nkSub, #a subtraction
nkIf #an if statement
Node = ref NodeObj
NodeObj = object
case kind: NodeKind # the kind field is the discriminator
of nkInt: intVal: int
of nkFloat: floatVal: float
of nkString: strVal: string
of nkAdd, nkSub:
leftOp, rightOp: Node
of nkIf:
condition, thenPart, elsePart: Node

var
aUnion = Node(kind:nkInt, intVal:22)

var file = newFileStream("msg.pack", fmReadWrite)

file.pack(aUnion)

When I try to install msgpack4nim, I get the following warning. I wanted to open an issue to track this issue.

 Installing msgpack4nim@>= 0.2.2
Downloading https://github.com/jangko/msgpack4nim using git
   Warning: Package 'msgpack4nim' has an incorrect structure. The top level of the package source directory should contain at most one module, named 'msgpack4nim.nim', but a file named 'msgpack4collection.nim' was found. This will be an error in the future.
      Hint: If this is the primary source file in the package, rename it to 'msgpack4nim.nim'. If it's a source file required by the main module, or if it is one of several modules exposed by 'msgpack4nim', then move it into a 'msgpack4nim/' subdirectory. If it's a test file or otherwise not required to build the the package 'msgpack4nim.nim', prevent its installation by adding `skipFiles = @["msgpack4collection.nim"]` to the .nimble file. See https://github.com/nim-lang/nimble#libraries for more info.
  Verifying dependencies for [email protected]
 Installing [email protected]
   Success: msgpack4nim installed successfully.

Example

y.nim

import msgpack4nim

template f* =
  discard pack(1)

x.nim

import y 

f()

Output

/home/user/programming/nim/x.nim(3, 2) template/generic instantiation of `f` from here
/home/user/programming/nim/y.nim(4, 15) template/generic instantiation of `pack` from here
/home/user/.nimble/pkgs/msgpack4nim-#head/msgpack4nim.nim(1207, 4) template/generic instantiation of `pack` from here
/home/user/.nimble/pkgs/msgpack4nim-#head/msgpack4nim.nim(1202, 79) template/generic instantiation of `undistinct_pack` from here
/home/user/.nimble/pkgs/msgpack4nim-#head/msgpack4nim.nim(174, 37) Error: undeclared identifier: 'pack_type'

the title says it all

I am trying to use this library to communicate with a Python client. For reasons not under my control, the client serializes some nonhomogeneous lists, such as [1, "hello", {"a": "b"}].

Is there a way to use the library to deserialize this? I have been able to do this in other typed languages by unpacking into a sequence of yet-unkown types (for instance in Scala I deserialize a Seq[Any]), and then manually mapping by position and casting. It is not the cleanest approach, but it would be very useful for me to be able to do something like this

Per title, it'd make package management quite a bit easier. Thank you! 🙏

import msgpack4nim

proc unpack[T](s: string, typ: typedesc[T]): T =
    s.unpack result

echo((-557853050, 0).pack.unpack((int, int)))
echo((-5578530500, 0).pack.unpack((int, int)))

echo((0, -557853050, 0).pack.unpack((int, int, int)))
echo((0, -5578530500, 0).pack.unpack((int, int, int)))

echo((0, -5578530500, 0, 0).pack.unpack((int, int, int, int)))

(Field0: -557853050, Field1: 0)
(Field0: -5578530500, Field1: -1283563204)
(Field0: 0, Field1: -557853050, Field2: 0)
(Field0: 0, Field1: -5578530500, Field2: -1283563204)
(Field0: 0, Field1: -5578530500, Field2: -1283563204, Field3: 0)

Since the serialisation produces a map keyed by field name, I incorrectly assumed that deserialisation would work similarly and that field order would not matter.

Apparently that's not the case. The code here just assumes that the fields are in the same order (and that the number of fields is the same), which makes interop between a nim program and any other language difficult, because data produced in other programs needs to match the defined spec in the nim program.

Ideally the code there should be updated to first deserialise the map first, and then assign values to the object's fields based on the names in the map.

-d:msgpack_obj_to_map is not an option for libraries which are using msgpack4nim. Can we establish a stricter correspondence between objects/tuples/maps/arrays and remove the possibility to tweak it by -d flag? Currently there's no way of producing or reading a map without the flag.

One option would be to support JsonNode for pack/unpack.
Another option would be to make objects correspond to maps, and tuples correspond to lists.

Version: 0.2.9

Commit: 9c4bc6e66d5d193692be0e7c5605166dc61756dc

Observed:
Object variants using standard idiom (with case) packs correctly, but fails to unpack.

Expected:
Object variants with zero-initialized variables should unpack correctly

Steps to Replicate:

import streams, msgpack4nim

type MsgType* = enum
  MSG_FOO,
  MSG_BAR

type Foo* = object
  name: string

type Bar* = object
  price: int

type FooBarMsg* = object
  case msgType*: MsgType
  of MSG_FOO:
    foo*: Foo
  of MSG_BAR:
    bar*: Bar

when isMainModule:
  var fooOriginal = Foo(name: "Baz")
  var barOriginal = Bar(price: 11)
  var outboundMsgFoo = FooBarMsg(
  msgType: MSG_FOO,
  foo: fooOriginal
  )
  var outboundMsgBar = FooBarMsg(
  msgType: MSG_BAR,
  bar: barOriginal
  )

  var sFoo = MsgStream.init()
  sFoo.pack(outboundMsgFoo)

  var sBar = MsgStream.init()
  sBar.pack(outboundMsgBar)

  var incomingFoo: FooBarMsg
  var sFooReception = MsgStream.init(sFoo.data)
  try:
    sFooReception.unpack(incomingFoo)
  except:
    echo "unpack error for foo"

  var incomingBar: FooBarMsg
  var sBarReception = MsgStream.init(sBar.data)
  try:
    sBarReception.unpack(incomingBar)
  except:
    echo "unpack error for bar"

Hello and thanks for developing msgpack4nim
While compiling code that packs an object containing an enum I got:

.nimble/pkgs/msgpack4nim-0.1.6/msgpack.nim(1006, 14) Error: attempting to call undeclared routine: 'unpack_enum_proxy'

line 1006 reads as:
n = s.unpack_enum_proxy(n)

Sounds related to #4

The current method for always falling back to the undistinct base type does not allow implementing custom packing for distinct types. For example, something like this:

type  
  Guid = distinct string
      
proc pack_type*[ByteStream](s: ByteStream, v: Guid) =
  s.pack_bin(len(v))
  s.write(v)
      
var b: Guid = Guid("AA")

var s = b.pack
echo s.tohex
echo s.stringify

I'm pretty new to Nim, and not proficient enough in macro-land, but would it be possible to check at compile time if there is a matching pack function for a type to use, and only fall back to the base type if not?

Thanks

It would be really nice to have. See the description on how to do it at https://github.com/msgpack/website/blob/master/README.md#how-to-list-up-your-project-on-msgpackorg

Related to #65. Same issue with

Moving var x: T inside for loop will fix the problem.

My unittest

type
  Node = ref object
    nName: string
    nodes: seq[Node]

let main = Node(nName: "main")
main.nodes.add(Node(nName: "one"))
main.nodes.add(Node(nName: "two"))
main.nodes.add(Node(nName: "three"))
main.nodes.add(Node(nName: "four"))

var refNode: Node
var s = MsgStream.init()
s.pack(main)
unpack(s.data, refNode)

check refNode.nName == "main"
var names: seq[string]
for sub in refNode.nodes:
  check sub.nName notin names
  names.add(sub.nName) # one, thwo, three, four

Hey,

I'm using your awesome package to pack/unpack big AST (ref nodes) in Bro package, a wip stylesheet language (alternative to SASSC / DartSass, SassJS.)

Unpacking 114mb with v0.3.1:
0.734594596

Unpacking the same file with #head version:
1.201364966

I'm using cpuTime() to get the numbers

When I try to unpack an object type that contains int64 types I get:

C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(1201) unpack
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(938) unpack_type
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(1201) unpack
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(1085) unpack_type
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(1201) unpack
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(1085) unpack_type
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(1201) unpack
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(845) unpack_type
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(842) unpack_int_imp_select
C:\Users\user\.nimble\pkgs\msgpack4nim-0.3.1\msgpack4nim.nim(511) unpack_imp_int64
Error: unhandled exception: int64 [ObjectConversionDefect]

The object I'm packing/unpacking is a seq of an object type. This is to and from a FileStream.

I filed this issue with nimble: nim-lang/nimble#1188

I don't really know what the fix is. The trouble is that this repo's .gitattributes file causes a different hash to be computed on Windows vs Linux/macOS, but I can understand why you want that file in this repo. Do you have a suggestion?

nim-serialization looks promising. can we implement msgpack backend for it?

import msgpack4nim, streams

var xs: array[160, char]
newStringStream().pack(xs)

Error: ambiguous call; both msgpack4nim.pack_type(s: Stream, val: openarray[T])[declared in C:\Users\kamil.ni
mble\pkgs\msgpack4nim-0.1.7\msgpack4nim.nim(631, 5)] and msgpack4nim.pack_type(s: Stream, val: cstring)[declared in C:\Users\kamil.nimble\pkgs\msgpack4nim-0.1.7\msgpack4nim.nim(1097
, 5)] match for: (Stream, array[0..159, char])

I have a problem. There is a bug in Nim's memory allocation on devel, so I need to use 0.18.0 until this is solved. I think I had to switch to devel because of something in your new code. We need to revert msgpack4nim to a version that works with a released version of Nim, namely 0.18.0. What SHA1 do you think would work?

• https://forum.nim-lang.org/t/4088

Warnings (that will eventually turn into errors) are raised when unpacking to an object that has properties which are ranges.

Here is my object and the type definition for its property:

type
    StreamId* = uint32 ## A stream ID
    NonBlankStreamId* = range[((StreamId) 1)..high(StreamId)] ## A non-blank stream ID
    CSendStreamData* = object
        ## Sent to convert the connection into a stream data input pipe.
        
        streamId*: NonBlankStreamId ## The stream to send data to

Unpacking it with the following code:

body.unpack(CSendStreamData)

The compiler generates the following warning:

<home>/.nimble/pkgs/msgpack4nim-0.3.1/msgpack4nim.nim(1217, 9) Warning: Cannot prove that 'result' is initialized. This will become a compile time error in the future. [ProveInit]

While this doesn't stop the code from compiling (for now), it does show as an error in the VS Code extension and is annoying in that way. The main issue here is the implication that this code may not compile in later versions of Nim. I believe that if the data being unpacked violates the range, that it should raise a ObjectConversionDefect, since that would be consistent.

My nim -v output:

Nim Compiler Version 1.6.2 [Linux: amd64]
Compiled at 2021-12-17
Copyright (c) 2006-2021 by Andreas Rumpf

git hash: 9084d9bc02bcd983b81a4c76a05f27b9ce2707dd
active boot switches: -d:release

Thanks! Other than this, your project has been working amazingly for me.

Nim Compiler Version 0.16.0

Installed via nimble without problems

Warning: File inside package 'msgpack4nim' is outside of permitted namespace, should be named 'msgpack4nim.nim' but was named 'restriction_example.nim' instead. This will be an error in the future.

Hint: Rename this file to 'msgpack4nim.nim', move it into a 'msgpack4nim/' subdirectory, or prevent its installation by adding skipFiles = @["restriction_example.nim"] to the .nimble file. See https://github.com/nim-lang/nimble#libraries for more info.

nimble build on a project that depends on this one gave me these warnings.

Versions:

• nimble v0.8.2 compiled at 2017-01-09 23:41:29
• Nim Compiler Version 0.16.0 (2017-01-09) [Linux: amd64]