iryont / lua-struct Goto Github PK

Again, i found myself in need of this feature from the original/C struct, so added it to my copy by a small modification:

--function struct.unpack(format, stream)
--  local vars = {}
--  local iterator = 1
function struct.unpack(format, stream, start_pos)
  local vars = {}
  local iterator = start_pos or 1

(Btw, i find your code clean and very easy to work with)

If there exists a way to specify the byte order ?

I ran into this unpleasantry:

>>> struct.pack('b', -1)
"\127"  -- should be '\255'
>>> struct.pack('b', -7)
"y"    -- should be '\249'

Fix - comment out these 3 lines:

      if val < 0 then
        val = val + 2 ^ (n * 8 - 1)
      end

struct.pack('l', -2)
"\0\0\0\0\0\0\0\0"    -- should be "\254\255\255\255\255\255\255\255"

I guess i was wrong that leaving this was okay:

      if val < 0 then
        val = val + 2 ^ (n * 8)
      end

Comment it out and conversion works

Minor improvement, so 'c' format can be handled for single characters w/o n appended (i.e. 'c', not only 'c1'):

      local n = format:sub(i + 1):match('^%d+') or ''
      local length = tonumber(n) or 1

It's a cheap fix. Also because of no '^' in pattern before, it would jump ahead to find a number and then i = i + n:len() would mess up progress in the pattern.

The code i am trying to port and uses struct needs this feature of Ierusalimschy's struct.unpack:

... After the read values, this function also returns the index in s where it stopped reading, which is also where you should start to read the rest of the string.

So i added it to my copy of lua-struct, as easy as adding a line at the end:

  table.insert(vars, iterator)   --<-- one line

  return unpack(vars)

I find myself in need of a non-zero terminated string with variable length to be packed (it's part of the protocol i am dealing with), so i made a change to my copy to match the behavior of Ierusalimschy struct for 'c0' - here is my mod:

      --if length > 0 then
      --  local str = tostring(table.remove(vars, 1))
      --  if length - str:len() > 0 then
      --    str = str .. string.rep(' ', length - str:len())
      --  end
      --  table.insert(stream, str:sub(1, length))
      --end
      local str = tostring(table.remove(vars, 1))
      if length == 0 then
        length = str:len() -- 'c0' meaning use the given string length
      elseif length > str:len() then
        str = str .. string.rep(' ', length - str:len())
      end
      table.insert(stream, str:sub(1, length))

Submitting in case you find useful. I have not looked into adding 'c0' for unpack, since i haven't had need for it (yet?).

I ran into this unpleasantry:

>>> struct.unpack('b', struct.pack('b', 10))
-118

Ironically, -1 from the example is the only negative number for which 'b' worked correct.
Fix:

      if signed and val >= 2 ^ (8*n - 1) then
        val = val - 256^n
      end

Like subject says, unpacking fixed-size string returns one too many chars. If you ask for N, you'll get N+1. Note the iterator was not (d)effected, i.e. is advanced by the correct length N and following fields are not messed up.

It's a trivial bug, the fix is in to subtract 1 in this one line
https://github.com/iryont/lua-struct/blob/master/struct.lua#L188

188-      table.insert(vars, stream:sub(iterator, iterator + tonumber(n)))
188+      table.insert(vars, stream:sub(iterator, iterator + tonumber(n)-1))

I faced the need to unpack zero-terminated strings where each character is 2 bytes in big-endian (hi-lo) order, even as most all strings are ASCII range. So what i added to unpack is:

    elseif opt == 'S' then   -- wide-character string, hi-lo

      local str = ''
      while true do
        local wch = stream:byte(iterator) + 256 * stream:byte(iterator + 1)
        iterator = iterator + 2
        if wch == 0 then
          break
        end        
        str = str .. (wch < 128 and string.char(wch) or '~')
      end
      table.insert(vars, str)      

    elseif

This is the most controversial/unfinished of my mods, since it assumes little-endian encoding (many apps do lo-hi, even as the default per RFC-2781 is big endian - see https://en.wikipedia.org/wiki/UTF-16#Byte_order_encoding_schemes ). In addition i don't check for https://en.wikipedia.org/wiki/Byte_order_mark .

Nor do i handle correctly code points over 255. Which is a puzzle, how to correctly handle that in Lua? I am guessing the right thing would be to convert to UTF-8 for the internal string (which matches ASCII for <128). In any case - not production ready but existing need.

We noticed an issue when we were trying to install this module earlier today for one of our builds.

You can pretty simply reproduce it by just trying to install the luarock:

$ luarocks install lua-struct 0.9.0-1
Warning: falling back to wget - install luasec to get native HTTPS support
Installing https://luarocks.org/lua-struct-0.9.0-1.rockspec
Cloning into 'lua-struct'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 0), reused 4 (delta 0), pack-reused 0
Receiving objects: 100% (10/10), 4.29 KiB | 1.07 MiB/s, done.


Error: Failed copying struct.lua

Digging into this, it looks like y'all have been reorganizing the code for better maintenance, which is great! Unfortunately, though, if you look at lua-struct on luarocks.org, it:

• Only has version 0.9.0-1
• And that luarocks rockspec just points to the HEAD of this repo, looking for struct.lua on the root level.

Would it be possible to either:

• Publish a new version of 0.9.0-1 with the latest directory structure? (I'm not sure if LuaRocks will let you republish such an old version, though. It's a shame that old version doesn't have a tag associated with it lol.)
• Or publish the latest version of 0.9.2-1?

thank you for making the library available.

I would like to draw your attention to a few things that I can help others

In that LUA library using 2B 4B or 8B is not possible, arguments must be BB BBBB or BBBBBBBB.

When i was trying to convert 4 byte to float it is necessary to pass it to function one by one.

roll={140, 43, 88, 58}
b=struct.pack('BBBB',roll[1],roll[2],roll[3],roll[4])