iryont / lua-struct Goto Github PK
View Code? Open in Web Editor NEWImplementation of binary packing/unpacking in pure lua
Implementation of binary packing/unpacking in pure lua
Again, i found myself in need of this feature from the original/C struct, so added it to my copy by a small modification:
--function struct.unpack(format, stream)
-- local vars = {}
-- local iterator = 1
function struct.unpack(format, stream, start_pos)
local vars = {}
local iterator = start_pos or 1
(Btw, i find your code clean and very easy to work with)
If there exists a way to specify the byte order ?
I ran into this unpleasantry:
>>> struct.pack('b', -1)
"\127" -- should be '\255'
>>> struct.pack('b', -7)
"y" -- should be '\249'
Fix - comment out these 3 lines:
if val < 0 then
val = val + 2 ^ (n * 8 - 1)
end
struct.pack('l', -2)
"\0\0\0\0\0\0\0\0" -- should be "\254\255\255\255\255\255\255\255"
I guess i was wrong that leaving this was okay:
if val < 0 then
val = val + 2 ^ (n * 8)
end
Comment it out and conversion works
Minor improvement, so 'c' format can be handled for single characters w/o n
appended (i.e. 'c', not only 'c1'):
local n = format:sub(i + 1):match('^%d+') or ''
local length = tonumber(n) or 1
It's a cheap fix. Also because of no '^' in pattern before, it would jump ahead to find a number and then i = i + n:len()
would mess up progress in the pattern.
The code i am trying to port and uses struct
needs this feature of Ierusalimschy's struct.unpack
:
... After the read values, this function also returns the index in s where it stopped reading, which is also where you should start to read the rest of the string.
So i added it to my copy of lua-struct, as easy as adding a line at the end:
table.insert(vars, iterator) --<-- one line
return unpack(vars)
I find myself in need of a non-zero terminated string with variable length to be packed (it's part of the protocol i am dealing with), so i made a change to my copy to match the behavior of Ierusalimschy struct for 'c0' - here is my mod:
--if length > 0 then
-- local str = tostring(table.remove(vars, 1))
-- if length - str:len() > 0 then
-- str = str .. string.rep(' ', length - str:len())
-- end
-- table.insert(stream, str:sub(1, length))
--end
local str = tostring(table.remove(vars, 1))
if length == 0 then
length = str:len() -- 'c0' meaning use the given string length
elseif length > str:len() then
str = str .. string.rep(' ', length - str:len())
end
table.insert(stream, str:sub(1, length))
Submitting in case you find useful. I have not looked into adding 'c0' for unpack
, since i haven't had need for it (yet?).
I ran into this unpleasantry:
>>> struct.unpack('b', struct.pack('b', 10))
-118
Ironically, -1 from the example is the only negative number for which 'b' worked correct.
Fix:
if signed and val >= 2 ^ (8*n - 1) then
val = val - 256^n
end
Like subject says, unpacking fixed-size string returns one too many chars. If you ask for N, you'll get N+1. Note the iterator was not (d)effected, i.e. is advanced by the correct length N and following fields are not messed up.
It's a trivial bug, the fix is in to subtract 1 in this one line
https://github.com/iryont/lua-struct/blob/master/struct.lua#L188
188- table.insert(vars, stream:sub(iterator, iterator + tonumber(n)))
188+ table.insert(vars, stream:sub(iterator, iterator + tonumber(n)-1))
I faced the need to unpack zero-terminated strings where each character is 2 bytes in big-endian (hi-lo) order, even as most all strings are ASCII range. So what i added to unpack is:
elseif opt == 'S' then -- wide-character string, hi-lo
local str = ''
while true do
local wch = stream:byte(iterator) + 256 * stream:byte(iterator + 1)
iterator = iterator + 2
if wch == 0 then
break
end
str = str .. (wch < 128 and string.char(wch) or '~')
end
table.insert(vars, str)
elseif
This is the most controversial/unfinished of my mods, since it assumes little-endian encoding (many apps do lo-hi, even as the default per RFC-2781 is big endian - see https://en.wikipedia.org/wiki/UTF-16#Byte_order_encoding_schemes ). In addition i don't check for https://en.wikipedia.org/wiki/Byte_order_mark .
Nor do i handle correctly code points over 255. Which is a puzzle, how to correctly handle that in Lua? I am guessing the right thing would be to convert to UTF-8 for the internal string (which matches ASCII for <128). In any case - not production ready but existing need.
We noticed an issue when we were trying to install this module earlier today for one of our builds.
You can pretty simply reproduce it by just trying to install the luarock:
$ luarocks install lua-struct 0.9.0-1
Warning: falling back to wget - install luasec to get native HTTPS support
Installing https://luarocks.org/lua-struct-0.9.0-1.rockspec
Cloning into 'lua-struct'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 0), reused 4 (delta 0), pack-reused 0
Receiving objects: 100% (10/10), 4.29 KiB | 1.07 MiB/s, done.
Error: Failed copying struct.lua
Digging into this, it looks like y'all have been reorganizing the code for better maintenance, which is great! Unfortunately, though, if you look at lua-struct on luarocks.org, it:
HEAD
of this repo, looking for struct.lua
on the root level.Would it be possible to either:
0.9.0-1
with the latest directory structure? (I'm not sure if LuaRocks will let you republish such an old version, though. It's a shame that old version doesn't have a tag associated with it lol.)0.9.2-1
?thank you for making the library available.
I would like to draw your attention to a few things that I can help others
In that LUA library using 2B
4B
or 8B
is not possible, arguments must be BB
BBBB
or BBBBBBBB
.
When i was trying to convert 4 byte to float it is necessary to pass it to function one by one.
roll={140, 43, 88, 58}
b=struct.pack('BBBB',roll[1],roll[2],roll[3],roll[4])
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.