mostawesomedude / construct Goto Github PK

View Code? Open in Web Editor NEW

152.0 152.0 173.0 1.21 MB

Python binary parsing library

License: MIT License

Shell 1.44% Python 98.56%

construct's People

Contributors

Stargazers

Watchers

Forkers

adamatlas pombredanne pokeylope crass jesseward aarrtteemm construct sdiehl parsing scraping-xx swoertz dyno bitcraft stormbay sydp grunnpi nachtwaffen codassassin idaandidapython

construct's Issues

[Question] Is it active now?

May be it is Python's bindata which is a Ruby gem.

pretty printing long Collections

It would be nice if collections included a way to print them that truncated long sequences. For now, I'm using this:

def trunc(text):
ret = []
for l in str(text).splitlines():
if len(l) > 100:
l = l[:40] + ' ... ' + l[-40:]
ret.append(l)

return "\n".join(ret)

TextualAdapter builds incorrect 0 and negative numbers

the next tests fail:

>>> from construct import *

from construct.text import *
TextualIntAdapter(Field("textintadapter", 3)).build(-12)
'12-'
TextualIntAdapter(Field("textintadapter", 1)).build(0)
Traceback (most recent call last):
File "", line 1, in
File "construct/core.py", line 206, in build
self.build_stream(obj, stream)
File "construct/core.py", line 214, in build_stream
self._build(obj, stream, Container())
File "construct/core.py", line 284, in _build
self.subcon._build(self._encode(obj, context), stream, context)
File "construct/core.py", line 324, in _build
_write_stream(stream, self.length, obj)
File "construct/core.py", line 306, in _write_stream
raise FieldError("expected %d, found %d" % (length, len(data)))
construct.core.FieldError: expected 1, found 0

JPEG parser

This parses EXIF and JFIF files. this is my first construct, and while making it I noticed I was missing a couple things:

a PascalString that includes the size bytes in the length - this is fairly common in protocols. That is what the "- 2" below is for.
Embed() on a Switch() doesn't work very well - because it discards the non-struct substructs.
FastReader() below improves speed by about 10X.

class FastReader(Construct):
def _parse(self, stream, context):
return stream.read()

def _build(self, obj, stream, context):
    stream.write(obj)

SegBody = Struct(None,
UBInt16('size'),
Field('data', lambda ctx: ctx['size'] - 2),
)

Seg = Struct('seg',
Literal('\xff'),
Byte('kind'),
Switch('body', lambda c: c['kind'],
{
SOS: FastReader('data'),
},
default = Embed(SegBody),
)
)

JPEG = Struct('jpeg',
Literal('\xff\xd8'),
GreedyRange(Seg),
)

Password ? Install faild

pi@raspberrypi:~/gr-satellites/build/construct $ make upload
./setup.py sdist upload
running sdist
running egg_info
writing requirements to construct.egg-info/requires.txt
writing construct.egg-info/PKG-INFO
writing top-level names to construct.egg-info/top_level.txt
writing dependency_links to construct.egg-info/dependency_links.txt
reading manifest file 'construct.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'construct.egg-info/SOURCES.txt'
running check
creating construct-2.9.45
creating construct-2.9.45/construct
creating construct-2.9.45/construct.egg-info
creating construct-2.9.45/construct/lib
copying files to construct-2.9.45...
copying LICENSE -> construct-2.9.45
copying MANIFEST.in -> construct-2.9.45
copying README.rst -> construct-2.9.45
copying setup.py -> construct-2.9.45
copying construct/__init__.py -> construct-2.9.45/construct
copying construct/core.py -> construct-2.9.45/construct
copying construct/debug.py -> construct-2.9.45/construct
copying construct/expr.py -> construct-2.9.45/construct
copying construct/version.py -> construct-2.9.45/construct
copying construct.egg-info/PKG-INFO -> construct-2.9.45/construct.egg-info
copying construct.egg-info/SOURCES.txt -> construct-2.9.45/construct.egg-info
copying construct.egg-info/dependency_links.txt -> construct-2.9.45/construct.egg-info
copying construct.egg-info/requires.txt -> construct-2.9.45/construct.egg-info
copying construct.egg-info/top_level.txt -> construct-2.9.45/construct.egg-info
copying construct/lib/__init__.py -> construct-2.9.45/construct/lib
copying construct/lib/binary.py -> construct-2.9.45/construct/lib
copying construct/lib/bitstream.py -> construct-2.9.45/construct/lib
copying construct/lib/containers.py -> construct-2.9.45/construct/lib
copying construct/lib/hex.py -> construct-2.9.45/construct/lib
copying construct/lib/py3compat.py -> construct-2.9.45/construct/lib
Writing construct-2.9.45/setup.cfg
Creating tar archive
removing 'construct-2.9.45' (and everything under it)
running upload
Password:

LICENSE

Please separate the license into a LICENSE file. This makes for better packaging.

I want a document

Hello, engineer,I want to see the historical version of the document, where can I get it?

Lazy(Bytes(this.length)) always has zero length inside GreedyRange

I am dealing with large Length-Value packages, so I memory-mapped the file I am reading and trying to use lazy evaluation to skip the value field, till I actually need to read it from file.

However, the parsing fails as soon as I start to use Lazy(Bytes(this.length)) instead of LazyArray(this.length)).

I created a little example to highlight the issue:

import construct as c

data = b"\x02ab\x03abc\x04abcd" 

success = c.GreedyRange(c.Struct("len" / c.Byte, 
                                 "val" / c.LazyArray(c.this.len, c.Byte))).parse(data)
print(success)
print()

failed = c.GreedyRange(c.Struct("len" / c.Byte,
                                "val" / c.Lazy(c.Bytes(c.this.len)))).parse(data)
print(failed)

This is the resulting output:

ListContainer: 
    Container: 
        len = 2
        val = <LazyListContainer: 0 of 2 items cached>
    Container: 
        len = 3
        val = <LazyListContainer: 0 of 3 items cached>
    Container: 
        len = 4
        val = <LazyListContainer: 0 of 4 items cached>

ListContainer: 
    Container: 
        len = 2
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd3c3f28>
    Container: 
        len = 97
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd3c39d8>
    Container: 
        len = 98
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd3c3d08>
    Container: 
        len = 3
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd316510>
    Container: 
        len = 97
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd316598>
    Container: 
        len = 98
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd316620>
    Container: 
        len = 99
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd3166a8>
    Container: 
        len = 4
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd316730>
    Container: 
        len = 97
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd3167b8>
    Container: 
        len = 98
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd316840>
    Container: 
        len = 99
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd3168c8>
    Container: 
        len = 100
        val = <function Lazy._parse.<locals>.execute at 0x7fdecd316950>

What am I doing wrong or might this be a bug?

CString.sizeof(context) throws "can't calculate size" SizeofError

This seems to be an issue with CString being an adapted RepeatUntil, which always throws an error for sizeof calls.

>>> from construct import *
>>> str = CString("a_string")
>>> str_parsed = str.parse("foo\x00")
>>> print str_parsed
foo
>>> str.sizeof()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/construct-2.06-py2.7.egg/construct/core.py", line 242, in sizeof
    raise SizeofError(e)
construct.core.SizeofError: can't calculate size
>>> str.sizeof(str_parsed)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/construct-2.06-py2.7.egg/construct/core.py", line 242, in sizeof
    raise SizeofError(e)
construct.core.SizeofError: can't calculate size

Install?

Hi sorry about this, the 'setup.py' is supposed to install construct isn't it?
I ran it, CMD pops up then closes immediately.
Any help? :(

After a quick check, it appears that construct works to some degree after running the 2to3 tool... Cstring and Repeater objects can be created, can parse, and can build. However, the Bytes object cannot parse - it wants a StringIO, not a bytesIO, yet it can't accept a str object.

Anyways, it would be nice to have a python3 version that worked (and was on PyPi)!

Discussion/Mailing List

Hi,

I've tried to post a message to http://construct.wikispaces.com/message/list/home but I need to join the wiki and it's invite-only.
Wouldn't be easier to move everything to github or use google groups?

MetaFiled requires length attribute to be present on build

This is more like a question rather an issue. I'd expect MEtaField's related length fiekd to be autopopulated on building a structure, however this does not happen:

>>> from construct import *
>>> foo = Struct("foo",
... Byte("length"),
... MetaField("data", lambda ctx: ctx["length"])
... )

>>> foo.build(Container(data="test"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/pymodules/python2.6/construct/core.py", line 206, in build
    self.build_stream(obj, stream)
  File "/usr/lib/pymodules/python2.6/construct/core.py", line 214, in build_stream
    self._build(obj, stream, Container())
  File "/usr/lib/pymodules/python2.6/construct/core.py", line 662, in _build
    subobj = getattr(obj, sc.name)
AttributeError: 'Container' object has no attribute 'length'

Could you explain correct behavior?

Cant pass byte param to Padding, gets converted to int

I am trying to pass a param which contains a padding byte b"\x55", but somewhere along the process it is getting converted to an int and then I get the following:

construct.core.PaddingError: pattern expected to be bytes of length 1

Anchor requires key presence in container

When building a Struct which contains an Anchor, not providing the anchor's label as a key in the Container instance yields an AttributeError.

#!/usr/bin/python2
from construct import *

struct = Struct(None, Anchor('foo_anchor'))
struct.build(Container())

Running the testcase yields: AttributeError: 'Container' object has no attribute 'foo_anchor'

As an anchor's value is solely determined by it's position in the construct, it shouldn't have to be specified in the Container?

Printing a container causes a ValueError

There seems to be a missing .iteritems() in the pretty_str function which causes the printing of containers to fail. Changing the line to "for k, v in self.iteritems()" fixes it for me. Traceback is included below. Running on Python 2.7.2 on Windows 7.

Traceback (most recent call last):
  File "replay.py", line 25, in <module>
    print y
  File "C:\Python27\lib\site-packages\construct-2.05-py2.7.egg\construct\lib\container.py", line 86, in __str__
    return self.__pretty_str__()
  File "C:\Python27\lib\site-packages\construct-2.05-py2.7.egg\construct\lib\container.py", line 14, in wrapper
    return func(self, *args, **kw)
  File "C:\Python27\lib\site-packages\construct-2.05-py2.7.egg\construct\lib\container.py", line 92, in __pretty_str__
    for k, v in self:
 ValueError: too many values to unpack

Differential Encoding

Hi,

I'm trying to build a Construct for a specific protocol and am a bit stuck. I wonder whether you guys can give me a tip.
Our message contains a list of records. The first entry is uncompressed, the following entries are compressed. That means that each field of a compressed entry depends on the value of the previous entry.

Message = Struct("Message",
                       Embed( Uncompressed ),
                       OptionalGreedyRange( Compressed )
                      )

At the moment I've parsing the raw bits of a Compressed entry as am not sure what I should do to refer to the previous entry.
I can think of two strategies:

Implement an Adapter for each field of the Compressed message where somehow I can refer to the previous entry (it could be a global)
Use the Context. But I haven't been example other than unit tests that use Repeaters or the Context. I would avoid using meta construct if possible

Ideally my struct would return a list containing all entries. Right now, with the above Struct, I'm getting the fields of the uncompressed entry plus a list of the other entries.

Thank you!! I love Construct and have been making progress with my port to Java: https://github.com/ziglionz/construct

adaptors doc not displaying correctly on github

https://github.com/MostAwesomeDude/construct/blob/master/docs/adaptors.rst

STICKY: upstream moved

The upstream repository has been https://github.com/construct/construct for a long while. For disclosure, I am its new maintainer. You should probably move there. This repository probably was not looked at for years now.