happyleavesaoc / aoc-mgz Goto Github PK

View Code? Open in Web Editor NEW

193.0 21.0 41.0 15.59 MB

Age of Empires II recorded game parsing and summarization in Python 3.

License: MIT License

Python 99.95% Makefile 0.05%

aoc mgz python aoe2record aoe2 parser

aoc-mgz's Introduction

mgz

Age of Empires II recorded game parsing and summarization in Python 3.

Supported Versions

Age of Kings (.mgl)
The Conquerors (.mgx)
Userpatch 1.4 (.mgz)
Userpatch 1.5 (.mgz)
HD Edition >= 4.6 (.aoe2record)
Definitive Edition (.aoe2record)

Architecture

The core functionality of mgz is a parser that produces a Python data structure based on a recorded game file. It also offers abstracted representations that make it easier to use the data.

Parsers

mgz offers two parsers, fast and full. The fast parser skips data that is rarely needed, while the full parser tries to parse as much as possible. Naturally, the fast parser is faster than the full parser. The full parser can do just about everything, the fast only maybe 80-90%. The summary will automatically try the fast parser and fall back to the full parser if needed.

Abstractions

Abstractions take parser output as input and return an object with normalized data that is easier to use for most cases. There are two abstractions available, summary and model. The summary abstraction attempts to expose the maximum amount of usable data. The model abstraction is more limited but automatically performs more lookups.

Support

Version	model	summary	fast (header)	fast (body)	full (header)	full (body)
Age of Kings (`.mgl`)		✓		✓	✓
The Conquerors (`.mgx`)		✓		✓	✓
Userpatch <= 1.4 (`.mgz`)		✓		✓	✓	✓
Userpatch 1.5 (`.mgz`)	✓	✓	✓	✓	✓	✓
HD Edition >= 4.6		✓		✓	✓	✓
HD Edition 5.8	✓	✓	✓	✓	✓	✓
Definitive Edition <= 13.34 (`.aoe2record`)		✓		✓	✓	✓
Definitive Edition > 13.34, <= 26.21 (`.aoe2record`)	✓	✓	✓	✓	✓	✓
Definitive Edition > 26.21 (`.aoe2record`)	✓	✓	✓	✓	✓

Examples

Full Parser (header) + Fast Parser (body)

import os
from mgz import header, fast

with open('/path/to/file', 'rb') as data:
    eof = os.fstat(data.fileno()).st_size
    header.parse_stream(data)
    fast.meta(data)
    while data.tell() < eof:
        fast.operation(data)

Summary

from mgz.summary import Summary

with open('/path/to/file', 'rb') as data:
    s = Summary(data)
    s.get_map()
    s.get_platform()
    # ... etc

Model

from mgz.model import parse_match

with open('/path/to/file', 'rb') as data:
    match = parse_match(data)
    match.map.name
    match.file.perspective.number
    # ... etc

To JSON

import json
from mgz.model import parse_match, serialize

with open('/path/to/file', 'rb') as h:
    match = parse_match(h)
    print(json.dumps(serialize(match), indent=2))

Frequently Asked Questions

Q: Where are the end-of-game achievements/statistics?

A: In the postgame action, available only from Userpatch version.

Q: How can I tell the number of resources/kills/etc at a certain point?

A: You can't, without replaying the match in-game.

Q: How does a recorded game file work?

A: The first portion (the header) is a snapshot of the initial game state. The second portion (the body) is a list of moves made by players. The game loads the header, then applies each move to mutate the state according to the game rules.

Q: How can I install this package?

A: pip install mgz

Contribution

Pull requests & patches welcome

Resources

aoc-mgx-format: https://github.com/stefan-kolb/aoc-mgx-format
recage: https://github.com/goto-bus-stop/recage
recanalyst: http://sourceforge.net/p/recanalyst/
genie-rs: https://github.com/SiegeEngineers/genie-rs/tree/default/crates/genie-rec
bari-mgx-format: https://web.archive.org/web/20090215065209/http://members.at.infoseek.co.jp/aocai/mgx_format.html

aoc-mgz's People

Contributors

Stargazers

Watchers

Forkers

coffenbacher jineapple philipp94831 happy-forks gregstein catorda inspectorfowler abductedplatypus chesteripz heavenlychorus connorjclark diogoalmeida sighalt cjds borisgarri denniske hwmrocker devdefcon aoeinsights mleadbetter grandyb superphil0 nymanjens qwyt jchmb hiiwave jacobalundgren laksannathan vivianthayil santolucito namek billyoconnor dj0wns sobeaa jdekarske bergerjohannes vesper8 dayjaby gourav-shinde carreter dmwever

aoc-mgz's Issues

Faster Parsing

Hi, I want to parse a lot recorded matches. At the moment parsing one match takes 1-10 seconds on my machine. I think I am already using the fast parsing option.

I am only interested in game duration and which player has won so I do not need to parse all data.

Is there a way to speed up the parsing process?

I have noticed that a large portion of the processing is parsing the map tiles. Can I somehow skip this?

construct.core.MappingError - Same error consistently

I keep getting the following error with any example I attempt using. I have nothing really to offer other than pointing out the obvious that the error I'm getting is occurring when I call the methods from mgz. I've tried to get around this by installing the latest version on github, and through pypi. Both versions of this library seem to have the exact same error which is shown below after a few notes:

Notes: I've tried both having the aoe2 save game file in the same folder as my python program (in case the spaces in the directory cause any issue), and I've also used the directory of file in the actual aoe2 save folder.

Traceback (most recent call last):
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 2844, in _decode
    return self.decoding[obj]
KeyError: 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 311, in _parse
    return self._decode(self.subcon._parse(stream, context, path), context)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 2849, in _decode
    raise MappingError("no decoding mapping for %r" % (obj,))
construct.core.MappingError: no decoding mapping for 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\barke\OneDrive\Desktop\aoe2_stats_parser\parser.py", line 8, in <module>
    header.parse_stream(data)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\mgz\util.py", line 58, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\Users\barke\AppData\Local\Programs\Python\Python310\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.MappingError: no decoding mapping for 8
    (parsing) -> de -> de

Here's my code that I'm using, although if I use any of the examples - none of them work.

import os
from mgz import header, body

PATH = 'example.aoe2record'

with open(PATH, 'rb') as data:
    eof = os.fstat(data.fileno()).st_size
    header.parse_stream(data)
    body.meta.parse_stream(data)
    while data.tell() < eof:
        body.operation.parse_stream(data)

Parsing latest patch saved .aoe2record files

Hi @happyleavesaoc !

First of all, great work and thanks for developing this recorded games parses.

I'm trying to parse some recorded games from yesterday (latest patch applied), and I'm getting the following error when running the Summary(data) example in the README:

Traceback (most recent call last):
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/mgz/cli.py", line 285, in <module>
    main()
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/mgz/cli.py", line 281, in main
    loop.run_until_complete(run(get_args()))
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete
    return future.result()
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/mgz/cli.py", line 223, in run
    print_info(rec)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/mgz/cli.py", line 75, in print_info
    header = mgz.header.parse_stream(handle)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/mgz/util.py", line 55, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/Users/thepalbi/.pyenv/versions/3.8.1/lib/python3.8/site-packages/construct/core.py", line 1036, in _parse
    raise RangeError("unsane min %s and max %s" % (min, max))
construct.core.RangeError: unsane min -1 and max -1

Any clue what's the cause? Maybe I can help.

RuntimeError: invalid mgz file: expected 3 to 3, found 2

Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 369, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "C:\dev\ant_league\backend\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 59, in __call__
    return await self.app(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\fastapi\applications.py", line 208, in __call__
    await super().__call__(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
    raise exc from None
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\middleware\sessions.py", line 75, in __call__
    await self.app(scope, receive, send_wrapper)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\exceptions.py", line 82, in __call__
    raise exc from None
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\routing.py", line 52, in app
    response = await func(request)
  File "C:\dev\ant_league\backend\lib\site-packages\fastapi\routing.py", line 219, in app
    raw_response = await run_endpoint_function(
  File "C:\dev\ant_league\backend\lib\site-packages\fastapi\routing.py", line 154, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "C:\dev\ant_league\backend\lib\site-packages\starlette\concurrency.py", line 40, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "C:\Users\Namek\AppData\Local\Programs\Python\Python38\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\dev\ant_league\backend\src\website.py", line 127, in post_match
    match_info = utils.get_match_info(file.file)
  File "C:\dev\ant_league\backend\src\utils.py", line 44, in get_match_info
    s = Summary(data)
  File "c:\dev\ant_league\backend\mgz\mgz\summary\__init__.py", line 85, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: expected 3 to 3, found 2
    (parsing) -> initial

MP_Replay_v101.101.53347.0_2021.09.16_214614_1.zip

Question

Hi!
I found your program yesterday when I was looking around the net to figure out the structure of AoE2DE's replays. I immediately downloaded it and tried to figure out how this works, but I'm very new to this thing, so I would like to ask some questions:
Is there some kind of tutorial/documentation how to use this program?
Can I use it to convert the starting state of the game and the player's action into an .xml or .csv file?
Sorry, if it's not the right place to ask, and thank you for your work.

ValueError: 135 is not a valid Action

Hi, I am parsing DE matches again and this error occurs quite often:

match_id: 66710120
profile_id: 4732824

ValueError: 135 is not a valid Action

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 81, in __init__
    self._process_body()
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 103, in _process_body
    operation, payload = fast.operation(self._handle)
  File "/root/pypy-latest-linux64/site-packages/mgz/fast.py", line 287, in operation
    return op_type, action(data)
  File "/root/pypy-latest-linux64/site-packages/mgz/fast.py", line 229, in action
    action_type = Action(action_id)
  File "/root/pypy-latest-linux64/lib-python/3/enum.py", line 315, in __call__
    return cls.__new__(cls, value)
  File "/root/pypy-latest-linux64/lib-python/3/enum.py", line 569, in __new__
    raise exc
  File "/root/pypy-latest-linux64/lib-python/3/enum.py", line 553, in __new__
    result = cls._missing_(value)
  File "/root/pypy-latest-linux64/lib-python/3/enum.py", line 582, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 135 is not a valid Action

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/process.py", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 84, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))

Current Patch DE Game Loading Error

Im working on a data analytics site for a league. Im trying to analyze games with your library but can't seem to get games to load from the newest patch. Your info says that the library works for version 25.02 and earlier. How do these version correspond to game version? Ex. Current version is: 101.101.54684 7486483.

The error I am getting:
`Exception has occurred: RuntimeError (note: full exception trace is shown but execution is paused at: setUpClass)
could not parse
File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 418, in parse
de = parse_de(header, version, save)
File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 296, in parse_de
lobby = de_string(data)
File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 47, in de_string
assert data.read(2) == b'\x60\x0a'

During handling of the above exception, another exception occurred:

File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 47, in de_string
assert data.read(2) == b'\x60\x0a'
File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 296, in parse_de
lobby = de_string(data)
File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 418, in parse
de = parse_de(header, version, save)
File "C:\Users\Owner\Git Repos\aoc-mgz\mgz\fast\header.py", line 426, in parse
raise RuntimeError("could not parse")
File "C:\Users\Owner\Git Repos\aoc-mgz\tests\test_fast.py", line 60, in setUpClass (Current frame)
cls.data = parse(handle)`

Let me know if what I'm doing wrong or if the current version of games is not supported.

Test_Game.zip
.0

no decoding mapping for 18480 - (parsing) -> scenario -> game_settings -> player_info

Following error occured:

match_id: 66868053
profile_id: 752012

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2844, in _decode
    return self.decoding[obj]
KeyError: 18480

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 311, in _parse
    return self._decode(self.subcon._parse(stream, context, path), context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2849, in _decode
    raise MappingError("no decoding mapping for %r" % (obj,))
construct.core.MappingError: no decoding mapping for 18480

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.MappingError: no decoding mapping for 18480
    (parsing) -> scenario -> game_settings -> player_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 9 to 9, found 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 72, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/mgz/util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 9 to 9, found 1
    (parsing) -> scenario -> game_settings

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/process.py", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 84, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: expected 9 to 9, found 1
    (parsing) -> scenario -> game_settings

Why my recordings can't be parsed?

Some recordings downloaded from Voobly are working for me but unfortunately, some are not. Including my own.

This is what happens when I try to load my recording:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 506, in _parse
    return packer.unpack(self.fmtstr, _read_stream(stream, self.sizeof()))[0]
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 69, in _read_stream
    raise FieldError("could not read enough bytes, expected %d, found %d" % (length, len(data)))
construct.core.FieldError: could not read enough bytes, expected 4, found 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 508, in _parse
    raise FieldError("packer %r error during parsing" % self.fmtstr)
construct.core.FieldError: packer '<f' error during parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.FieldError: packer '<f' error during parsing
    (parsing) -> initial -> players -> objects -> objects -> properties -> building -> combat -> ai -> ai -> orders -> order -> target_location -> z

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 4294918016 to 4294918016, found 35248

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/mnt/c/aoe2/aoc-mgz/mgz/util.py", line 196, in _parse
    subobj = self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 4294918016 to 4294918016, found 35248
    (parsing) -> initial -> players -> objects -> objects -> properties -> building -> combat -> ai -> ai -> orders

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 6 to 6, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(header.parse_stream(data))
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/mnt/c/aoe2/aoc-mgz/mgz/util.py", line 54, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/defozo/.local/lib/python3.7/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 6 to 6, found 0
    (parsing) -> initial

We are using:
Game Mod: v1.5 RC | Allied Vision

You can download the recording here:
https://www.voobly.com/files/view/60445255/nfw5tdx83ea9eqmior7wkqsus9zo87ik

Can't get field 'unit_ids' for 'move' action

Steps to reproduce:

Recorded game used (I've tried with many others and there is the same problem): https://www.voobly.com/files/view/53123676/qnc2045gpo3x3vkyyqks5h526pu725er
Code used:

import sys
from mgz import header, body
import pprint

if __name__ == "__main__":
    pp = pprint.PrettyPrinter(indent=4)
    # For each input filename
    for arg in sys.argv[1:]:
        with open(arg, 'rb') as f:
            # Remember end of file
            f.seek(0, 2)
            eof = f.tell()
            f.seek(0)
            # Parse the header
            h = header.parse_stream(f)
            # Parse the body
            while f.tell() < eof:
                # Parse a body operation
                o = body.operation.parse_stream(f)
                if o.type == 'action' and o.action.type == 'move':
                    pp.pprint(o)

Problem:
None of the 'move' action has the field 'unit_ids'

I see that this field should be present (excepted if moving the same unit as previous command):
https://github.com/happyleavesaoc/aoc-mgz/blob/master/mgz/body/actions.py#L63

I'm not fluent in python so please forgive me if I made a simple mistake

Thanks for your work!

New aoe2 de recorded files parsing ending early

Hi,

First of all want to thank for for this great work.

I was trying to parse files in aoe2 de recorded files of the past few days. The parsing is ending early with the last operation being embedded with default consuming the rest of the file (from my understanding). Maybe they added new embedded operation which is not handled right now. I would like to help in anyway.

Again thanks.

TypeError: a bytes-like object is required, not 'str'

Hello Happyleaves,

All your examples in the \examples\ directory work like a charm except for \examples\rec.py which returns this error "TypeError: a bytes-like object is required, not 'str'"

I tried to fix that function mgz\recorded_game_init_.py as follows

    def _parse_lobby_chat(self, messages, source, timestamp):
        """Parse a lobby chat message."""
        for message in messages:
            message = message.encode() #fix:  <------ encoding
            if message.message_length == 0:
                continue
            chat = ChatMessage(message.message, timestamp, self._players(), source=source)
            self._parse_chat(chat)

But.. more errors came up and basically the fix didn't resolve anything. Btw I'm using pyinstaller to convert the examples to windows compatibility.

There is another error in mgz\recorded_game\chat.py as you can observe in the image above.

I would appreciate it if you could take a look at that.
Thank you,

Warm regards
GregStein

question: Way to calculate unit/building ids during game

I am trying to use a rec to reconstruct the units and buildings built during the game but I can't figure out how aoe generates and associates new ids to units and buildings.

I've noticed that the building's ids are given when they are placed and same to units when they are queued, but I can't map out the entire rec's ids only by this because it seems that walls and other entities also are given ids. Is there a way to for me to run a summary function to gather all ids and what they are (vills, houses, etc) of a rec?

Btw, thanks for the library! If this id part works out, I plan to use it in a research project.

DE version 25.22 fails on header parse

Fast reader crashes at this line:

aoc-mgz/mgz/fast/header.py

Line 323 in 77216bc

lobby = de_string(data)

MP_Replay_v101.101.58259.0_2022.02.01_214243_2.zip

How to install and use it?

When I run sudo python3.6 setup.py install
I get:

running install
Checking .pth file support in /usr/local/lib/python3.6/dist-packages/
/usr/bin/python3.6 -E -c pass
TEST PASSED: /usr/local/lib/python3.6/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing mgz.egg-info/PKG-INFO
writing dependency_links to mgz.egg-info/dependency_links.txt
writing entry points to mgz.egg-info/entry_points.txt
writing requirements to mgz.egg-info/requires.txt
writing top-level names to mgz.egg-info/top_level.txt
reading manifest file 'mgz.egg-info/SOURCES.txt'
writing manifest file 'mgz.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying mgz.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mgz.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mgz.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mgz.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mgz.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mgz.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
  File "setup.py", line 26, in <module>
    'Programming Language :: Python :: 3',
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 67, in run
    self.do_egg_install()
  File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 109, in do_egg_install
    self.run_command('bdist_egg')
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/setuptools/command/bdist_egg.py", line 209, in run
    os.path.join(archive_root, 'EGG-INFO'), self.zip_safe()
  File "/usr/lib/python3/dist-packages/setuptools/command/bdist_egg.py", line 245, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/usr/lib/python3/dist-packages/setuptools/command/bdist_egg.py", line 355, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/usr/lib/python3/dist-packages/setuptools/command/bdist_egg.py", line 392, in scan_module
    code = marshal.load(f)
ValueError: bad marshal data (unknown type code)

What's the LICENSE of this project

Hi @happyleavesaoc , thx for sharing this amazing repo!
As I didn't find a LICENSE file, I guessed you forgot to add it. Are you planning to add a license? Thank you!

DE saved files aren't parsed correctly

I understand DE saved files are kinda unsupported right now, Just wanna offer a corpus of ~250 DE games if that would be useful. Only 10 of my saved files parse without errors. Here is an example of the type of errors I get (each error message is from a different save file):

invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: unspecified builtin map: 124
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: expected 16 to 16, found 6
    (parsing) -> scenario -> players
invalid mgz file: unspecified builtin map: 126
invalid mgz file: unspecified builtin map: 123
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 10
    (parsing) -> lobby
invalid mgz file: unspecified builtin map: 123
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: no decoding mapping for 11
    (parsing) -> lobby
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 4
    (parsing) -> scenario -> players
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: unspecified builtin map: 0
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 1032, found 718
    (parsing) -> scenario -> triggers
invalid mgz file: could not read enough bytes, expected 65535, found 29825
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31170
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 65535, found 29871
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 2515
    (parsing) -> scenario
invalid mgz file: could not read enough bytes, expected 32768, found 29204
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27996
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27167
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 2
    (parsing) -> scenario
invalid mgz file: expected 16 to 16, found 15
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 10
    (parsing) -> scenario -> players
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 59392, found 40667
    (parsing) -> scenario -> messages -> victory
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 31621
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31020
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 27266
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 0
    (parsing) -> scenario -> players
invalid mgz file: could not read enough bytes, expected 32768, found 31179
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 31623
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 30962
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: expected 23 to 23, found 0
    (parsing) -> de -> de -> strings
invalid mgz file: could not read enough bytes, expected 32768, found 31599
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 32768, found 27066
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: could not read enough bytes, expected 12544, found 5855
    (parsing) -> scenario
invalid mgz file: could not read enough bytes, expected 32768, found 27266
    (parsing) -> scenario -> messages -> instructions
invalid mgz file: expected 16 to 16, found 1
    (parsing) -> scenario -> players

If it'd be useful, I could upload all of my saves.

I'll start to study this code myself and see if I can help fix things.

RangeError: expected 50 to 50, found 0

Tried the following code:

from mgz import header, body
from os import fstat

with open('C:\\Users\\rajje\\PycharmProjects\\helloworld\\replay.aoe2record', 'rb') as data:
    eof = fstat(data.fileno()).st_size
    header.parse_stream(data)
    while data.tell() < eof:
        body.operation.parse_stream(data)

Go this error:

Traceback (most recent call last):
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 1235, in _parse
    pad = _read_stream(stream, padlen)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 67, in _read_stream
    data = stream.read(length)
OverflowError: Python int too large to convert to C ssize_t

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 50 to 50, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "app.py", line 6, in <module>
    header.parse_stream(data)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\mgz\util.py", line 36, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\rajje\PycharmProjects\helloworld\venv\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 50 to 50, found 0
    (parsing) -> map_info

could not read enough bytes - (parsing) -> scenario -> messages -> instructions

Following error occured:

match_id: 66708725
profile_id: 4849383

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 399, in _parse
    return _read_stream(stream, length)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 69, in _read_stream
    raise FieldError("could not read enough bytes, expected %d, found %d" % (length, len(data)))
construct.core.FieldError: could not read enough bytes, expected 25956, found 23848

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 72, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/mgz/util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.FieldError: could not read enough bytes, expected 25956, found 23848
    (parsing) -> scenario -> messages -> instructions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/process.py", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 84, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: could not read enough bytes, expected 25956, found 23848
    (parsing) -> scenario -> messages -> instructions

No postgame detected for any DE matches?

I have tested this with many 2x2 online DE matches, and get_postgame always returns None, even though matches completed (i.e., get_completed returns True) irrespective of whether I resigned, enemies resigned, I was party leader, or someone else was.
May I be missing something?

My intention is to be able to list how much of each resource each player got by the end of the match, age advancement times, etc., and as I see it that kind of info is offered by get_players, which depends on get_postgame not being None.

Let me know if I can provide any further info or offer any specific help!

Error: unpack requires a buffer of 8 bytes

Having an issue with the byte unpacking here:

with open(game_file_path, 'rb') as data:
    s = Summary(data)
    s.get_map()
    s.get_platform()
    match = parse_match(data)

Any thoughts? Wondering if its a version change but I tried recs from Feb. 2 and still got this issue.

Stack trace:

error                                     Traceback (most recent call last)
c:\python39\lib\site-packages\mgz\fast\header.py in parse(data)
    454     try:
--> 455         header = decompress(data)
    456         version, game, save, log = parse_version(header, data)

c:\python39\lib\site-packages\mgz\fast\header.py in decompress(data)
    409     prefix_size = 8
--> 410     header_len, _ = unpack('<II', data)
    411     zlib_header = data.read(header_len - prefix_size)

c:\python39\lib\site-packages\mgz\fast\header.py in unpack(fmt, data)
     32     """Unpack bytes according to format string."""
---> 33     output = struct.unpack(fmt, data.read(struct.calcsize(fmt)))
     34     if len(output) == 1:

error: unpack requires a buffer of 8 bytes

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_4256/3714053892.py in <module>
      6     s.get_map()
      7     s.get_platform()
----> 8     match = parse_match(data)

c:\python39\lib\site-packages\mgz\model\__init__.py in parse_match(handle)
     91     """
     92 
---> 93     data = parse(handle)
     94     body_pos = handle.tell() - 4 # log version
     95     consts = get_consts()

c:\python39\lib\site-packages\mgz\fast\header.py in parse(data)
    465         lobby = parse_lobby(header, version, save)
    466     except (struct.error, zlib.error, AssertionError, MemoryError):
--> 467         raise RuntimeError("could not parse")
    468     return dict(
    469         version=version,

RuntimeError: could not parse

ValueError: could not detect encoding

The parse_match method:

File "c:\dev\project\backend\mgz\mgz\model\__init__.py", line 31, in parse_match
  map_data, encoding, language = get_map_data(
File "c:\dev\project\backend\mgz\mgz\summary\map.py", line 171, in get_map_data
  encoding, language, name = extract_from_instructions(instructions)
File "c:\dev\project\backend\mgz\mgz\summary\map.py", line 91, in extract_from_instructions
  raise ValueError('could not detect encoding')

replay.zip

module not working with pip

program works normally when I change the files directly from github. If I try to install them using pip, programs won't work. it's not a problem doing it locally, but when i want to use it on the server, there is no possibility for that.

when will mgz be available to install by pip

achievements parsing for DE returning null results

Hello Happyleaves,

get_players() for DE summary seem to return None for most entries:

[{'name': 'Fur_27', 'civilization': 6, 'human': True, 'number': 1, 'color_id': 2, 'winner': False, 'mvp': None, 'score': None, 'position': (97.0, 56.0), 'rate_snapshot': None, 'user_id': 403294, 'cheater': False, 'achievements': {'military': {'score': None, 'units_killed': None, 'hit_points_killed': None, 'units_lost': None, 'buildings_razed': None, 'hit_points_razed': None, 'buildings_lost': None, 'units_converted': None}, 'economy': {'score': None, 'food_collected': None, 'wood_collected': None, 'stone_collected': None, 'gold_collected': None, 'tribute_sent': None, 'tribute_received': None, 'trade_gold': None, 'relic_gold': None}, 'technology': {'score': None, 'feudal_time': None, 'castle_time': None, 'imperial_time': None, 'explored_percent': None, 'research_count': None, 'research_percent': None}, 'society': {'score': None, 'total_wonders': None, 'total_castles': None, 'total_relics': None, 'villager_high': None}}}, {'name': '[the][big][one]', 'civilization': 6, 'human': True, 'number': 2, 'color_id': 0, 'winner': True, 'mvp': None, 'score': None, 'position': (12.0, 59.0), 'rate_snapshot': None, 'user_id': 1228905, 'cheater': False, 'achievements': {'military': {'score': None, 'units_killed': None, 'hit_points_killed': None, 'units_lost': None, 'buildings_razed': None, 'hit_points_razed': None, 'buildings_lost': None, 'units_converted': None}, 'economy': {'score': None, 'food_collected': None, 'wood_collected': None, 'stone_collected': None, 'gold_collected': None, 'tribute_sent': None, 'tribute_received': None, 'trade_gold': None, 'relic_gold': None}, 'technology': {'score': None, 'feudal_time': None, 'castle_time': None, 'imperial_time': None, 'explored_percent': None, 'research_count': None, 'research_percent': None}, 'society': {'score': None, 'total_wonders': None, 'total_castles': None, 'total_relics': None, 'villager_high': None}}}]

is it a bug or are you still working on it?

And thanks btw big fan of this library <3

WiC mod support?

Hi HappyLeaves,

Do you know if there's something about WiC mod that breaks this parser? I was trying to parse the Clan Masters recs here (for example: http://aoczone.net/viewtopic.php?f=1342&t=123008) and they're all crashing near the end with:

/root/miniconda3/envs/mgz/lib/python2.7/site-packages/construct/core.pyc in _read_stream(stream, length)
    303     data = stream.read(length)
    304     if len(data) != length:
--> 305         raise FieldError("expected %d, found %d" % (length, len(data)))
    306     return data
    307 

FieldError: expected 1512, found 4

I'm guessing it may be something to do with achievements since I can't get that action to extract no matter what I do, but I'm not sure exactly.

Exception: Unpack Requires a Buffer of 24 Bytes

Hello,
when trying to analyse a recent replay of AoE II DE (from early March) with the following simple code snippet:

from mgz.model import parse_match
with open('path/replay.aoe2record', 'rb') as data:
    match = parse_match(data)
    match.map.name
    match.file.perspective.number

I came across this exception (with stack trace for your convenience):

unpack requires a buffer of 24 bytes
Traceback (most recent call last):
  File "/path/aoc-mgz/mgz/fast/header.py", line 469, in parse
    players, mod = parse_players(header, num_players, version)
  File "/path/aoc-mgz/mgz/fast/header.py", line 434, in parse_players
    players = [parse_player(header, number, num_players) for number in range(num_players)]
  File "/path/aoc-mgz/mgz/fast/header.py", line 434, in <listcomp>
    players = [parse_player(header, number, num_players) for number in range(num_players)]
  File "/path/aoc-mgz/mgz/fast/header.py", line 120, in parse_player
    start_x, start_y, civilization_id, color_id = unpack('<xff9xb3xbx', header)
  File "/path/aoc-mgz/mgz/fast/header.py", line 34, in unpack
    output = struct.unpack(fmt, data.read(struct.calcsize(fmt)))
struct.error: unpack requires a buffer of 24 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "aoe2test.py", line 7, in <module>
    match = parse_match(data)
  File "/path/aoc-mgz/mgz/model/__init__.py", line 93, in parse_match
    data = parse(handle)
  File "/path/aoc-mgz/mgz/fast/header.py", line 475, in parse
    raise RuntimeError("could not parse")
RuntimeError: could not parse

I am guessing a recent format change has caused this. Attached you can also find an example record file.

replay.aoe2record.zip

Can the after-game statistics be extracted from AOE2DE record files?

As the title question: Can the after-game statistics be extracted from AOE2DE record files?

Because in the README it's mentioned that can only be done from userpatch record files. But just to clarify.

Is possible to extract real time information?

Hi! I'm trying to build something that allows me to get real time information like K/D, and other stats, but right now it has been kinda hard to extract these from the recording as it only records the actions performed by users. Is this possible to do with this parser?

What I can deduct is that will be impossible without simulating the entire game, but I might be wrong. I have found some information that reads directly from game memory, that is something that I want to avoid as is not an easy task to perform.

Any advice will be appreciated

Summary reports wrong starting age

Collecting Summary on a regular ranked 1v1 RM match returns 'starting_age': (2, 'castle') which is obviously wrong.

Version: (<Version.DE: 21>, 'VER 9.4', 20.06, 5)

AgeIIDE_Replay_85017375.zip

Using the latest version of aoc-mgz from pip.

SyntaxError: can't use starred expression here

With Python 3.9 this throws a syntax error. It seems to work fine with Python 3.6, however.

Traceback (most recent call last):
  File "(removed)\test.py", line 1, in <module>
    from mgz.model import parse_match
  File "(removed)\LocalCache\local-packages\Python39\site-packages\mgz\model\__init__.py", line 8, in <module>
    from mgz.fast.header import parse
  File "(removed)\LocalCache\local-packages\Python39\site-packages\mgz\fast\header.py", line 108
    type_, (*diplomacy), name_length = unpack(f'<bx{num_players}x9i5xh', header)
            ^
SyntaxError: can't use starred expression here

When I change the line to the version below it works with both Python versions.

type_, *diplomacy, name_length = unpack(f'<bx{num_players}x9i5xh', header)

construct.core.MappingError: no decoding mapping for 8

Similar error (maybe the exact same) from the full parser.
Using a very recent map from aoe2.net I get the following error:
AgeIIDE_Replay_160758221.zip

could not fast parse; falling back
Traceback (most recent call last):
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 2844, in _decode
    return self.decoding[obj]
KeyError: 8
[AgeIIDE_Replay_160758221.zip](https://github.com/happyleavesaoc/aoc-mgz/files/8748751/AgeIIDE_Replay_160758221.zip)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 311, in _parse
    return self._decode(self.subcon._parse(stream, context, path), context)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 2849, in _decode
    raise MappingError("no decoding mapping for %r" % (obj,))
construct.core.MappingError: no decoding mapping for 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/mgz/summary/full.py", line 73, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/mgz/util.py", line 58, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.MappingError: no decoding mapping for 8
    (parsing) -> de -> de

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jason/code/aoemaprender/python/generatemap.py", line 5, in <module>
    s = Summary(data)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/mgz/summary/__init__.py", line 22, in __call__
    return FullSummary(data, playback)
  File "/home/jason/code/aoemaprender/python/.venv/lib/python3.8/site-packages/mgz/summary/full.py", line 85, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: no decoding mapping for 8
    (parsing) -> de -> de

import os
from mgz.summary import Summary

with open('AgeIIDE_Replay_160758221.aoe2record', 'rb') as data:
    s = Summary(data)
    s.get_map()
    s.get_platform()

Installed from source:

pip freeze
aiohttp==3.8.1
aiosignal==1.2.0
aocref==2.0.1
async-timeout==4.0.2
attrs==21.4.0
certifi==2021.10.8
charset-normalizer==2.0.12
construct==2.8.16
flatbuffers==2.0
frozenlist==1.3.0
idna==3.3
mgz==1.7.3
multidict==6.0.2
pycountry==22.1.10
requests==2.27.1
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
tabulate==0.8.9
tqdm==4.63.0
urllib3==1.26.8
yarl==1.7.2

No "largest army" from Summary

    {
      "player_name": "EDITED",
      "total_score": 20300,
      "total_scores": [
        32981,
        20300,
        23559,
        4825,
        4281,
        5857,
        31856,
        12565
      ],
      "victory": false,
      "civilization": 2,
      "color_id": 2,
      "team": 3,
      "ally_count": 3,
      "random_civ": true,
      "mvp": false,
      "result": 0,
      "military": {
        "score": 9863,
        "units_killed": 447,
        "hit_points_killed": 49315,
        "units_lost": 452,
        "buildings_razed": 16,
        "hit_points_razed": 3433,
        "buildings_lost": 115,
        "units_converted": 0
      },
      "economy": {
        "score": 4763,
        "food_collected": 45740,
        "wood_collected": 48911,
        "stone_collected": 3451,
        "gold_collected": 34607,
        "tribute_sent": 6500,
        "tribute_received": 2700,
        "trade_gold": 21294,
        "relic_gold": 0
      },
      "technology": {
        "score": 5479,
        "feudal_time_int": 746,
        "feudal_time": "00:12:26",
        "castle_time_int": 1301,
        "castle_time": "00:21:41",
        "imperial_time_int": 2596,
        "imperial_time": "00:43:16",
        "explored_percent": 75,
        "research_count": 58,
        "research_percent": 39
      },
      "society": {
        "score": 195,
        "total_wonders": 0,
        "total_castles": 2,
        "relics_captured": 0,
        "villager_high": 92
      }

I cannot find any "largest army" information from the summary. Is this expected? How can I retrieve this information?

expected b'`\n' but parsed b'\x89\xda' - (parsing) -> de -> de -> ai_files

Following error occured:

match_id: 64686137
profile_id: 1670341

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1936, in _parse
    raise ConstError("expected %r but parsed %r" % (self.value, obj))
construct.core.ConstError: expected b'`\n' but parsed b'\x89\xda'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.ConstError: expected b'`\n' but parsed b'\x89\xda'
    (parsing) -> de -> de -> ai_files -> name

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 4205749679821946880 to 4205749679821946880, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 72, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/mgz/util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 4205749679821946880 to 4205749679821946880, found 0
    (parsing) -> de -> de -> ai_files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/process.py", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 84, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: expected 4205749679821946880 to 4205749679821946880, found 0
    (parsing) -> de -> de -> ai_files

Unable to run any of the examples provided on the main page

Hello,

For some reason, I cannot run any of the examples provided on the main page. Are there special requirements that are not listed? My python version is 3.8.7.
Here is a simple script that is used to test the five examples and log the error traceback to a file. All you need to change is EXAMPLE = 1|2|3|4|5 in the constants.

# Imports.
import mgz
import os

# Constants.
PATH = "test.aoe2record"
EXAMPLE = 5

# Redirect the error to a file.
with open(f"log_example_{EXAMPLE}.txt", "w") as log_file:

    # Open a game and get information.
    try:
        with open(PATH, "rb") as data:

            if EXAMPLE == 1:
                from mgz import header, body
                eof = os.fstat(data.fileno()).st_size
                header.parse_stream(data)
                body.meta.parse_stream(data)
                while data.tell() < eof:
                    body.operation.parse_stream(data)

            elif EXAMPLE == 2:
                from mgz import header, fast
                eof = os.fstat(data.fileno()).st_size
                header.parse_stream(data)
                fast.meta(data)
                while data.tell() < eof:
                    fast.operation(data)

            elif EXAMPLE == 3:
                from mgz.summary import Summary
                s = Summary(data)
                s.get_map()
                s.get_platform()

            elif EXAMPLE == 4:
                from mgz.model import parse_match
                match = parse_match(data)
                match.map.name
                match.file.perspective.number

            elif EXAMPLE == 5:
                import json
                from mgz.model import parse_match, serialize
                match = parse_match(data)
                print(json.dumps(serialize(match), indent=2))

    except Exception:
        import traceback
        log_file.write(traceback.format_exc())

test.zip contains test.aoe2record.

Manually looking at the header, this game has the following characteristics:

data = open(test.aoe2record, "rb")
from mgz.summary import *
header = decompress(data)
version, game, save, log = parse_version(header, data)
>>> version = <Version.DE: 21>
>>> game = 'VER 9.4'
>>> save = 25.22
>>> log = 5

The 5 tracebacks are here:
log_example_1.txt
log_example_2.txt
log_example_3.txt
log_example_4.txt
log_example_5.txt

The 5 top-most errors are:

construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

  File "C:\WPy64-3870\python-3.8.7.amd64\lib\site-packages\mgz\fast\header.py", line 48, in de_string
    assert data.read(2) == b'\x60\x0a'
AssertionError

  File "C:\WPy64-3870\python-3.8.7.amd64\lib\site-packages\mgz\fast\header.py", line 48, in de_string
    assert data.read(2) == b'\x60\x0a'
AssertionError

Thank you in advance for your help.

crash since v101.101.61591.0 - construct.core.MappingError: no decoding mapping for 8

Previous versions were parsed just fine, but it seems this new v101.101.61591.0 makes things explode:

v101.101.41855.0 ok
v101.101.47820.0 ok
v101.101.59165.0 ok
v101.101.61591.0 crashes

Traceback (most recent call last):

  File "C:\Python39\lib\site-packages\construct\core.py", line 2844, in _decode
    return self.decoding[obj]
KeyError: 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 311, in _parse
    return self._decode(self.subcon._parse(stream, context, path), context)
  File "C:\Python39\lib\site-packages\construct\core.py", line 2849, in _decode
    raise MappingError("no decoding mapping for %r" % (obj,))
construct.core.MappingError: no decoding mapping for 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\git\aoe-analysis\analyser.py", line 896, in <module>
    rp.parse(args.file, accept_cache = True)
  File "D:\git\aoe-analysis\analyser.py", line 281, in parse
    self.get_player_names_from_recording()
  File "D:\git\aoe-analysis\analyser.py", line 798, in get_player_names_from_recording
    for player in header.parse_stream(h).de.players:
  File "C:\Python39\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Python39\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\mgz\util.py", line 58, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "C:\Python39\lib\site-packages\construct\core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "C:\Python39\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Python39\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\Python39\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.MappingError: no decoding mapping for 8
    (parsing) -> de -> de
> c:\python39\lib\site-packages\construct\core.py(2788)_parse()
-> raise e.__class__("%s\n    %s" % (e, path))

some pdb debugging:

-> raise e.__class__("%s\n    %s" % (e, path))
(Pdb) l
2783                path += " -> %s" % (self.name)
2784                return self.subcon._parse(stream, context, path)
2785            except ConstructError as e:
2786                if "\n" in str(e):
2787                    raise
2788 ->             raise e.__class__("%s\n    %s" % (e, path))
2789        def _build(self, obj, stream, context, path):
2790            try:
2791                path += " -> %s" % (self.name)
2792                return self.subcon._build(obj, stream, context, path)
2793            except ConstructError as e:
(Pdb) u
> c:\python39\lib\site-packages\construct\core.py(1594)_parse()
-> obj = self.cases.get(key, self.default)._parse(stream, context, path)
(Pdb) print(stream, context, path)
<_io.BytesIO object at 0x00000200D8E46E00> Container:
    game_version = VER 9.4 (total 7)
    save_version = 26.16
    version = Version.DE (parsing) -> de

let's query the stream object to figure out where we are:

(Pdb) print(stream.tell())
64
(Pdb) print(stream.seek(0))
0
(Pdb) print(stream.read(256))
b'VER 9.4\x00\xaeG\xd1A\x97\xf0\x00\x00N\xb8~b\x00\xa0\x8aD\xe8\x03\x00\x00\x01\x00\x00\x00\t\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\n\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00M\x00\x00\x00M\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\xa3_\x02\x00\xa3_\x02\x00\xecQ\xd8?\x00\x00\x00\x00\xc8\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xa3_\x02\x00\x01\x00\x00\x00\x00\x01\x01\x01\x00\x01\x01\x01\x00\x00\x01\x00\x00\x00\x00\x1e\x00\x00\x00\x00\xa3_\x02\x00\x0b\x00\x00\x00\x03\x00\x00\x00\xff\x02\x02\x9cm\x91\xf1H\x00\x00\x00\x00(\x00\x00\x00`\n\x00\x00\x00`\n\x00\x00`\n\x05\x00NOPEN\x02\x00\x00\x00\x03\xa1\x16\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\xff\xff\xff\xffd\x00\x00\x00\x0b\x00\x00\x00\x04\x00\x00\x00\xff\x03\x03\x9c'

"NOPEN" is a redacted 5-letter username.

Interestingly enough it's not the 256 first bytes of the file, could it be possible that there are several streams contained in a .aoe2record file ?

$ hd -n 256 MP\ Replay\ v101.101.61591.0\ \@2022.05.13\ 215806\ \(7\).aoe2record
00000000  45 c9 09 00 00 00 00 00  ec bd 7f cc 75 d9 75 df  |E...........u.u.|
00000010  75 df 77 de 99 79 e7 97  fd 8e ed b1 1d 27 4e 5e  |u.w..y.......'N^|
00000020  27 55 f1 1f 36 b1 1d 3b  36 23 ab 63 bb 09 c9 1f  |'U..6..;6#.c....|
00000030  93 b4 54 51 45 a4 4a f1  84 b8 78 4a e2 d8 49 8b  |..TQE.J...xJ..I.|
00000040  a2 a8 2a 43 05 a8 8a 50  15 22 04 69 a8 2a 43 a1  |..*C...P.".i.*C.|
00000050  a4 88 56 a1 a2 28 ad 08  84 00 51 54 39 51 0a 14  |..V..(....QT9Q..|
00000060  a2 62 52 8b bf 22 05 a4  d0 3f 22 f8 27 c3 39 f7  |.bR.."...?".'.9.|
00000070  3e e7 de 73 ee 3d 67 ef  b5 f6 5e 6b ed b5 f6 f9  |>..s.=g...^k....|
00000080  7e b6 8f df 35 cf 3d 77  ff 58 df bd f7 f7 9e 73  |~...5.=w.X.....s|
00000090  ef 73 9f 3f f9 9d 7f e2  f1 3f f7 cf 7e f4 f0 b7  |.s.?.....?..~...|
000000a0  bf eb 1f 7e fa df ff bd  c3 e1 7b 7f f1 2f fc e0  |...~......{../..|
000000b0  e1 cb 3f f5 1d bf f3 c4  e1 70 ef 70 38 3c 33 1c  |..?......p.p8<3.|
000000c0  f7 87 63 f8 cf c3 83 e1  78 72 38 9e 1a 8e a7 87  |..c.....xr8.....|
000000d0  e3 e1 dd e3 cf ce 1e ff  9e bb 63 64 7c fe 78 8c  |..........cd|.x.|
000000e0  cf 1f 9f 37 f2 d7 7f e0  fe f1 f8 bf fe 85 df 7a  |...7...........z|
000000f0  65 fc ef 5f bb ab 67 62  7c 6c 7c ce e1 de bd 7b  |e.._..gb|l|....{|
00000100

could not read enough bytes - (parsing) -> scenario -> players -> ai -> file

Hi, thanks for fixing the bug with the Action 135. I found some more errors. I will create separate issues for better tracking. I f you need more sample games where the errors occur I can provide them.

match_id: 66711026
profile_id: 316055

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 311, in _parse
    return self._decode(self.subcon._parse(stream, context, path), context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2293, in _parse
    stream2 = BytesIO(_read_stream(stream, length))
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 69, in _read_stream
    raise FieldError("could not read enough bytes, expected %d, found %d" % (length, len(data)))
construct.core.FieldError: could not read enough bytes, expected 4294967295, found 624234

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.FieldError: could not read enough bytes, expected 4294967295, found 624234
    (parsing) -> scenario -> players -> ai -> file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 16 to 16, found 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 72, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/mgz/util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 16 to 16, found 2
    (parsing) -> scenario -> players

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/process.py", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 84, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: expected 16 to 16, found 2
    (parsing) -> scenario -> players

RuntimeError: invalid mgz file: unsane min -1 and max -1

When parsing DE match 41599594 the following error occurs. Note that the match is only 25 seconds long, so I am not sure whether this really needs to be fixed. Maybe the recording is just broken?

Traceback (most recent call last):
File \"/opt/pypy/site-packages/mgz/summary/__init__.py\", line 72, in __init__
    self._header = mgz.header.parse_stream(self._handle)
File \"/opt/pypy/site-packages/construct/core.py\", line 171, in parse_stream
    return self._parse(stream, context2, \"(parsing)\")
File \"/opt/pypy/site-packages/construct/core.py\", line 867, in _parse
    subobj = sc._parse(stream, context, path)
File \"/opt/pypy/site-packages/construct/core.py\", line 295, in _parse
    return self.subcon._parse(stream, context, path)
File \"/opt/pypy/site-packages/construct/core.py\", line 867, in _parse
    subobj = sc._parse(stream, context, path)
File \"/opt/pypy/site-packages/construct/core.py\", line 295, in _parse
    return self.subcon._parse(stream, context, path)
File \"/opt/pypy/site-packages/mgz/util.py\", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
File \"/opt/pypy/site-packages/construct/core.py\", line 357, in _parse
    return self.subcon.parse(data, context)
File \"/opt/pypy/site-packages/construct/core.py\", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
File \"/opt/pypy/site-packages/construct/core.py\", line 171, in parse_stream
    return self._parse(stream, context2, \"(parsing)\")
File \"/opt/pypy/site-packages/construct/core.py\", line 867, in _parse
    subobj = sc._parse(stream, context, path)
File \"/opt/pypy/site-packages/construct/core.py\", line 1036, in _parse
    raise RangeError(\"unsane min %s and max %s\" % (min, max))
construct.core.RangeError: unsane min -1 and max -1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"process.py\", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
File \"/opt/pypy/site-packages/mgz/summary/__init__.py\", line 84, in __init__
    raise RuntimeError(\"invalid mgz file: {}\".format(e))
RuntimeError: invalid mgz file: unsane min -1 and max -1

shutil.SameFileError: '/tmp/tmp246qmkok' and '/tmp/tmp246qmkok' are the same file

Investigating what's causing it.

$ PYTHONPATH="." python mgz/cli.py extract tests/recs/de-20.06.aoe2record                                                                                                                                                                                                                        
INFO:mgz.summary:parsed header in 5.24 seconds
INFO:mgz.summary:parsed body in 0.71 seconds
Traceback (most recent call last):
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/cli.py", line 285, in <module>
    main()
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/cli.py", line 281, in main
    loop.run_until_complete(run(get_args()))
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/cli.py", line 220, in run
    await extract_rec(args.playback.split(',')[0], rec, args.select)
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/cli.py", line 61, in extract_rec
    data = await summary.async_extract(30000)
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/summary/__init__.py", line 338, in async_extract
    return await get_extracted_data(
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/summary/extract.py", line 396, in get_extracted_data
    client = await Client.create(playback, handle.name, start_time, duration, interval)
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/playback.py", line 92, in create
    self.session, self.state_ws = await self.start_instance(interval, cycles)
  File "/home/tass/dev/aoe2/aoc-mgz/mgz/playback.py", line 98, in start_instance
    shutil.copyfile(
  File "/usr/lib/python3.9/shutil.py", line 244, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/tmp/tmp246qmkok' and '/tmp/tmp246qmkok' are the same file
python mgz/cli.py extract tests/recs/de-20.06.aoe2record  6.56s user 0.33s system 98% cpu 6.996 total

construct.core.PaddingError: subcon parsed more bytes than was allowed by length

I'm seeing the following error when parsing a match played today:

  File "/home/jnyman/Ubuntu/scripts/aoe2/replay_plotter/aoe2_plot.py", line 205, in parse_match
    container = mgz.body.operation.parse_stream(f)
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "/home/jnyman/.local/lib/python3.9/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.PaddingError: subcon parsed more bytes than was allowed by length
    (parsing) -> operation -> data -> action

Can't run summary example

I'm new to python, so I'm probably missing something, but I've run the setup.py install / build, and I'm getting this error message that it can't find the AOC module

Traceback (most recent call last):
  File "summary.py", line 1, in <module>
    from mgz.summary import Summary
  File "/mnt/c/Users/Trina/Development/aoc-mgz/mgz/summary/__init__.py", line 24, in <module>
    from mgz.summary.extract import get_extracted_data
  File "/mnt/c/Users/Trina/Development/aoc-mgz/mgz/summary/extract.py", line 6, in <module>
    from mgz.playback import Client, Source
  File "/mnt/c/Users/Trina/Development/aoc-mgz/mgz/playback.py", line 
16, in <module>
    from AOC import ConfigMessage
ModuleNotFoundError: No module named 'AOC'

I tried searching to see how to install this AOC module but I can't find it.

Can there be an example of how to use the Embedded struct in body to extract the action being performed

I'm trying to understand how to use the struct to extract the action but am having trouble since Construct moved to 2.10 there are no docs there and the docs here parse_stream but do not show how to use the data

interpret new civs: Poles and Bohemians

 File "c:\dev\project\backend\mgz\mgz\model\__init__.py", line 83, in parse_match
    dataset['civilizations'][str(player['civilization_id'])]['name'],
KeyError: '38'

Someone played a new Civ "Poles". The other new one is "Bohemians".

no decoding mapping for 33620224 - (parsing) -> scenario -> game_settings

Following error occured:

match_id: 66710619
profile_id: 2821304

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2844, in _decode
    return self.decoding[obj]
KeyError: 33620224

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 311, in _parse
    return self._decode(self.subcon._parse(stream, context, path), context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2849, in _decode
    raise MappingError("no decoding mapping for %r" % (obj,))
construct.core.MappingError: no decoding mapping for 33620224

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 72, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/mgz/util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "/root/pypy-latest-linux64/site-packages/construct/core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.MappingError: no decoding mapping for 33620224
    (parsing) -> scenario -> game_settings

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/process.py", line 42, in <module>
    summary = mgz.summary.Summary(handle, None)
  File "/root/pypy-latest-linux64/site-packages/mgz/summary/__init__.py", line 84, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: no decoding mapping for 33620224
    (parsing) -> scenario -> game_settings

SyntaxError: can't use starred expression here

I've installed the library via pip install mgzdb and python 3.9, but I get

Traceback (most recent call last):
  File "/home/tass/.local/bin/mgzdb", line 5, in <module>
    from mgzdb.__main__ import setup
  File "~/.local/lib/python3.9/site-packages/mgzdb/__main__.py", line 8, in <module>
    from mgzdb.api import API
  File "~/.local/lib/python3.9/site-packages/mgzdb/api.py", line 12, in <module>
    from mgzdb.add import AddFile
  File "~/.local/lib/python3.9/site-packages/mgzdb/add.py", line 14, in <module>
    import mgz.summary
  File "~/.local/lib/python3.9/site-packages/mgz/summary/__init__.py", line 19, in <module>
    from mgz import fast
  File "~/.local/lib/python3.9/site-packages/mgz/fast.py", line 125
    player_id, formation_id, (*object_ids) = struct.unpack_from('<xhI' + str(data[0]) + 'I', data)
                              ^
SyntaxError: can't use starred expression here

Same with python 3.8.

Doesn't work with msx format

I've tried to run it against the new .msx (AOE2HD) files and have had no luck. I guess that's obvious since it's intended to work against mgz files, but do you have any tips for figuring out the new max format?

Can't get examples to run

Hello,

I am trying to use this just to extract score, but I can't run the examples without erroring out. Below is an example of the issues I get when I run the parser. I have also tried to chase the issue down the chain, but even init is failing. I am not very experienced so I could have installed struct wrong or something, but these pointers don't seem to indicate that.

Traceback (most recent call last):
File "C:\Users\Matthew\Desktop\aoc-mgz-master\parse.py", line 2, in
from mgz import header, body
File "C:\Users\Matthew\Desktop\aoc-mgz-master\mgz_init_.py", line 9, in
from mgz.header.map_info import map_info
File "C:\Users\Matthew\Desktop\aoc-mgz-master\mgz\header\map_info.py", line 17, in
"elevation"/Byte
File "C:\Users\Matthew\Desktop\aoc-mgz-master\construct\core.py", line 1963, in init
self.subcons = mergefields(*subcons)
File "C:\Users\Matthew\Desktop\aoc-mgz-master\construct\core.py", line 197, in mergefields
result.extend(select(sc))
File "C:\Users\Matthew\Desktop\aoc-mgz-master\construct\core.py", line 189, in select
return select(sc.subcon)
File "C:\Users\Matthew\Desktop\aoc-mgz-master\construct\core.py", line 189, in select
return select(sc.subcon)
File "C:\Users\Matthew\Desktop\aoc-mgz-master\construct\core.py", line 192, in select
raise ConstructError("Embedding only works with: Struct Sequence FocusedSeq Union LazyStruct")
construct.core.ConstructError: Embedding only works with: Struct Sequence FocusedSeq Union LazyStruct

ConstError: expected b'\x00\x0b' but parsed b'\x00\x00' -> expected 3 to 3, found 0

Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 1936, in _parse
    raise ConstError("expected %r but parsed %r" % (self.value, obj))
construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.ConstError: expected b'\x00\x0b' but parsed b'\x00\x00'
    (parsing) -> initial -> players

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 3 to 3, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\mgz\summary\__init__.py", line 73, in __init__
    self._header = mgz.header.parse_stream(self._handle)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\mgz\util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\dev\ant_league\backend\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 3 to 3, found 0
    (parsing) -> initial

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dev\ant_league\backend\lib\site-packages\bottle.py", line 868, in _handle
    return route.call(**args)
  File "C:\dev\ant_league\backend\lib\site-packages\bottle.py", line 1748, in wrapper
    rv = callback(*a, **ka)
  File "C:\dev\ant_league\backend\lib\site-packages\bottle_sqlite.py", line 141, in wrapper
    rv = callback(*args, **kwargs)
  File "C:\dev\ant_league\backend\website.py", line 164, in post_match
    match_info = get_match_info(file.file)
  File "C:\dev\ant_league\backend\website.py", line 48, in get_match_info
    s = Summary(data)
  File "C:\dev\ant_league\backend\lib\site-packages\mgz\summary\__init__.py", line 85, in __init__
    raise RuntimeError("invalid mgz file: {}".format(e))
RuntimeError: invalid mgz file: expected 3 to 3, found 0
    (parsing) -> initial

replay.zip

Newest version of DE breaks parser

Something's changed in the replay format that I don't quite have the savvy to comprehend, which causes parsing to flip out.

Running through body.operation.parse_stream in a normal (I think) fashion yields this traceback:

Traceback (most recent call last):
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 506, in _parse
    return packer.unpack(self.fmtstr, _read_stream(stream, self.sizeof()))[0]
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 69, in _read_stream
    raise FieldError("could not read enough bytes, expected %d, found %d" % (length, len(data)))
construct.core.FieldError: could not read enough bytes, expected 4, found 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 508, in _parse
    raise FieldError("packer %r error during parsing" % self.fmtstr)
construct.core.FieldError: packer '<f' error during parsing

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.FieldError: packer '<f' error during parsing
    (parsing) -> initial -> players -> objects -> objects -> properties -> combat -> ai -> ai -> order_history -> order_history -> target_position -> x

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 3473408 to 3473408, found 55951

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1042, in _parse
    obj.append(self.subcon._parse(stream, context._, path))
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\mgz\util.py", line 223, in _parse
    subobj = self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1594, in _parse
    obj = self.cases.get(key, self.default)._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 3473408 to 3473408, found 55951
    (parsing) -> initial -> players -> objects -> objects -> properties -> combat -> ai -> ai -> order_history

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2784, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 1050, in _parse
    raise RangeError("expected %d to %d, found %d" % (min, max, len(obj)))
construct.core.RangeError: expected 3 to 3, found 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Arx\Documents\Python Scratch\se-mgzparse.py", line 11, in <module>
    header = header.parse_stream(data)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 295, in _parse
    return self.subcon._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\mgz\util.py", line 57, in _parse
    return self.subcon._parse(new_stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 357, in _parse
    return self.subcon.parse(data, context)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 158, in parse
    return self.parse_stream(BytesIO(data), context, **kw)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 171, in parse_stream
    return self._parse(stream, context2, "(parsing)")
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 867, in _parse
    subobj = sc._parse(stream, context, path)
  File "C:\Users\Arx\AppData\Local\Programs\Python\Python36\lib\site-packages\construct\core.py", line 2788, in _parse
    raise e.__class__("%s\n    %s" % (e, path))
construct.core.RangeError: expected 3 to 3, found 0
    (parsing) -> initial

This might already be on someone's radar, but if not, here it is.

Question: AOC_PLAYBACK

Hello! Thanks for putting this parser together. Really helpful. I have a question over something I don't quite understand yet. By reading the code I found out about the env variable AOC_PLAYBACK which I assume is something external to replay recorded games in order to parse record information. How does it work? How do I link this to the CLI so I can get the full information about a match?

Thank you for your help.

Playing back aoe2record files

Hi,

Apologies if this is obvious, but it is not clear to me what should I run when trying to playback a aoe2record file. I use the cli script as

python3 mgz/cli.py play myfile.aoe2record

to which I get

INFO:mgz.summary:parsed header in 6.56 seconds
INFO:mgz.summary:parsed body in 1.40 seconds
INFO:mgz.playback:trying to connect @ ws://
INFO:mgz.playback:trying again to connect ...
INFO:mgz.playback:trying again to connect ...
INFO:mgz.playback:trying again to connect ...
INFO:mgz.playback:trying again to connect ...
INFO:mgz.playback:trying again to connect ...
ERROR:mgz.playback:failed to launch playback

What is supposed to be running at this stage? Is that an instance of aoc-headless? In that case, is there any hope for this to work with a definitive edition replay?

Thank you so much for your time!

happyleavesaoc / aoc-mgz Goto Github PK

aoc-mgz's Introduction

mgz

Supported Versions

Architecture

Parsers

Abstractions

Support

Examples

Full Parser (header) + Fast Parser (body)

Summary

Model

To JSON

Frequently Asked Questions

Contribution

Resources

aoc-mgz's People

Contributors

Stargazers

Watchers

Forkers

aoc-mgz's Issues

Recommend Projects

Recommend Topics

Recommend Org