How to update the address of the ECU in A2L generated by MATLAB from the map file

In some cases, the input file can be encoded in a non utf-8 encoding. At the moment, the function process_input_file(fp, parser: Parser, allow_partial: bool) -> RootNodeType does not provide any way to handle this. The new prototype should be process_input_file(fp, parser: Parser, allow_partial: bool, encoding: str = None) -> RootNodeType

Missing LAYOUT property for measurements

When parsing A2L measurement objects, the ast does not contain layout information.

The property seems correctly defined in a2l-grpc/grammar/A2L.g4

Reproduce

When using the A2LParser.tree_from_a2l() on the attached dumb.txt (convert to .a2l), the properties of the "Dumbo" measurement contain no LAYOUT.
dumb.txt

This happens with a newly build pya2l and newest artifacts from a2l-grpc

The following a2l sample leads to a parser crash:

...
/begin SEGMENT
    2       /* segment logical number */
    0x02       /* number of pages */
    0x00       /* address extension */
    0x00       /* compression method */
    0x00       /* encryption method */
    /begin PAGE
        0x0 /* page number */
        ECU_ACCESS_WITH_XCP_ONLY
        XCP_READ_ACCESS_WITH_ECU_ONLY
        XCP_WRITE_ACCESS_WITH_ECU_ONLY
        INIT_SEGMENT 0x0
    /end PAGE
    /begin ADDRESS_MAPPING
        0xe0000
        0x40000400
        0x21c
    /end ADDRESS_MAPPING
/end SEGMENT
...

      /begin MOD_COMMON
        ""    /* Comment */
        ALIGNMENT_BYTE 1
        ALIGNMENT_WORD 1
        ALIGNMENT_LONG 1
        ALIGNMENT_FLOAT32_IEEE 1
        ALIGNMENT_FLOAT64_IEEE 1
        BYTE_ORDER MSB_FIRST
        DEPOSIT ABSOLUTE
    /end MOD_COMMON

Traceback (most recent call last):
File "E:/CanD/test.py", line 15, in
a2l = A2lParser(a2l_string)
File "D:\python3.5\lib\site-packages\pya2l-0.0.1-py3.5.egg\pya2l\parser\grammar\parser.py", line 38, in init
self._yacc.parse(string)
File "D:\python3.5\lib\site-packages\ply\yacc.py", line 333, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "D:\python3.5\lib\site-packages\ply\yacc.py", line 1201, in parseopt_notrack
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "D:\python3.5\lib\site-packages\ply\yacc.py", line 192, in call_errorfunc
r = errorfunc(token)
File "D:\python3.5\lib\site-packages\pya2l-0.0.1-py3.5.egg\pya2l\parser\grammar\parser.py", line 49, in p_error
raise A2lFormatException('invalid sequence at position ', p.lexpos, string=p.lexer.lexdata)
pya2l.parser.grammar.parser.A2lFormatException: invalid sequence at position 59861
...SB_FIRST DEPOSIT ABSOLUTE /end MOD_COMMON /begin IF_DATA XCPplus 0x102 /begin PAG 0x01 /* MAX_SEGMENTS */ /end PAG /begin XCP_ON_CAN

Hello!

I just updated to release 0.0.9 and unfortunately, my virus scanner ESET Endpoint Antivirus deleted one dll as suspicious object:

Time;Scanner;Object type;Object;Detection;Action;User;Information;Hash;First seen here
06/09/2023 14:14:33;Real-time file system protection;file;C:\Program Files\Python39\Lib\site-packages\pya2l\a2l_grpc\a2l_grpc_windows_amd64.dll;Suspicious Object;cleaned by deleting;;Event occurred on a new file created by the application: C:\Program Files\Python39\python.exe (46B758081313D2968A2E4293C6DDAB7DD25FBC0B).;9047E0CE705F52EDBC96215570904D129210247B;06/09/2023 14:14:24

use slots A2lNode classes in order to spare memory.

I install pya2l directly via pip. The test code is copied from introduction.

Traceback (most recent call last):
  File "pya2ltest.py", line 58, in <module>
    assert python_object_to_json_string(a2l.tree.project.json, sort_keys=True, indent=4) == """{
AttributeError: 'Project' object has no attribute 'json'

Traceback (most recent call last):
File "E:/CanD/test.py", line 14, in
a2l = A2lParser(a2l_string)
File "D:\python3.5\lib\site-packages\pya2l\parser\grammar\parser.py", line 163, in init
self._yacc.parse(string)
File "D:\python3.5\lib\site-packages\ply\yacc.py", line 333, in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
File "D:\python3.5\lib\site-packages\ply\yacc.py", line 1201, in parseopt_notrack
tok = call_errorfunc(self.errorfunc, errtoken, self)
File "D:\python3.5\lib\site-packages\ply\yacc.py", line 192, in call_errorfunc
r = errorfunc(token)
File "D:\python3.5\lib\site-packages\pya2l\parser\grammar\parser.py", line 174, in p_error
raise A2lFormatException('invalid sequence at position ', p.lexpos, string=p.lexer.lexdata)
pya2l.parser.grammar.parser.A2lFormatException: invalid sequence at position 58037
..._ECU_ONLY INIT_SEGMENT 0x0 /end PAGE /begin ADDRESS_MAPPING 0xe0000 0x40000400 0

the error format in a2l file as follow:

                   /begin PAGE
                        0x0 /* page number */
                        ECU_ACCESS_WITH_XCP_ONLY
                        XCP_READ_ACCESS_WITH_ECU_ONLY
                        XCP_WRITE_ACCESS_WITH_ECU_ONLY
                        INIT_SEGMENT 0x0
                    /end PAGE
                    /begin ADDRESS_MAPPING
                    0xe0000
                    0x40000400
                    0x21c
                    /end ADDRESS_MAPPING 

Please notice "ETK_XETK_ACCESS" after the '/begin' and another place "XCPplus" is using in this a2l file.

/begin MEMORY_SEGMENT Dst80100000 "" DATA FLASH INTERN 0x80100000 0x7FAE0 -1 -1 -1 -1 -1 

       /begin IF_DATA ETK_XETK

           /begin ETK_XETK_ACCESS

              SERIAL_INTERFACE

           /end ETK_XETK_ACCESS

       /end IF_DATA







        /begin IF_DATA XCPplus 0x0102

          /begin SEGMENT

            2       /* segment logical number */

            0x02       /* number of pages */

            0x00       /* address extension */

            0x00       /* compression method */

            0x00       /* encryption method */

            /begin CHECKSUM

                XCP_CRC_32 /* checksum: add bytes to 16bit result */

                MAX_BLOCK_SIZE 0xFFFFFFFF

            /end CHECKSUM

            /begin PAGE

              0x00       /* page number */

              ECU_ACCESS_DONT_CARE

              XCP_READ_ACCESS_DONT_CARE

              XCP_WRITE_ACCESS_NOT_ALLOWED

            /end PAGE

            /begin PAGE

              0x01       /* page number */

              ECU_ACCESS_DONT_CARE

              XCP_READ_ACCESS_DONT_CARE

              XCP_WRITE_ACCESS_WITH_ECU_ONLY

            /end PAGE

          /end SEGMENT

        /end IF_DATA

    /end MEMORY_SEGMENT

I have a working python script which uses your pya2l module. As long as i run my script from VisualStudioCode or CMD everything is working.
But when i'm trying to generate an exe file out of my python script via PyInstaller or Nuitka, in both cases the generated exe file is not running and in the console i get the following error message. The following message is reported by the exe generated with PyInstaller. The error message from the one generated with Nuitka is, besides the filepaths, identical.

Traceback (most recent call last):
  File "my_script.py", line 17, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "my_lib.py", line 5, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "my_a2l_parser.py", line 5, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pya2l\__init__.py", line 12, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pya2l\cli.py", line 18, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pya2l\parser.py", line 15, in <module>
  File "PyInstaller\loader\pyimod02_importers.py", line 419, in exec_module
  File "pya2l\protobuf\API_pb2.py", line 15, in <module>
ModuleNotFoundError: No module named 'protobuf'
[10064] Failed to execute script 'my_script' due to unhandled exception!

Used versions (All packages installed via pip):
Python: 3.11.1
pya2l 0.1.2
Nuitka 2.0.1
pyinstaller 6.3.0

Looking into one of the autogenerated files, e.g. protobuf\API_pb2.py, Pylance installed with VisualStudioCode reports reportMissingImports for line 15. This fits to the error message shown above.

If i change all similar includes in all autogenerated files from e.g.
from protobuf import A2L_pb2 as protobuf_dot_A2L__pb2
to
from . import A2L_pb2 as protobuf_dot_A2L__pb2
i can generate working exe files with both PyInstaller and Nuitka and the Pylance warning vanishes.

For me it seems as if the script which generates the content of the protobuf folder makes an error.

Additional note:
When using PyInstaller and Nuitka you have to explicitly add a2l_grpc_windows_amd64.dll to get a working exe file. But this is not related to this issue.

File "\pya2l-0.0.1\pya2l\parser\grammar\parser.py", line 174, in p_error raise A2lFormatException('invalid sequence at position ', p.lexpos, string=p.lexer.lexdata)

Here's the repeating part of the a2l file that I am using:
/begin CHARACTERISTIC SFB_R_FFO_DE.Properties.1.Qly "" VALUE 36453 _UBYTE 0 NO_COMPU_METHOD 0 255
/begin ANNOTATION
/begin ANNOTATION_TEXT
"Factor 1.0"
"Offset 0"
/end ANNOTATION_TEXT
ANNOTATION_LABEL "CalInPatcher Additional Attributes"
/end ANNOTATION
SYMBOL_LINK "SFB_R_FFO_DE.Properties.1.Qly" 0
/end CHARACTERISTIC

If I remove the line with SYMBOL_LINK, the error is gone.

Could you explain how to debug this?

I got invalid sequence error when test the pya2l for my project's a2l file like below.
`from pya2l.parser import A2lParser as Parser

a2l_string = """
ASAP2_VERSION 1 60
    /begin PROJECT prj "PRJ"
        /begin MODULE DIM ""
            /begin A2ML
                struct aaaa{
                    float;
                    taggedstruct{
                        block "BBBB" struct{
                            uchar;
                            taggedstruct{                              
                                (block "C2222" struct{
                                    ulong;                            
                                })*;
                                (block "C3333" struct{
                                    taggedunion{
                                        "TEST";  ////////////// ERROR //////////A2lFormatException: invalid sequence
                                        "DDDD" struct{
                                            "TEST111" ulong;  ////////////// ERROR //////////A2lFormatException: invalid sequence
                                        };
                                    };
                                    taggedstruct{
                                       "TEST222" (ulong)*;  ////////////// ERROR //////////A2lFormatException: invalid sequence
                                    };
                                })*;   
                            };
                        };
                    };
                };
            /end A2ML
        /end MODULE
    /end PROJECT"""
a2l = Parser(a2l_string)`

Hi,

I wanted to use this module for merging A2L files. Works fine so far, but I noticed, that for the "COMPU_METHOD" objects with linear conversion rule, the conversion coefficents are not parsed by the "tree_from_a2l" method.

When converting a2l --> tree --> a2l and saving the result, I get these error-messages in the Vector ASAP2 Studio:

The information is lost directly with the tree_from_a2l method.

(Test file had ASAP2 version 1.61)