IndexError: bitarray index out of range about pyais HOT 11 CLOSED

Thank you again for the messages. I will dig into it.

from pyais.

Hey,

I integrated a native UDPStream in the newest release. It includes out-of-order message assembly. Usage is straight forward:

from pyais.stream import UDPStream

host = "127.0.0.1" # change this
port = 55555 # and change that

for msg in UDPStream(host, port):
    msg.decode()
    # do something 

@void4main Would you like to check it out and give me some feedback on how it works for you?

Greetings

from pyais.

I had some time to test it with live AIS data and it worked perfectly fine for me.
Since I get broadcasted UDPs I had to set an emtpy host address:
host = "" port = 10110

It crashed once in the beginning but only inside the json conversion:
message = msg.decode().to_json() and I couldn't reproduce the problem.
With the UDPStream I inserted ~300.000 entries to my mongodb by now.

I had one more crash adding data to the db:
b'!AIVDM,2,1,1,A,14S1fp00002E5k8A5Ejv1c>200Rfr>l00000000s1rA<<5kQ0ClmAFP0,0*78!AIVDM,2,2,1,A,000000000000000,2*25'

Traceback (most recent call last):
  File "./get3.py", line 39, in <module>
    result = posts.insert_one(message)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 657, in insert_one
    bypass_doc_val=bypass_document_validation),
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 562, in _insert
    check_keys, manipulate, write_concern, op_id, bypass_doc_val)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 551, in _insert_one
    self.__write_response_codec_options)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 480, in _legacy_write
    rqst_id, msg, max_size = func(*args)
OverflowError: MongoDB can only handle up to 8-byte ints

but haven't looked into it and probably hasn't got anthing to do with your code.
Thank you for the implementation.

from pyais.

Hey,

thanks your issue. Could you provide the exact raw message that leads to this crash?

Thank you

from pyais.

Yes, I try to get these ...

from pyais.

I have two examples for now.
The decoding of part 1 is alwasy ok but part 2 leads to a crash:

1)

b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07'
b'!AIVDM,2,2,1,A,F@V@00000000000,2*35'

Traceback (most recent call last):
File "./get2.py", line 34, in
ais_print = message.decode()
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 139, in decode
return AISMessage(self)
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 151, in init
self.content = decode(self.nmea)
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 677, in decode
return DECODE_MSGmsg.ais_id
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 451, in decode_msg_22
'addressed': bit_arr[139],
IndexError: bitarray index out of range

2)

b'!AIVDM,2,1,9,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*0F'
b'!AIVDM,2,2,9,A,F@V@00000000000,2*3D'

Traceback (most recent call last):
File "./get2.py", line 34, in
ais_print = message.decode()
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 139, in decode
return AISMessage(self)
File "/usr/local/lib/python3.7/dist-packages/pyais/messages.py", line 151, in init
self.content = decode(self.nmea)
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 677, in decode
return DECODE_MSGmsg.ais_id
File "/usr/local/lib/python3.7/dist-packages/pyais/decode.py", line 451, in decode_msg_22
'addressed': bit_arr[139],
IndexError: bitarray index out of range

Looks like a problem of multipart-messages?

from pyais.

So. I took a look at your issue. There wasn't actually a bug in the decoding itself. Instead, you slightly misused the interface.

When working with multi-line messages you need to assemble them before you can decode them.

So you either have to do it like this:

msg_1_part_0 = b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07'
msg_1_part_1 = b'!AIVDM,2,2,1,A,F@V@00000000000,2*35'

NMEAMessage.assemble_from_iterable(
        messages=[
            NMEAMessage(msg_1_part_0),
            NMEAMessage(msg_1_part_1)
        ]
    ).decode().to_json()

msg_2_part_0 = b'!AIVDM,2,1,9,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*0F'
msg_2_part_1 = b'!AIVDM,2,2,9,A,F@V@00000000000,2*3D'

NMEAMessage.assemble_from_iterable(
        messages=[
            NMEAMessage(msg_2_part_0),
            NMEAMessage(msg_2_part_1)
        ]
    ).decode().to_json()

or with the newest release it is also possible to do soimething like this:

 messages = [
            b'!AIVDM,2,1,1,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*07',
            b'!AIVDM,2,2,1,A,F@V@00000000000,2*35',
            b'!AIVDM,2,1,9,A,538CQ>02A;h?D9QC800pu8@T>0P4l9E8L0000017Ah:;;5r50Ahm5;C0,0*0F',
            b'!AIVDM,2,2,9,A,F@V@00000000000,2*3D',
        ]
for msg in ByteStream(messages):
        decoded = msg.decode()

But depending on the source of your messages you might want to take a look at the FileReaderStream o TCPStream.

I hope this helps. If you have further questions, feel free to ask. But I mark this issue as resolved.

from pyais.

Thank you, it helps!

(I get the data via antenna and udp broadcast, so it might get tricky to assemble the messages in the correct order but let's see .. :)

from pyais.

Interesting. I thought about adding out-of-order message support. I might add this feature in a future release together with an UDPStream.

from pyais.

Hey, that's great. I'll test it asap ...

from pyais.

I am happy to hear, that it works for you! :-)

If you observe any new issues or if you need another feature, feel free to open a new issue.

from pyais.