Different payload after decode/encode roundtrip about pyais HOT 8 CLOSED

Thanks, that definitely fixes it for me! I like the idea of encoding the special values as enums.

Just one addition, unrelated to this issue: when I inspect MessageX instances, the special turn values have been converted to floats. I thinks that might be due to the attrs library, since the bit_field the to_turn function acts as a converter on is declared as a float attribute.

from pyais.

I got a bit closer to the issue: the logic in messages.Payload.to_bitarray skips None fields so they don't show up in the serialized payload. Here, None is interpreted as "skip this data while serializing".

It is used in another way in the messages.from_turn function, which is responsible for converting the AIS representation for rate of turn to degrees/s. The AIS representation allows for special values meaning "no information available", which that function converts to None.

This means that any AIS position report with a turn field containing the "no info" special values gets serialized to a payload where the 8 Bits encoding turn are missing. Looking at the payload length before and after the round trip, it shrinks from 168 Bit to 160 Bit.

from pyais.

The messages.Payload.from_bitarray function did not indicate a problem, but this might be intended. As far as I can tell, it

• assigns None to fields that start past the length of the input payload,
• uses as many bits as there are left for a field that starts within the input payload but goes past its end.

In my example above, the final radio field expects 19 Bits but gets only 11, which are cast to an integer without raising an error.

from pyais.

Here's another snippet highlighting what's happening

from pyais.messages import NMEAMessage

nmea = NMEAMessage(b'!AIVDM,1,1,,A,13HOI:0P0000VOHLCnHQKwvL05Ip,0*23')
ais = nmea.decode()
print(f'AIS payload 1: {nmea.bit_array.to01()}')
bits = ais.to_bitarray().to01()
print(f'AIS payload 2: {bits[:42]}{" "*8}{bits[42:]}')

Output

AIS payload 1: 000001000011011000011111011001001010000000100000000000000000000000000000100110011111011000011100010011110110011000100001011011111111111110011100000000000101011001111000
AIS payload 2: 000001000011011000011111011001001010000000        0000000000000000000000100110011111011000011100010011110110011000100001011011111111111110011100000000000101011001111000

The input message contains a 128 decimal in the turn field (no info) and those 8 bits get snipped out of the re-serialized data.

from pyais.

There's another implication for Message 21's final name_ext field. Looks like it can be between 0 and 88 Bits long. The current implicit conversion of missing Bits -> None and None -> no Bits works but we might want to keep it in mind when fixing this issue.

from pyais.

@pgorczak

Thank you for your investigation. I can reproduce your examples. And I agree that the encoding and decoding of turn values should be consistent. Actually, there was a bug in there. If the value for turn was any of 127,-127,-128 the value was set to None. I changed this behavior slightly, by adding an IntEnum TurnRate that is returned instead:

class TurnRate(IntEnum):
    # Source: https://gpsd.gitlab.io/gpsd/AIVDM.html#_types_1_2_and_3_position_report_class_a
    # turning right at more than 5deg/30s (No TI available)
    NO_TI_RIGHT = 127
    # turning left at more than 5deg/30s (No TI available)
    NO_TI_LEFT = -127
    # 80 hex) indicates no turn information available (default)
    NO_TI_DEFAULT = -128

This has two benefits:

• the decoding/encoding functions behave consistently
• the original integer value remains accessible. Therefore, as a user, I can differentiate between:
  • turning right at more than 5deg/30s (No TI available)
  • turning left at more than 5deg/30s (No TI available)
  • indicates no turn information available (default)

By doing so, your example no longer produces different payloads. In your three way roundtrip example the last three parts are now always identical. But the first message might still differ from the last three messages. In my observation, however, this was due to rounding errors for the lat, lon fields. E.g. 18.321188 vs 18.321187.

I encourage you to try my changes on this PR. Does it resolve your issue?

from pyais.

I inspect MessageX instances, the special turn values have been converted to floats

This isn't really an issue. As a user, I don't know what type of value I might get for turn in advance. Therefore, I always need to check for special cases like that:

assert decoded.turn == TurnRate.NO_TI_DEFAULT

and this is True if the value is both -128 or -128.0.

from pyais.

This fix is released in version 2.2.2

from pyais.