Skipping RTCM messages about pyubx2 HOT 8 CLOSED

@flytrex-vadim FYI added branch accommodate-RTCM-mixed-msg-stream with some minor tweaks to your 'patch', including an RTCM protocol filter setting so users can select whether or not to see any RTCM3 data in the stream.

from pyubx2.

Hi @flytrex-vadim,

Feedback and contributions always welcome, but if it was your intention to submit a code change for pyubx2, could I ask that you refer to the guidelines in CONTRIBUTING.md and submit a formal Pull Request rather than a code snippet in a .txt file?

That being said, I'm unclear what you mean by 'broken iteration' or your expectations were here?pyubx2 is intended to be a UBX protocol parser, not an RTCM protocol parser. The UBXReader module does - as of v1.2.4 - also parse NMEA messages (via the companion pynmeagps library), but it should simply ignore and discard any incoming data that is not a valid UBX or NMEA message, and either continue with the next valid UBX or NMEA message or terminate if there are no such messages remaining in the input stream.

Are you saying that this is not the case? What actual error message are you getting? Would you be able to provide a copy of the raw binary datalog you're attempting to process so that I can test it at our end?

from pyubx2.

Hi @semuadmin ,

1. Submitting a pull request would require me figure out how to do that, a task I'm not sure I have the time for at the moment. So I chose to share the code immediately in a way I'm familiar with, in the hope that it would be useful for somebody :)
2. Indeed the scenario you described is not the case in the vanilla 1.2.4. Unknown message types cause a few "Unknown protocol" messages at misc binary locations followed by stop of iteration. I had to make my changes so that it ignores the RTCM messages and continues parsing the UBX and NMEA until the end of the logfile.
3. What's the best way to send you the logfile privately ?

from pyubx2.

@flytrex-vadim

Can I confirm what you're setting the optional quitonerror keyword argument to? By default, this is set to 1 (= log errors to terminal and continue), the other options being 0 (= ignore any errors) or 2 (= re-raise errors and terminate).

If you have this set to 2 and your data stream contains (for example) RTCM data, then you will indeed get an error - this is expected behaviour. If you're using the default value of 1, then you will see an 'invalid protocol' error message written to the terminal but the iteration should not terminate at that point unless there happen to be no further UBX or NMEA messages in the data stream. If you want to suppress all 'invalid protocol' messages, you can set quitonerror to 0. Can you confirm?

You can send a binary data log to my email address [email protected] but I recommend checking the above first. Cheers.

from pyubx2.

I can confirm using the default flags. this is also validated by the fact that I see multiple 'invalid protocol' messages as the reader goes over all the bytes of the unknown message, two at a time.
Then at some point it finds a pair that accidentally match the header, goes into the parsing code and fails quietly somewhere (couldn't trace exactly where).
This code seems to not be built for re-synchronization after encountering unknown bytes for several reasons:

• two bytes are read at each iteration, so after odd-sized unknown message the reader would not be positioned on header
• after accidentally going into parsing and failing due to incorrect CRC or other error, the unparsed bytes are not returned to the stream to be tried again from next valid header.

from pyubx2.

OK if you could send me a copy of your raw datalog I'll test it at our end and see exactly how and why it's failing. Thanks.

I had a quick look at your 'patch' and it appears to be attempting to parse RTCM data headers, which is (currently at least) beyond the scope of what pyubx2 is designed for. If there is indeed a bug in the UBXReader.read() algorithm I'd ideally like to fix it for all possible non-UBX data streams, not just RTCM.

from pyubx2.

sent. I've trimmed the log to reasonable size while making sure the offending section is close to the beginning.

I'm afraid that any attempt to generically handle all possible binary data inclusion would be ad-hoc and error prone. According to the documentation, at least the F9 units support only UBX, NMEA and RTCM output format. So adding the RTCM header detection allows clean support of all ublox outputs regarding of the device configuration. At the cost of only a few extra lines of code.

from pyubx2.

Having seen the problem - thanks for the log and code suggestion - I'm inclined to agree. Accommodating UBX, NMEA and RTCM would seem sensible and proportionate. At some point we may get round to implementing a proper Python RTCM parser, but for now I'll endeavour to at least accommodate a mixed stream (but only actually parse UBX or NMEA). Surprised this hasn't come up before.

Leave it with me - thanks again for the shout.

from pyubx2.