dimameshcharakou / fit_parser Goto Github PK

We are using this gem in order to parse FIT file data and we are running on a kubernetes cluster. We have noticed that this gem is throwing lots of errors depending on our pod configuration. We wanted to test this theory so over the past 12 hours we have some graphs of the error activity.

The first error we see which was starting to become common was a

No definition for local message type: [INSERT NUMBER HERE]
ERROR: /root/vendor/bundle/ruby/2.7.0/gems/fit_parser-1.0.9/lib/fit_parser/file/record.rb:25:in `read'

This error bubbles up directly from this gem - we also see the following error which comes from bindata but since this is a dependency here, we know it is being triggered via this gem as well.

selection [INSERT NUMBER HERE] does not exist in :choices for obj.global_message_number
ERROR: /root/vendor/bundle/ruby/2.7.0/gems/bindata-2.2.0/lib/bindata/choice.rb:119:in `instantiate_choice'

Currently, you have locked bindata at v2.2.0 and that gem is now on v0.2.4.8. Looking at the commit history, it seems like you tried to upgrade with v2.3.1 but were having multithread issues.

In order to test to see if these errors were getting out of control as threads were increased per pod, we changed our configs and noticed this trend over time. The following numbers explain the event lines from left to right:

1. We move the current config from 1 pod with 2 threads to 1 pod with 6 threads as we were not getting near our CPU or memory limits
2. Again, we keep pushing threads as we have not come close to our CPU limits and have not noticed these errors a lot (thinking they are random and happening with corrupt FIT files). We move the current config of 1 pod with 6 threads to 1 pod with 10 threads. This shows a massive uptick in these errors
3. Curious after some digging into this, we move the current config of 1 pod with 10 threads to 5 pods with 1 thread each to see if these errors are reacting to single pod multithreading issues

The top graph is simply showing the processing activity and the number of FIT files rolling through the system (you can ignore the different colors). The bottom graph is showing the number of these errors in the logs. The No definition error is aqua here and the selection error is in yellow.

As you can see, from this evidence we are confident that this is a multithreading issues since as soon as we moved to single-thread pods, all these errors vanished. My main question is if you have tried the newer versions of bindata here to see if this fixes your multithreading issue or if you have tried to fix this issue in any way? Do you know what the main issue is and why this gem seems to throw these errors only during multithreaded processes?

Thanks,
Sven