'utf-8' codec can't decode byte about cantools HOT 15 CLOSED

I implemented what I think is a fix for your error. Install version 20.3.1 with pip3 install -U cantools.

File encoding is hard to get right in Python, especially if one want to be compatible with both Python 2 and 3.

from cantools.

Still dumps to console

pi@raspberrypi:~ $ cdump can0 | cantools decode --single-line j1939.dbc
'utf-8' codec can't decode byte 0xb0 in position 4832: invalid start byte
pi@raspberrypi:~ $

from cantools.

Actually looks like it didn't upgrade, I'll try again tomorrow.

cantools --version still shows 20.3.1

from cantools.

Ok.

As a workaround you can remove all degree-symbols in the DBC-file. That said, lets continue the troubleshooting to solve the issue.

I can successfully read a DBC-file with a degree-symbol on my computer.

from cantools.

So I got cantools to upgrade and the error about utf-8 went away. But now I don't see any decoding.

pi@raspberrypi:~ $ candump can0
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
can0 1CF004FF [8] 00 00 00 00 00 00 00 00
^C
pi@raspberrypi:~ $ candump can0 | cantools decode --single-line j1939.dbc
^C
pi@raspberrypi:~ $ cantools --version
20.3.1
pi@raspberrypi:~ $

from cantools.

I let the following command set awhile:
candump can0 | cantools decode --single-line j1939.dbc

And I get this:
pi@raspberrypi:~ $ candump can0 | cantools decode --single-line j1939.dbc
DBC: "Invalid DBC syntax at line 8870, column 80: 'CM_ SG_ 2633785086 StOfHealth "This signal represents the expected remaining ">!<Life" of the High Voltage Energy Storage 1 pack (or system), in percent. This value should begin at 100%, and end at 0% when �no� energy is able to be put in or taken out. �No� could be absolute zero, ...";': Expected ";".", KCD: "syntax error: line 1, column 0", SYM: "Only SYM version 6.0 is supported."

from cantools.

So I tried removing the degree-symbols and I still got this error:
pi@raspberrypi:~ $ candump can0 | cantools decode --single-line j1939_test.dbc DBC: "Invalid DBC syntax at line 8870, column 80: 'CM_ SG_ 2633785086 StOfHealth "This signal represents the expected remaining \">!<Life\" of the High Voltage Energy Storage 1 pack (or system), in percent. This value should begin at 100%, and end at 0% when �no� energy is able to be put in or taken out. �No� could be absolute zero, ...";': Expected ";".", KCD: "syntax error: line 1, column 0", SYM: "Only SYM version 6.0 is supported."

So I removed all the backward-slash" and I got the following:
pi@raspberrypi:~ $ candump can0 | cantools decode --single-line j1939_test.dbc
invalid literal for int() with base 10: '0M'

from cantools.

I'll have a look at it soon :)

from cantools.

Can you by any chance share your DBC-file? It would greatly speed up the troubleshooting. At least run candump can0 | cantools --debug decode --single-line j1939_test.dbc for a backtrace.

from cantools.

untouched file.
j1939.zip

from cantools.

Will likely have a fix in a few hours.

from cantools.

Fixed in 20.3.2.

Though, the M suffix for ManufacturerCodeQualifierFlag m0M is ignored, so that message might not be decoded correctly. I have limited knowledge about multiplexed signals in DBC files, and I simply don't know how to use it. Any help is appreciated.

Btw, the reason it takes some time before error messages are printed is that cantools uses the pyparsing package to parse the DBC-file. Pyparsing is quite slow, and the DBC-file is huge.

from cantools.

It's now working enough I can use it. It isn't decoding my messages because it seems the CANID has to be an exact match which is OK. Beyond that you would be getting into J1939 standard specifics.

Currently this is what I'm seeing:
can0 1CF004FE [8] 00 00 00 00 00 00 00 00 :: Unknown frame id 485491966

It's not decoding because the frame id (CANID) isn't an exact match as stated. You could decode based on PGN, but that will not cover all the specific j1939 frames. You would have to dig into the J1939 standard to cover all/most things. You may look at the https://rbei-etas.github.io/busmaster/ project as it supports J1939 some what.

Anyway it's working enough now I can use it, anything else would just be added features.

I realized it's slow because of the parsing, it's a huge dbc but has almost everything in it. Would be great if it's faster but not necessary. I guess it would be best to data log the device in question then make just a dbc for that device. (which is what I will end up doing).

Thank you for the quick fix/update!

from cantools.

What about adding an optional argument to load_file(), frame_id_mask, which can be used to ignore certain bits in the frame id when decoding (and possibly other places)? Does it make any sense to add that?

from cantools.

I added a frame id mask argument that can be used to ignore bits in the frame id. I'm not sure it will solve the general case, but at least it works for your example.

It's part of version 20.4.0.

$ echo " can0  1CF004FE   [8] 00 00 00 00 00 00 00 00" | cantools decode --frame-id-mask 0x03ffff00 j1939.dbc
 can0  1CF004FE   [8] 00 00 00 00 00 00 00 00 ::
EEC1(
    EngTorqueMode: 0,
    ActlEngPrcntTorqueHighResolution: '+0.000%' %,
    DriversDemandEngPercentTorque: -125 %,
    ActualEngPercentTorque: -125 %,
    EngSpeed: 0.0 rpm,
    SrcAddrssOfCntrllngDvcForEngCtrl: 0,
    EngStarterMode: 'start not requested',
    EngDemandPercentTorque: -125 %
)

from cantools.