Code Monkey home page Code Monkey logo

Comments (11)

geeksville avatar geeksville commented on August 26, 2024

oh that is super interesting. I'll write more on this bug later tonight (just got back from sickness)

from firmware.

geeksville avatar geeksville commented on August 26, 2024

ok, checked the code, this means we have a bug (likely a race condition) where the lora radio is staying stuck in transmit mode (rather than returning to rxmode after sending). I'm now looking to try to find the cause and something to check for it and report the bug earlier.

from firmware.

geeksville avatar geeksville commented on August 26, 2024

@spattinson I'm releasing a build later today 0.1.9 that I think probably fixes this (though I'm having a hard time reproducing it). I fixed one possible root cause, but also added a watchdog to see if this bug occurs, if it does it 'fixes' it and reports back to the mothership (eventually).

Can you check with 0.1.9 if it prints "ERROR! Bug! Tx packet took too long to send, forcing radio into rx mode"? that will tell me if the root bug still exists? (though it will also apply a workaround to make the bug not fatal)

from firmware.

spattinson avatar spattinson commented on August 26, 2024

Yes sure, here are two logs, I think only 2c.txt exhibits the issue, 44.txt is its partner node, there is also a backtrace in there, I don't know if that is related?
logs.zip

from firmware.

geeksville avatar geeksville commented on August 26, 2024

yeah - that was super helpful. Two interesting things:

I see rxBad going up, which should almost never happen (I have very few packets with bad crc received on my test bench) and twice it screwed up and tried to send when we were already sending. I have a bug that needs fixing and will work on this later today.

for my reference the relevant bits are:


fix type 3
fix type 3
fix type 3
new gps pos lat=51.480528, lon=-0.616718, alt=52
got gps notify
Sending position to mesh
Update DB node 0x2c for variant 1, rx_time=1585144143
Providing time to mesh 1585144143
immedate send on mesh (txGood=2,rxGood=0,rxBad=1)
fix type 3
new gps pos lat=51.480466, lon=-0.616749, alt=58
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144173
ERROR! Bug! Tx packet took too long to send, forcing radio into rx modefix type 3
new gps pos lat=51.480512, lon=-0.616759, alt=61
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144203
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144223
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144373
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144403
fix type 3
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144443
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144463
fix type 3
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144503
fix type 3
fix type 3
new gps pos lat=51.480403, lon=-0.616723, alt=50
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144543
fix type 3
new gps pos lat=51.480540, lon=-0.616819, alt=57
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144573
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144583
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144733
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144793
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144833
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144873
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144913
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144953
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144993
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Sending position to mesh
Update DB node 0x2c for variant 1, rx_time=1585145053
Providing time to mesh 1585145053
immedate send on mesh (txGood=2,rxGood=0,rxBad=2)
assertion "!sendingPacket" failed: file "src/CustomRF95.cpp", line 155, function: void CustomRF95::startSend(MeshPacket*)
abort() was called at PC 0x4010362f on core 1

Backtrace: 0x40094cb4:0x3ffd8f80 0x40094ee5:0x3ffd8fa0 0x4010362f:0x3ffd8fc0 0x400d2686:0x3ffd8ff0 0x400d2789:0x3ffd9010 0x400d3d51:0x3ffd9040 0x400d3ed1:0x3ffd9070 0x400d4469:0x3ffd9090 0x400d4492:0x3ffd90b0 0x400d2ba9:0x3ffd90d0 0x400d4d63:0x3ffd9130 0x400d29b9:0x3ffd9160 0x400d5993:0x3ffd9180 0x400e78ad:0x3ffd91b0 0x400913c9:0x3ffd91d0

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee

from firmware.

geeksville avatar geeksville commented on August 26, 2024

actually - looking more, only screwed up once. My little emergency txwatchdog was forcing the radio into rxmode correctly, but not clearing the flag that said we were busy sending.

Hmm - I had two nodes running all night ping ponging to each other that didn't show this problem. hmm... still looking.

from firmware.

geeksville avatar geeksville commented on August 26, 2024

also, I see you were running a local build. I just noticed that platform IO doesn't pull down the latest git dependencies unless you do "platformio update", so you might not have had my fixes in the RadioHead library. But I'll still think about this problem later today.

from firmware.

spattinson avatar spattinson commented on August 26, 2024

I have updated radiohead, it was behind:
Updating RadioHead @ 57ac37e [0b279d3]
I have made some changes to my local copy for #48 and updated that issue. I have those changes and updated RadioHead so I will leave it logging again. I am just leaving them on all the time and periodically waking with button and send a msg from app

from firmware.

geeksville avatar geeksville commented on August 26, 2024

@spattinson btw: I just reproduced the original problem on one of my boards. Your your 0x2c board that showed the problem, did that board have a battery installed? On my board which can show the problem - it occurs 100% of the time if I don't have a battery installed, 0% of the time if I do have one installed. I'm beginning to suspect there is something marginal in the power rail that powers the LORA radio and with no battery, if the USB power isn't up-to-snuff the radio never sets TX complete.

from firmware.

spattinson avatar spattinson commented on August 26, 2024

All my testing is being done with batteries, I basically leave them on all the time now, and plugged in. I have not witnessed the problem recently, but I have not been watching closely either, I am mostly tee'ing output to logs and there are no reboots in them.

from firmware.

geeksville avatar geeksville commented on August 26, 2024

ok sounds good! Sounds like bug probably squashed. Thanks!

from firmware.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.