Comments (11)
oh that is super interesting. I'll write more on this bug later tonight (just got back from sickness)
from firmware.
ok, checked the code, this means we have a bug (likely a race condition) where the lora radio is staying stuck in transmit mode (rather than returning to rxmode after sending). I'm now looking to try to find the cause and something to check for it and report the bug earlier.
from firmware.
@spattinson I'm releasing a build later today 0.1.9 that I think probably fixes this (though I'm having a hard time reproducing it). I fixed one possible root cause, but also added a watchdog to see if this bug occurs, if it does it 'fixes' it and reports back to the mothership (eventually).
Can you check with 0.1.9 if it prints "ERROR! Bug! Tx packet took too long to send, forcing radio into rx mode"? that will tell me if the root bug still exists? (though it will also apply a workaround to make the bug not fatal)
from firmware.
Yes sure, here are two logs, I think only 2c.txt exhibits the issue, 44.txt is its partner node, there is also a backtrace in there, I don't know if that is related?
logs.zip
from firmware.
yeah - that was super helpful. Two interesting things:
I see rxBad going up, which should almost never happen (I have very few packets with bad crc received on my test bench) and twice it screwed up and tried to send when we were already sending. I have a bug that needs fixing and will work on this later today.
for my reference the relevant bits are:
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480528, lon=-0.616718, alt=52
got gps notify
Sending position to mesh
Update DB node 0x2c for variant 1, rx_time=1585144143
Providing time to mesh 1585144143
immedate send on mesh (txGood=2,rxGood=0,rxBad=1)
fix type 3
new gps pos lat=51.480466, lon=-0.616749, alt=58
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144173
ERROR! Bug! Tx packet took too long to send, forcing radio into rx modefix type 3
new gps pos lat=51.480512, lon=-0.616759, alt=61
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144203
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144223
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144373
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144403
fix type 3
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144443
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144463
fix type 3
fix type 3
new gps pos lat=51.480459, lon=-0.616718, alt=60
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144503
fix type 3
fix type 3
new gps pos lat=51.480403, lon=-0.616723, alt=50
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144543
fix type 3
new gps pos lat=51.480540, lon=-0.616819, alt=57
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144573
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144583
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
fix type 3
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144733
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144793
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144833
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144873
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144913
Transition powerFSM transition=Bluetooth timeout, from=DARK to=LS
lsEnter begin, ls_secs=3600
canSleep, mode=4, isRx=0, txEmpty=1, txGood=2
lsEnter end
lsIdle begin ls_secs=3600
wakeCause 7
Trigger powerFSM 2
Transition powerFSM transition=Wake timer, from=LS to=DARK
Looking for GPS lock
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144953
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Update DB node 0x2c for variant 1, rx_time=1585144993
fix type 3
fix type 3
fix type 3
fix type 3
new gps pos lat=51.480525, lon=-0.616834, alt=56
got gps notify
Sending position to mesh
Update DB node 0x2c for variant 1, rx_time=1585145053
Providing time to mesh 1585145053
immedate send on mesh (txGood=2,rxGood=0,rxBad=2)
assertion "!sendingPacket" failed: file "src/CustomRF95.cpp", line 155, function: void CustomRF95::startSend(MeshPacket*)
abort() was called at PC 0x4010362f on core 1
Backtrace: 0x40094cb4:0x3ffd8f80 0x40094ee5:0x3ffd8fa0 0x4010362f:0x3ffd8fc0 0x400d2686:0x3ffd8ff0 0x400d2789:0x3ffd9010 0x400d3d51:0x3ffd9040 0x400d3ed1:0x3ffd9070 0x400d4469:0x3ffd9090 0x400d4492:0x3ffd90b0 0x400d2ba9:0x3ffd90d0 0x400d4d63:0x3ffd9130 0x400d29b9:0x3ffd9160 0x400d5993:0x3ffd9180 0x400e78ad:0x3ffd91b0 0x400913c9:0x3ffd91d0
Rebooting...
ets Jun 8 2016 00:22:57
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
from firmware.
actually - looking more, only screwed up once. My little emergency txwatchdog was forcing the radio into rxmode correctly, but not clearing the flag that said we were busy sending.
Hmm - I had two nodes running all night ping ponging to each other that didn't show this problem. hmm... still looking.
from firmware.
also, I see you were running a local build. I just noticed that platform IO doesn't pull down the latest git dependencies unless you do "platformio update", so you might not have had my fixes in the RadioHead library. But I'll still think about this problem later today.
from firmware.
I have updated radiohead, it was behind:
Updating RadioHead @ 57ac37e [0b279d3]
I have made some changes to my local copy for #48 and updated that issue. I have those changes and updated RadioHead so I will leave it logging again. I am just leaving them on all the time and periodically waking with button and send a msg from app
from firmware.
@spattinson btw: I just reproduced the original problem on one of my boards. Your your 0x2c board that showed the problem, did that board have a battery installed? On my board which can show the problem - it occurs 100% of the time if I don't have a battery installed, 0% of the time if I do have one installed. I'm beginning to suspect there is something marginal in the power rail that powers the LORA radio and with no battery, if the USB power isn't up-to-snuff the radio never sets TX complete.
from firmware.
All my testing is being done with batteries, I basically leave them on all the time now, and plugged in. I have not witnessed the problem recently, but I have not been watching closely either, I am mostly tee'ing output to logs and there are no reboots in them.
from firmware.
ok sounds good! Sounds like bug probably squashed. Thanks!
from firmware.
Related Issues (20)
- [Bug]: Imported docker image error
- [Feature Request]: Include device "role" in MQTT when receiving a packet on NODEINFO_APP portnum HOT 1
- [Feature Request]: temperature offset HOT 1
- [Feature Request]: Allow Battery ADC Pin override
- [Feature Request]: (very) stable router firmware HOT 1
- [Feature Request]: admin channel: setting location
- [Bug]: more conservative data transmission HOT 1
- Critical fault #11[Bug]:
- [Bug]: meshtastic exited with code 134
- [Bug]: Unreasonably high SNRs on LR1110-based board HOT 1
- [Bug]: MQTT Server Address Length HOT 5
- [Feature Request]: Network Status Widget Data Over the Phone API HOT 1
- [Bug]: Meshtastic: No GPS HOT 4
- [Bug]: Issues with GPIO Pins and SX1262 Radio Initialization HOT 3
- Meshtastic node, connected to pc via usb, disturbing the mouse pointer like crazy
- [Bug]: Board RAK4630 factory resets when updating settings from Android App, but works fine with CLI HOT 3
- Fancy Maps HOT 1
- [Bug]: "emoji" short names crashing Heltec V3 in "client" mode HOT 6
- [Bug]: 2.4.0 Rak4630 with BME-680 HOT 1
- [Feature Request]: Turn off Bluetooth on Standalone devices by pressing Fn+b
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from firmware.