RX2 fallback leads to endless downlinks about ttn HOT 15 CLOSED

This is an issue for https://github.com/TheThingsNetwork/ttn, not https://github.com/TheThingsNetwork/lorawan-stack. I'll transfer it.

from ttn.

Looking at the fallback code, the line that does the fallback magic is:
loraSettings.DataRate = "SF12BW125"

Can't we change that line to:

if loraSettings.DataRate == "SF9BW125" {
    loraSettings.DataRate = "SF12BW125"
} else {
    loraSettings.DataRate = "SF9BW125"
}

With this change (if I understand the code correctly) we will alternate between the incorrect (SF12) downlink and the correct (SF9) downlink.

from ttn.

I'm still stuck with one deployment because of this bug.

Steps to reproduce:

• Only one gateway should be in reach of the device
• Make the device join using OTAA and make sure ADR is switched on
• Over night switch off the gateway
• The next day switch the gateway back on
• The gateway will receive the device on SF12 and downlinks will be sent (ADR, ACK)
• Downlinks are ignored/not received by the device

I can't find a workaround to get the device to receive the downlinks again other than to power cycle it to force a re-join. Is there any other way I can force downlinks back to SF9?

from ttn.

Thanks for contributing 😄
I'm going to investigate the functionality of this fix the next couple of days with our test devices.
Restarting our production devices afterwards.

from ttn.

I just realised this fix will only be applied to devices that join after the fix is applied. Devices that are using a session from a join before today will still suffer from this bug. On my side that means half of my devices (that automatically rejoins periodically) are now working as expected, while the other half (which never performs a re-join) will still not receive downlinks.

from ttn.

It is a important bug, i have few customers which lost its devices due to this bug. I think in OTAA, there is no need to change the SF of the RX2 after the join-accept. If the device receives the join-accept then it is configured to receive in SF9 on RX2.

For ABP, I think it should be better to configure by using the RX1 window....

from ttn.

If you have problems with this in a private deployment, you can configure it to use a different RX2 data rate with the --eu-rx2-dr flag (or corresponding environment / yml config).

Since the core team is working full-time on v3, you may want to contribute a fix for this yourself.

My suggestion would be to add a "activated at" field to the device so that the NS can detect if it's an OTAA device or not. This field can then be set by HandleActivate after which the fallback can be bypassed for devices that have been activated with OTAA.

from ttn.

I don't know if everything is correct and compilable (my first contribution in GO 😀), but I've implemented the suggestions of @htdvisser.

Some things to remind: The activatedAt field is never reset to nil, I don't know if that has to be done if the device is changed over to ABP or if a new device is registered in that case. I couldn't find any source code for that case. Additionally I found a activation constraint (here) that is already identifying if the device was actived via OTAA. Maybe that can be used for the bypass, too.

Thanks so far!

from ttn.

I'm currently also affected by this. One of my devices joined using OTAA. Then we had the recent gateway outage and the device never received downlinks. The device (ADR) therefore moved all the way to SF12. Using an SDR I can see it is transmitting on SF12 and the downlinks are also sent on SF12, not on SF9 like it should be.

@sigmaroot @htdvisser how can I assist getting this bugfix tested and out on production TTN?

from ttn.

@jpmeijers
Please have a look into my commit mentioned earlier: edbe65d

from ttn.

I think there are two different discussions here. One is regarding privately hosted work, another is regarding existing apps deployed using TTN. As I don't have a privately-hosted V2 network, changes for private networks are interesting but don't address our issues. @htdvisser Will pushing a change request on the V2 code make it to production, or is TTN V2 basically frozen at this point, or is there some other way to fix it? Those of us with deployed systems need to understand so we can explain to our stakeholders. It would be good to avoid visits to the devices, and better to avoid firmware updates in devices; but it would be great, no matter how we mitigate, to make sure we only visit devices once.

Thanks!

from ttn.

As you can imagine, we're trying to spend all our time on v3 and only do really critical things on v2. Of course we do welcome contributions, so if you're willing to implement a fix and test it on a private network, I'll make sure it gets merged and deployed to the public community network.

from ttn.

I see this as a critical issue as it causes some of my devices to be stuck on SF12, sending retries and draining their batteries. It also prevents me from having remote control of these devices.

Because of this I tried the suggestions in this thread, implemented them on a private instance of the stack on my local computer, and tested. PR #768 contains a working fix.

from ttn.

from ttn.

As ADR-enabled OTAA devices that joined before October 2019, and all ADR-enabled ABP devices are still running into this^†, I wonder if it would be worthwhile to fix this for those in V2 too?

ONLY IF YES, then I wonder:

1. Is there any reason why ABP devices were excluded from making the fallback less persistent?

   @sigmaroot's suggested:

   The RX2 fallback should not be used more than 2 times (for example). After 2 missed answeres, the RX2 should be changed back to SF9 again. After 3 cycles of RX2/SF9-RX2/SF12 changes some notification should be generated ("end node not answering any MAC") and the downlinks should be stopped. A restart of the procedure after an amount of time may be possible.

   @htdvisser suggested what was actually implemented:

   My suggestion would be to add a "activated at" field to the device so that the NS can detect if it's an OTAA device or not. This field can then be set by HandleActivate after which the fallback can be bypassed for devices that have been activated with OTAA.

   I assume this seemed best/cleanest because OTAA devices should never need the fallback.

   @jpmeijers suggested something that would work for ABP as well (and, in retrospect, would also work for OTAA devices that joined before October 2019):

   Looking at the fallback code, the line that does the fallback magic is: loraSettings.DataRate = "SF12BW125"

   Can't we change that line to:

   if loraSettings.DataRate == "SF9BW125" {
       loraSettings.DataRate = "SF12BW125"
   } else {
       loraSettings.DataRate = "SF9BW125"
   }

   With this change (if I understand the code correctly) we will alternate between the incorrect (SF12) downlink and the correct (SF9) downlink.

2. So, what if @jpmeijers' suggestion would be added as well? Or, if that would not work, then what if the fallback would only be used for, say, odd or even values of the downlink (or uplink) counter?

^† A recent case from Slack shows an Elsys OTAA device that dropped to SF12 after some outage, and since already missed 39,000+ (still counting) RX2 downlinks on SF12, with an FOpts of 0503D2AD840345FF0001 telling it to use DR4/SF8. And that single user has about 20 such devices installed.

from ttn.