bpi-r3 branch 6.3-rc missing SFP support / transmit queue timeout about bpi-router-linux HOT 38 OPEN

which SFP do you use? which issue do you have with this? does sfp appear in dmesg? what does ethtool say?

there are several patches for sfp-support (mostly quirks for 2.5G rj45 sfp)

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/log/drivers/net/phy/sfp.c

network-stack is currently broken...but there are already fixes for it

#throughput for r3-gmac (wan sfp)
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/mediatek/mtk_eth_soc.c?id=193250ace270fecd586dd2d0dfbd9cbd2ade977f

fixes for "net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues" which i have reverted in 6.3-rc, so these should not be needed
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/mediatek/mtk_eth_soc.c?id=07b3af42d8d528374d4f42d688bae86eeb30831a

#mt7623 switch-throughput
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/drivers/net/ethernet/mediatek/mtk_eth_soc.c?id=e669ce46740a9815953bb4452a6bc5a7fdc21a50

from bpi-router-linux.

I pulled the 6.3-rc branch. compiled and booted from it and noticed the rj45 sfp stopped working. As I didn't have time to play with it and 6.2-rc has been working fine, I just went back to 6.2.

from bpi-router-linux.

which SFP do you use exactly?

c686900 2023-03-14 net: sfp: support 2.5g copper SFP v3

should be make 2g5 sfp from aliexpress work...if you have the tplink then daniel prepared a quirk-patch for it

from bpi-router-linux.

It's the tplink from aliexpress model TL-SM410U

I will give the 6.3-rc another try

from bpi-router-linux.

Ok,the tp-link needs patch from daniel i have not yet in my tree...

from bpi-router-linux.

Oh ok, got confused, do you have plans to add Daniels patch to your 6.3 tree?

from bpi-router-linux.

have added the tp-link quirk, but daniel reports that it is not clean working and they search the issue

https://www.spinics.net/lists/netdev/msg892902.html

btw. there is 6.2-main...better than rc to use

but i wonder why 6.2-rc works for you....

maybe you can try to revert the sfp-patch above...possibly it breaks your sfps

from bpi-router-linux.

Thanks for the tp-link quirk patch, the SFP is working now in 6.3-rc.

I will test the 6.2-main now, I noticed the fiber SFP is not detecting link.

from bpi-router-linux.

have added the tp-link quirk, but daniel reports that it is not clean working and they search the issue

https://www.spinics.net/lists/netdev/msg892902.html

btw. there is 6.2-main...better than rc to use

but i wonder why 6.2-rc works for you....

maybe you can try to revert the sfp-patch above...possibly it breaks your sfps

I couldn't find the 6.2-main branch

from bpi-router-linux.

which fibre sfp? mine (H!Fibre in left cage) is working in 6.3-rc

sorry about 6.2-main, had not pushed it...done now

from bpi-router-linux.

Thanks, I'll pull 6.3-main.

I tested two, I can see them but link is not detected

Brocade 57-1000013-01
Cisco GLC-SX-MM=

from bpi-router-linux.

6.2-main...6.3 is not final (sry typo from me...rc branches are till the kernel is released - when only rc are available)

which slot do you use?
what does ethtool and dmesg tell about them?

ethtool -m eth1/lan4

dmesg |grep -i 'eth\|sfp'

what happens when you do the autoneg workaround?

ethtool -s eth1 autoneg off
ethtool -s lan4 autoneg off

from bpi-router-linux.

Ok, my bad, last time I tried to make the SFP work I played too much with it and let auto negotiation disabled on the other side.. so after enabling it I can detect link and tcpdump traffic. Looks fine so far on 6.3-rc. vlan is not working, if I remember correctly left side SFP does not support vlan.

from bpi-router-linux.

Yes,there was a patch series from felix fietkau which cannot be applied...thought patch from vladimir fixed it but maybe it was another issue...felix patch disabled hw offloading on the gmac1

from bpi-router-linux.

This should be the patch fixing the vlan-issue on gmac

1382688

from bpi-router-linux.

i have pushed the vlan-fix to 6.3-pwm-branch you can cherry-pick to any other branch you want...it works so far on my bananapi-r3 and i send it as RFC/RFT to ML

https://patchwork.kernel.org/project/linux-mediatek/patch/[email protected]/

from bpi-router-linux.

I just build 6.3-pwm and booted from it and I can see the vlan tags in the packets. Amazing, thanks!

from bpi-router-linux.

I hope there will be no sideeffects...i guess the patch disables rx vlan offload completely but better than broken vlan support

from bpi-router-linux.

I've ran into this kernel panic twice now. I need to reboot to clear it. You think it's related to the a

[20795.193278] ------------[ cut here ]------------
[20795.197920] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 3 timed out
[20795.204916] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x26c/0x274
[20795.213182] Modules linked in: pps_ldisc tun nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables libcrc32c nfnetlink pps_gpio mt7915e mt76_connac_lib mt76 mac80211 libarc4 cfg80211 fuse ip_tables x_tables
[20795.234045] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-rc6-bpi-r3-pwm #3
[20795.241167] Hardware name: Bananapi BPI-R3 (DT)
[20795.245686] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[20795.252633] pc : dev_watchdog+0x26c/0x274
[20795.256638] lr : dev_watchdog+0x26c/0x274
[20795.260637] sp : ffffffc00800be00
[20795.263938] x29: ffffffc00800be00 x28: ffffffc009426000 x27: ffffffc00800bee8
[20795.271061] x26: ffffffc009106008 x25: 0000000000000000 x24: ffffffc0094288c8
[20795.278183] x23: 0000000000000100 x22: ffffffc009426000 x21: 0000000000000003
[20795.285306] x20: ffffff8001e31000 x19: ffffff8001e31488 x18: ffffffffffffffff
[20795.292427] x17: ffffffc076a73000 x16: ffffffc008008000 x15: 076e076107720774
[20795.299550] x14: 0720073a07290768 x13: 0000000000000255 x12: 00000000ffffffea
[20795.306673] x11: 00000000ffffefff x10: 00000000ffffefff x9 : ffffffc00949a438
[20795.313796] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001
[20795.320921] x5 : 0000000000000002 x4 : 0000000000000000 x3 : 0000000000000002
[20795.328046] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff8000134380
[20795.335168] Call trace:
[20795.337604]  dev_watchdog+0x26c/0x274
[20795.341262]  call_timer_fn+0x34/0x150
[20795.344917]  __run_timers.part.0+0x2f4/0x358
[20795.349177]  run_timer_softirq+0x3c/0x74
[20795.353089]  _stext+0x118/0x344
[20795.356220]  ____do_softirq+0x10/0x1c
[20795.359872]  call_on_irq_stack+0x24/0x4c
[20795.363782]  do_softirq_own_stack+0x1c/0x2c
[20795.367953]  __irq_exit_rcu+0xe0/0xfc
[20795.371606]  irq_exit_rcu+0x10/0x1c
[20795.375083]  el1_interrupt+0x38/0x54
[20795.378648]  el1h_64_irq_handler+0x18/0x24
[20795.382733]  el1h_64_irq+0x68/0x6c
[20795.386123]  default_idle_call+0x58/0x108
[20795.390121]  do_idle+0xa8/0x100
[20795.393253]  cpu_startup_entry+0x24/0x2c
[20795.397166]  secondary_start_kernel+0x128/0x14c
[20795.401686]  __secondary_switched+0xb8/0xbc
[20795.405859] ---[ end trace 0000000000000000 ]---

[ 7674.040708] ------------[ cut here ]------------
[ 7674.045340] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 8 timed out
[ 7674.052351] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x26c/0x274
[ 7674.060614] Modules linked in: pps_ldisc tun nft_masq nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables libcrc32c nfnetlink pps_gpio mt7915e mt76_connac_lib mt76 mac80211 libarc4 cfg80211 fuse ip_tables x_tables
[ 7674.081476] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.3.0-rc6-bpi-r3-pwm #3
[ 7674.088595] Hardware name: Bananapi BPI-R3 (DT)
[ 7674.093112] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 7674.100058] pc : dev_watchdog+0x26c/0x274
[ 7674.104061] lr : dev_watchdog+0x26c/0x274
[ 7674.108060] sp : ffffffc0096f3e00
[ 7674.111360] x29: ffffffc0096f3e00 x28: ffffffc009426000 x27: ffffffc0096f3ee8
[ 7674.118483] x26: ffffffc009106008 x25: 0000000000000000 x24: ffffffc0094288c8
[ 7674.125605] x23: 0000000000000100 x22: ffffffc009426000 x21: 0000000000000008
[ 7674.132726] x20: ffffff8000324000 x19: ffffff8000324488 x18: ffffffffffffffff
[ 7674.139847] x17: ffffffc076aab000 x16: ffffffc0096f0000 x15: ffffffc0896f3a47
[ 7674.146968] x14: 0000000000000000 x13: 00000000000001e8 x12: 00000000ffffffea
[ 7674.154089] x11: 00000000ffffefff x10: 00000000ffffefff x9 : ffffffc00949a438
[ 7674.161209] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : 0000000000000001
[ 7674.168330] x5 : 0000000000000000 x4 : ffffff807fbb2ac8 x3 : ffffff807fbbe7b0
[ 7674.175450] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff8000135e80
[ 7674.182572] Call trace:
[ 7674.185007]  dev_watchdog+0x26c/0x274
[ 7674.188660]  call_timer_fn+0x34/0x150
[ 7674.192315]  __run_timers.part.0+0x2f4/0x358
[ 7674.196574]  run_timer_softirq+0x3c/0x74
[ 7674.200485]  _stext+0x118/0x344
[ 7674.203616]  ____do_softirq+0x10/0x1c
[ 7674.207266]  call_on_irq_stack+0x24/0x4c
[ 7674.211178]  do_softirq_own_stack+0x1c/0x2c
[ 7674.215348]  __irq_exit_rcu+0xe0/0xfc
[ 7674.219001]  irq_exit_rcu+0x10/0x1c
[ 7674.222478]  el1_interrupt+0x38/0x54
[ 7674.226043]  el1h_64_irq_handler+0x18/0x24
[ 7674.230127]  el1h_64_irq+0x68/0x6c
[ 7674.233517]  default_idle_call+0x58/0x108
[ 7674.237514]  do_idle+0xa8/0x100
[ 7674.240647]  cpu_startup_entry+0x24/0x2c
[ 7674.244558]  secondary_start_kernel+0x128/0x14c
[ 7674.249078]  __secondary_switched+0xb8/0xbc
[ 7674.253252] ---[ end trace 0000000000000000 ]---

from bpi-router-linux.

I think this is related to felix patch adding the transmi queues...but i have no idea how to fix...there is a issue on openwrt github repo

from bpi-router-linux.

Do you know a way to trigger this bug?

This is the issue on openwrt repo where i sent an update: openwrt/openwrt#12143 (comment)

If we know how to trigger it we have a chance to debug it to root cause

from bpi-router-linux.

Let issue open for tracing the transmit error

from bpi-router-linux.

I don't, but it hasn't crashed today and yesterday when it crashed I was copying big files through SFP1/SFP2.

I'll run iperf3 test and see if it crashes again

from bpi-router-linux.

I have not seen it with iperf3 yet...thats my standard-test in both directions,but i guess the problem is when there are multiple streams which are spreaded over different tx queues over time

from bpi-router-linux.

Have you seen this with 6.2 too? If not you can try the vlan-patch there and look if this is the cause but imho this cannot be the cause as this is not in tx path

from bpi-router-linux.

I havent tested with 6.2, can I just checkout latest 6.2-main and build it?

I ran into the issue again yesterday and I think the issue has something to do with the right SFP, where I have the copper tplink SFP.

I ran into it three times when traffic is flowing as shown below.

              ------------------>
       eth1                           lan4
------------------------------------------
|   fiber/vlan   |   copper/untagged     |
------------------------------------------

from bpi-router-linux.

Is this reproducable for you (e.g. you can trigger it this way)?

Is the other end also 2.5g or do you run at lower speed there?

Could you try with another sfp (maybe only 1g) in the lan-sfp cage or towards one of the rj45 ports?

Basicly you can use every branch to build :) vlanfix is only applied to the last one (6.3-pwm) but for testing the sfp itself other branches can work too,but maybe need patches for supporting the tplink-sfp

from bpi-router-linux.

I can trigger it but unsure exactly how.. it just happens. I noticed it doesn't happen when I just stop using the right sfp2/lan4

Both ends on both SFP's are 2.5g

I tried a copper 1g SFP on the right sfp2/lan4 and it wont take it seems like it only allows 2.5g modules

root@bpi-r3:~# dmesg
[ 2265.225821] sfp sfp-2: module AVAGO            ABCU-5700RZ      rev      sn AN1001A2CX       dc 100107  
[ 2286.381969] mt7530 mdio-bus:1f lan4: configuring for inband/2500base-x link mode
[ 2286.454562] mt7530 mdio-bus:1f lan4: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[ 2286.464391] sfp sfp-2: sfp_add_phy failed: -EINVAL
[ 2315.659816] mt7530 mdio-bus:1f lan4: configuring for inband/2500base-x link mode
[ 2315.798868] mt7530 mdio-bus:1f lan4: validation with support 00,00000000,00000000,00000000 failed: -EINVAL
[ 2315.808653] sfp sfp-2: sfp_add_phy failed: -EINVAL
root@bpi-r3:~# ethtool lan4
Settings for lan4:
	Supported ports: [ MII ]
	Supported link modes:   2500baseX/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: No
	Supported FEC modes: Not reported
	Advertised link modes:  2500baseX/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: No
	Advertised FEC modes: Not reported
	Speed: Unknown!
	Duplex: Unknown! (255)
	Auto-negotiation: off
	Port: MII
	PHYAD: 0
	Transceiver: internal
	Supports Wake-on: d
	Wake-on: d
	Link detected: no

from bpi-router-linux.

My fiber sfp works in both cages and this is 1g only ..maybe your 1g module needs a quirk too...daniel reported from some sfps having checksum errors or having invalid data (e.g. supported modes) in eeprom

from bpi-router-linux.

Can you trigger the issue when only working over lan4/sfp2 (writing/reading to/from r3).

from bpi-router-linux.

Can you trigger the issue when only working over lan4/sfp2 (writing/reading to/from r3).

I took a while to reply because I was testing. The good news is that I haven't ran into this issue again. I have both SFP's populated with the same modules and same traffic going through them.

The only thing that changed is back then I was doing a lot of NTP/GPS/PPS testing, maybe the kernel panic had something to do with it? I can't think of anything else.

from bpi-router-linux.

I loaded 6.3.0-bpi-r3-main 24 hours ago and no crashes. When I initially ran into this issue it could crash after couple hours.

from bpi-router-linux.

Have you same connections as before (sfps,rj45 ports,vlan)? Have not seen any fix for these problems

from bpi-router-linux.

Something definitely changed, but I can't tell what.
The SFP's and the cages where they are inserted hasn't changed.
It hasn't crashed since I reported the initial crash so I'm happy

from bpi-router-linux.