While Ath10k crashed & recovered, clients weren't disconnected, but connectivity is dead

Hi there,

This has been happening in increasing frequency over the past 2 to 3 weeks:

  1. Everything working fine;
  2. Then suddenly, any devices connected to WLAN of Ath10k stopped being able to access the Internet, or any resources on LAN;
  3. A check on the system log reveals that ath10k had crashed, and recovered
  4. However, none of the clients connected to it knew what happened. The only way to reestablish connectivity is to either: restart the router (or restart ath10k), or restart the client (or its wifi).

What is happening?

Any advice/pointers appreciated.

Thanks!

Aug 11 20:29:07 node00 kernel: [ 6783.132480] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.183685] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 1, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.234876] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.286073] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 1, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.337269] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.388463] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 1, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.439662] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.490859] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 1, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.542055] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 0, skipped old beacon
Aug 11 20:29:07 node00 kernel: [ 6783.593254] ath10k_pci 0000:02:00.0: SWBA overrun on vdev 1, skipped old beacon
Aug 11 12:29:10 node00 hostapd: wlan0: AP-STA-DISCONNECTED dc:53:60:2d:c3:96
Aug 11 20:29:10 node00 kernel: [ 6786.140634] ieee80211 phy0: Hardware restart was requested
Aug 11 20:29:10 node00 kernel: [ 6786.146671] ------------[ cut here ]------------
Aug 11 20:29:10 node00 kernel: [ 6786.151445] WARNING: CPU: 0 PID: 8486 at backports-4.19.120-1/net/mac80211/sta_info.c:1002 sta_set_sinfo+0xbd4/0xcf0 [mac80211]
Aug 11 20:29:10 node00 kernel: [ 6786.162978] Modules linked in: sch_cake ath9k ath9k_common qcserial pppoe ppp_async ath9k_hw ath10k_pci ath10k_core ath uvcvideo usb_wwan pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cfg80211 at24 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NFQUEUE xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY videobuf2_v4l2 usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc nvmem_core nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4
Aug 11 20:29:10 node00 kernel: [ 6786.233934] nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_ftp nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack_ftp nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables gspca_sq905 gspca_main ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat br_netfilter fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred configs videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_heartbeat ledtrig_gpio cryptodev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface
Aug 11 20:29:10 node00 kernel: [ 6786.306440] ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs msdos ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel rpcsec_gss_krb5 auth_rpcgss oid_registry tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd_hwdep snd_compress snd soundcore ntfs lockd sunrpc grace hfsplus hfs cifs autofs4 dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax multipath raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid10 raid1 raid0 linear md_mod nls_utf8 nls_koi8_r nls_cp1255 nls_iso8859_6
Aug 11 20:29:10 node00 kernel: [ 6786.377369] nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_cp932 nls_cp866 nls_cp864 nls_cp862 nls_cp852 nls_cp850 nls_cp775 nls_cp1251 nls_cp1250 xts algif_skcipher algif_hash af_alg sha512_generic sha256_generic seqiv jitterentropy_rng drbg pcbc md5 md4 hmac gf128mul ecb cts ctr cmac ccm cbc authenc vfat fat nls_iso8859_1 nls_cp437 uas ledtrig_transient ahci xfs libcrc32c exfat
Aug 11 20:29:10 node00 kernel: [ 6786.411348] CPU: 0 PID: 8486 Comm: hostapd Not tainted 4.14.187 #0
Aug 11 20:29:10 node00 kernel: [ 6786.417542] Hardware name: Marvell Armada 380/385 (Device Tree)
Aug 11 20:29:10 node00 kernel: [ 6786.423485] [] (unwind_backtrace) from [] (show_stack+0x10/0x14)
Aug 11 20:29:10 node00 kernel: [ 6786.431249] [] (show_stack) from [] (dump_stack+0x94/0xa8)
Aug 11 20:29:10 node00 kernel: [ 6786.438495] [] (dump_stack) from [] (__warn+0xe4/0x100)
Aug 11 20:29:10 node00 kernel: [ 6786.445474] [] (__warn) from [] (warn_slowpath_null+0x20/0x28)
Aug 11 20:29:10 node00 kernel: [ 6786.453113] [] (warn_slowpath_null) from [] (sta_set_sinfo+0xbd4/0xcf0 [mac80211])
Aug 11 20:29:10 node00 kernel: [ 6786.462558] [] (sta_set_sinfo [mac80211]) from [] (__sta_info_destroy+0x20/0x28 [mac80211])
Aug 11 20:29:10 node00 kernel: [ 6786.472727] [] (__sta_info_destroy [mac80211]) from [] (sta_info_destroy_addr_bss+0x28/0x40 [mac80211])
Aug 11 20:29:10 node00 kernel: [ 6786.483945] [] (sta_info_destroy_addr_bss [mac80211]) from [] (nl80211_del_station+0xf8/0x11c [cfg80211])
Aug 11 20:29:10 node00 kernel: [ 6786.495326] [] (nl80211_del_station [cfg80211]) from [] (genl_rcv_msg+0x31c/0x3e0)
Aug 11 20:29:10 node00 kernel: [ 6786.504656] [] (genl_rcv_msg) from [] (netlink_rcv_skb+0x8c/0xfc)
Aug 11 20:29:10 node00 kernel: [ 6786.512504] [] (netlink_rcv_skb) from [] (genl_rcv+0x24/0x34)
Aug 11 20:29:10 node00 kernel: [ 6786.520005] [] (genl_rcv) from [] (netlink_unicast+0x15c/0x22c)
Aug 11 20:29:10 node00 kernel: [ 6786.527680] [] (netlink_unicast) from [] (netlink_sendmsg+0x378/0x3d4)
Aug 11 20:29:10 node00 kernel: [ 6786.535964] [] (netlink_sendmsg) from [] (___sys_sendmsg+0x21c/0x250)
Aug 11 20:29:10 node00 kernel: [ 6786.544161] [] (___sys_sendmsg) from [] (__sys_sendmsg+0x50/0x88)
Aug 11 20:29:10 node00 kernel: [ 6786.552014] [] (__sys_sendmsg) from [] (ret_fast_syscall+0x0/0x54)
Aug 11 20:29:10 node00 kernel: [ 6786.560194] —[ end trace e38a4fc3e0ddbb58 ]—
Aug 11 20:29:10 node00 kernel: [ 6786.564902] ------------[ cut here ]------------
Aug 11 20:29:10 node00 kernel: [ 6786.569618] WARNING: CPU: 0 PID: 8486 at backports-4.19.120-1/net/mac80211/sta_info.c:1010 sta_set_sinfo+0xc48/0xcf0 [mac80211]
Aug 11 20:29:11 node00 kernel: [ 6786.581157] Modules linked in: sch_cake ath9k ath9k_common qcserial pppoe ppp_async ath9k_hw ath10k_pci ath10k_core ath uvcvideo usb_wwan pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cfg80211 at24 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NFQUEUE xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY videobuf2_v4l2 usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc nvmem_core nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4
Aug 11 20:29:11 node00 kernel: [ 6786.652229] nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_ftp nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack_ftp nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables gspca_sq905 gspca_main ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat br_netfilter fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred configs videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_heartbeat ledtrig_gpio cryptodev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface
Aug 11 20:29:11 node00 kernel: [ 6786.725158] ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs msdos ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel rpcsec_gss_krb5 auth_rpcgss oid_registry tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd_hwdep snd_compress snd soundcore ntfs lockd sunrpc grace hfsplus hfs cifs autofs4 dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax multipath raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid10 raid1 raid0 linear md_mod nls_utf8 nls_koi8_r nls_cp1255 nls_iso8859_6
Aug 11 20:29:11 node00 kernel: [ 6786.796079] nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_cp932 nls_cp866 nls_cp864 nls_cp862 nls_cp852 nls_cp850 nls_cp775 nls_cp1251 nls_cp1250 xts algif_skcipher algif_hash af_alg sha512_generic sha256_generic seqiv jitterentropy_rng drbg pcbc md5 md4 hmac gf128mul ecb cts ctr cmac ccm cbc authenc vfat fat nls_iso8859_1 nls_cp437 uas ledtrig_transient ahci xfs libcrc32c exfat
Aug 11 20:29:11 node00 kernel: [ 6786.830043] CPU: 0 PID: 8486 Comm: hostapd Tainted: G W 4.14.187 #0
Aug 11 20:29:11 node00 kernel: [ 6786.837457] Hardware name: Marvell Armada 380/385 (Device Tree)
Aug 11 20:29:11 node00 kernel: [ 6786.843401] [] (unwind_backtrace) from [] (show_stack+0x10/0x14)
Aug 11 20:29:11 node00 kernel: [ 6786.851165] [] (show_stack) from [] (dump_stack+0x94/0xa8)
Aug 11 20:29:11 node00 kernel: [ 6786.858410] [] (dump_stack) from [] (__warn+0xe4/0x100)
Aug 11 20:29:11 node00 kernel: [ 6786.865390] [] (__warn) from [] (warn_slowpath_null+0x20/0x28)
Aug 11 20:29:11 node00 kernel: [ 6786.873036] [] (warn_slowpath_null) from [] (sta_set_sinfo+0xc48/0xcf0 [mac80211])
Aug 11 20:29:11 node00 kernel: [ 6786.882421] [] (sta_set_sinfo [mac80211]) from [] (__sta_info_destroy+0x20/0x28 [mac80211])
Aug 11 20:29:11 node00 kernel: [ 6786.892589] [] (__sta_info_destroy [mac80211]) from [] (sta_info_destroy_addr_bss+0x28/0x40 [mac80211])
Aug 11 20:29:11 node00 kernel: [ 6786.903806] [] (sta_info_destroy_addr_bss [mac80211]) from [] (nl80211_del_station+0xf8/0x11c [cfg80211])
Aug 11 20:29:11 node00 kernel: [ 6786.915161] [] (nl80211_del_station [cfg80211]) from [] (genl_rcv_msg+0x31c/0x3e0)
Aug 11 20:29:11 node00 kernel: [ 6786.924490] [] (genl_rcv_msg) from [] (netlink_rcv_skb+0x8c/0xfc)
Aug 11 20:29:11 node00 kernel: [ 6786.932339] [] (netlink_rcv_skb) from [] (genl_rcv+0x24/0x34)
Aug 11 20:29:11 node00 kernel: [ 6786.939838] [] (genl_rcv) from [] (netlink_unicast+0x15c/0x22c)
Aug 11 20:29:11 node00 kernel: [ 6786.947513] [] (netlink_unicast) from [] (netlink_sendmsg+0x378/0x3d4)
Aug 11 20:29:11 node00 kernel: [ 6786.955797] [] (netlink_sendmsg) from [] (___sys_sendmsg+0x21c/0x250)
Aug 11 20:29:11 node00 kernel: [ 6786.963994] [] (___sys_sendmsg) from [] (__sys_sendmsg+0x50/0x88)
Aug 11 20:29:11 node00 kernel: [ 6786.971846] [] (__sys_sendmsg) from [] (ret_fast_syscall+0x0/0x54)
Aug 11 20:29:11 node00 kernel: [ 6786.980120] —[ end trace e38a4fc3e0ddbb59 ]—
Aug 11 20:29:12 node00 kernel: [ 6788.311409] ath10k_warn: 55 callbacks suppressed
Aug 11 20:29:12 node00 kernel: [ 6788.311451] ath10k_pci 0000:02:00.0: pdev param 0 not supported by firmware
Aug 11 20:29:13 node00 kernel: [ 6789.439098] ath10k_pci 0000:02:00.0: device successfully recovered

1 Like