Ath10k triggers stack trace on 3.10.8


#1

Hi,

I’ve discovered that last night a stack trace popped up in the kernel log, I guess the trigger was the ath10k driver. Snippet from the beginning:

2018-12-02 21:03:54 warning kernel[]: [3057642.846467] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.161-0a333a8e606ab056173befac424900d2-1 #1
2018-12-02 21:03:54 warning kernel[]: [3057642.846474] Hardware name: Marvell Armada 380/385 (Device Tree)
2018-12-02 21:03:54 warning kernel[]: [3057642.846480] Backtrace:
2018-12-02 21:03:54 warning kernel[]: [3057642.846496] [<c001c4c4>] (dump_backtrace) from [<c001c704>] (show_stack+0x18/0x1c)
2018-12-02 21:03:54 warning kernel[]: [3057642.846503]  r6:00000000 r5:60000113 r4:c073ca68 r3:00000000

As far as I could see, it’s the same as in the frame in @Michelle’s post in [random reboots] Turris Omnia - po update na 3.10.6 (nebo 7) každých cca 20 minut samovolný restart, though I don’t speak Czech, so can’t tell for sure.

Otherwise the router is seemingly not affected, uptime is currently around 35 days.


#2

Ath10k is trying to load multiple matching firmwares before it succeeds. Every invalid firmware triggers stacktrace. You posted pretty short part so I am not sure if this is the case but if it is during boot or during ath10k firmware reload then it is ok.


#3

What could trigger a firmware reload? It happened 2 days ago around 9 at night, CET. It definitely was not at boot time - it’s been up for 37 days by now.

Here’s the complete snippet:

2018-12-02 21:03:54 warning kernel[]: [3057642.845925] ------------[ cut here ]------------
2018-12-02 21:03:54 warning kernel[]: [3057642.846015] WARNING: CPU: 0 PID: 0 at /home/beast/beast/workspace/omnia-stable/build_dir/target-arm_cortex-a9+vfpv3_musl-1.1.15_eabi/linux-mvebu/compat-wireless-2017-01-31/drivers/net/wireless/ath/ath10k/htt_rx.c:1006 ath10k_htt_t2h_msg_handler+0x14fc/0x17ac [ath10k_core]()
2018-12-02 21:03:54 warning kernel[]: [3057642.846023] Modules linked in: pl2303 qcserial option iptable_nat ip6table_nat ath9k usb_wwan rndis_host qmi_wwan pppoe nf_nat_pptp nf_nat_ipv6 nf_nat_ipv4 nf_nat_amanda nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_ipv6 nf_conntrack_ipv4 nf_conntrack_amanda ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cdc_ether ath9k_common armada_thermal xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_id xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TPROXY xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY usbserial usbnet ts_kmp ts_fsm ts_bm thermal_sys pppox ppp_mppe ppp_async nfnetlink nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_nat_h323 nf_nat_ftp nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast mvsdio iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables hwmon ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt cdc_wdm ath9k_hw sch_cake sch_teql sch_tbf sch_sfq sch_red sch_prio sch_pie sch_netem sch_htb sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_vlan act_police act_pedit act_nat act_ipt act_gact act_csum act_bpf act_connmark act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress ath10k_pci ath10k_core ath mac80211 cfg80211 compat ledtrig_usbdev ledtrig_oneshot xt_LED ledtrig_morse ledtrig_heartbeat ledtrig_gpio cryptodev ip6t_NPT ip6t_MASQUERADE nf_nat_masquerade_ipv6 nf_nat nf_conntrack ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables pppoatm ppp_generic slhc ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel veth tun snd_compress snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_rawmidi snd_seq_device snd_hwdep snd input_core soundcore br2684 atm algif_skcipher algif_hash af_alg sha1_generic marvell_cesa ecb des_generic authenc xhci_plat_hcd xhci_pci xhci_hcd uhci_hcd ledtrig_transient usbcore nls_base usb_common mii aead crypto_null
2018-12-02 21:03:54 warning kernel[]: [3057642.846467] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.161-0a333a8e606ab056173befac424900d2-1 #1
2018-12-02 21:03:54 warning kernel[]: [3057642.846474] Hardware name: Marvell Armada 380/385 (Device Tree)
2018-12-02 21:03:54 warning kernel[]: [3057642.846480] Backtrace:
2018-12-02 21:03:54 warning kernel[]: [3057642.846496] [<c001c4c4>] (dump_backtrace) from [<c001c704>] (show_stack+0x18/0x1c)
2018-12-02 21:03:54 warning kernel[]: [3057642.846503]  r6:00000000 r5:60000113 r4:c073ca68 r3:00000000
2018-12-02 21:03:54 warning kernel[]: [3057642.846526] [<c001c6ec>] (show_stack) from [<c02fca94>] (dump_stack+0x98/0xac)
2018-12-02 21:03:54 warning kernel[]: [3057642.846542] [<c02fc9fc>] (dump_stack) from [<c002921c>] (warn_slowpath_common+0x8c/0xbc)
2018-12-02 21:03:54 warning kernel[]: [3057642.846547]  r6:000003ee r5:bf339e44 r4:00000000 r3:00000006
2018-12-02 21:03:54 warning kernel[]: [3057642.846567] [<c0029190>] (warn_slowpath_common) from [<c00292f0>] (warn_slowpath_null+0x24/0x2c)
2018-12-02 21:03:54 warning kernel[]: [3057642.846573]  r8:34b17550 r7:0000ece5 r6:00000010 r5:ee7d15e0 r4:ee6149c0
2018-12-02 21:03:54 warning kernel[]: [3057642.846624] [<c00292cc>] (warn_slowpath_null) from [<bf339e44>] (ath10k_htt_t2h_msg_handler+0x14fc/0x17ac [ath10k_core])
2018-12-02 21:03:54 warning kernel[]: [3057642.846683] [<bf339784>] (ath10k_htt_t2h_msg_handler [ath10k_core]) from [<bf33ad70>] (ath10k_htt_txrx_compl_task+0xc7c/0x12a0 [ath10k_core])
2018-12-02 21:03:54 warning kernel[]: [3057642.846690]  r10:ee7d15e0 r9:c072bd70 r8:00000000 r7:c0732058 r6:00000006 r5:c0732058
2018-12-02 21:03:54 warning kernel[]: [3057642.846706]  r4:ee614840
2018-12-02 21:03:54 warning kernel[]: [3057642.846752] [<bf33a0f4>] (ath10k_htt_txrx_compl_task [ath10k_core]) from [<bf37baa4>] (ath10k_pci_napi_poll+0x50/0x108 [ath10k_pci])
2018-12-02 21:03:54 warning kernel[]: [3057642.846758]  r10:1239744d r9:00000000 r8:c072be08 r7:00000040 r6:ee7d1820 r5:ee7d15e0
2018-12-02 21:03:54 warning kernel[]: [3057642.846774]  r4:ee7d5220
2018-12-02 21:03:54 warning kernel[]: [3057642.846800] [<bf37ba54>] (ath10k_pci_napi_poll [ath10k_pci]) from [<c0481134>] (net_rx_action+0x13c/0x308)
2018-12-02 21:03:54 warning kernel[]: [3057642.846805]  r9:c072c100 r8:c072be08 r7:0000012c r6:00000040 r5:00000001 r4:ee7d5220
2018-12-02 21:03:54 warning kernel[]: [3057642.846830] [<c0480ff8>] (net_rx_action) from [<c002c73c>] (__do_softirq+0x114/0x260)
2018-12-02 21:03:54 warning kernel[]: [3057642.846836]  r10:00000000 r9:00000003 r8:00000100 r7:c072a000 r6:40000003 r5:c072c080
2018-12-02 21:03:54 warning kernel[]: [3057642.846851]  r4:c072c08c
2018-12-02 21:03:54 warning kernel[]: [3057642.846864] [<c002c628>] (__do_softirq) from [<c002cb14>] (irq_exit+0x80/0xb8)
2018-12-02 21:03:54 warning kernel[]: [3057642.846869]  r10:c072bf80 r9:f0803100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000
2018-12-02 21:03:54 warning kernel[]: [3057642.846884]  r4:c0724ff0
2018-12-02 21:03:54 warning kernel[]: [3057642.846898] [<c002ca94>] (irq_exit) from [<c0068850>] (__handle_domain_irq+0x68/0xb8)
2018-12-02 21:03:54 warning kernel[]: [3057642.846903]  r4:c0724ff0 r3:f0802100
2018-12-02 21:03:54 warning kernel[]: [3057642.846917] [<c00687e8>] (__handle_domain_irq) from [<c0009550>] (gic_handle_irq+0x50/0x94)
2018-12-02 21:03:54 warning kernel[]: [3057642.846922]  r8:c072bf10 r7:f080210c r6:c072d030 r5:c073cebc r4:f0802100 r3:c072bf10
2018-12-02 21:03:54 warning kernel[]: [3057642.846944] [<c0009500>] (gic_handle_irq) from [<c000a694>] (__irq_svc+0x54/0x70)
2018-12-02 21:03:54 warning kernel[]: [3057642.846951] Exception stack(0xc072bf10 to 0xc072bf58)
2018-12-02 21:03:54 warning kernel[]: [3057642.846960] bf00:                                     00000001 00000000 c072bf70 c000b520
2018-12-02 21:03:54 warning kernel[]: [3057642.846972] bf20: c072a000 c072c498 c061ebfc 00000000 00000000 c0727118 c072bf80 c072bf6c
2018-12-02 21:03:54 warning kernel[]: [3057642.846982] bf40: c072bf70 c072bf60 c00192b4 c00192b8 60000013 ffffffff
2018-12-02 21:03:54 warning kernel[]: [3057642.846987]  r9:c0727118 r8:00000000 r7:c072bf44 r6:ffffffff r5:60000013 r4:c00192b8
2018-12-02 21:03:54 warning kernel[]: [3057642.847014] [<c0019278>] (arch_cpu_idle) from [<c0060358>] (default_idle_call+0x28/0x34)
2018-12-02 21:03:54 warning kernel[]: [3057642.847028] [<c0060330>] (default_idle_call) from [<c00604b0>] (cpu_startup_entry+0x14c/0x22c)
2018-12-02 21:03:54 warning kernel[]: [3057642.847039] [<c0060364>] (cpu_startup_entry) from [<c000eca8>] (rest_init+0x80/0x84)
2018-12-02 21:03:54 warning kernel[]: [3057642.847044]  r7:c072c3c0
2018-12-02 21:03:54 warning kernel[]: [3057642.847056] [<c000ec28>] (rest_init) from [<c06e1de0>] (start_kernel+0x460/0x46c)
2018-12-02 21:03:54 warning kernel[]: [3057642.847066] [<c06e1980>] (start_kernel) from [<0000807c>] (0x807c)
2018-12-02 21:03:54 warning kernel[]: [3057642.847073] ---[ end trace bce1ac2b6009db1b ]---

#4

This is known issue in ath10k and was fixed in the newer kernel than omnia uses.
You can also reproduce it by heavy WiFi traffic load.

See my original post here.


#5

Thanks for the heads up, I hope the newer kernel sooner or later reaches Turris OS.