Frequent stack traces filling up /tmp/log/messages


#1

Hi,

since (I guess) about today or yesterday, there are very frequent kernel warnings which look like this:

2019-05-05 22:09:09 warning kernel[]: [ 1134.499288] ------------[ cut here ]------------
2019-05-05 22:09:09 warning kernel[]: [ 1134.499301] WARNING: CPU: 1 PID: 11 at net/sched/sch_hfsc.c:1426 hfsc_dequeue+0x34c/0x508 [sch_hfsc]()
2019-05-05 22:09:09 warning kernel[]: [ 1134.499303] Modules linked in: ifb qcserial option iptable_nat ath9k usb_wwan rndis_host qmi_wwan pppoe nf_nat_pptp nf_nat_ipv4 nf_nat_amanda nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_ipv6 nf_conntrack_ipv4 nf_conntrack_amanda ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cdc_ether ath9k_common armada_thermal xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_id xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY usbserial usbnet ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda ts_kmp ts_fsm ts_bm thermal_sys pppox ppp_mppe ppp_async nfnetlink nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntr
2019-05-05 22:09:09 notice kernel[]: nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast mvsdio iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables hwmon ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt cdc_wdm ath9k_hw fuse sch_teql sch_tbf sch_sfq sch_red sch_prio sch_pie sch_netem sch_htb sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_vlan act_police act_pedit act_nat act_ipt act_gact act_csum act_bpf act_connmark nf_conntrack act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress ath10k_pci ath10k_core ath mac80211 cfg80211 compat ledtrig_usbdev ledtrig_oneshot xt_LED ledtrig_morse ledtrig_heartbeat ledtrig_gpio cryptodev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables pppoatm ppp_generic slhc nfsd nfsv3 msdos ip_gre g
2019-05-05 22:09:09 notice kernel[]: sit ip6_tunnel tunnel6 tunnel4 ip_tunnel veth tun snd_compress snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_rawmidi snd_seq_device snd_hwdep snd input_core soundcore rxkad vfat fat udf crc_itu_t ntfs nfs_layout_nfsv41_files nfsv4 nfs auth_rpcgss oid_registry lockd sunrpc grace minix isofs hfsplus hfs cramfs configfs cifs autofs4 kafs af_rxrpc dns_resolver dm_crypt dm_mirror dm_region_hash dm_log dm_mod br2684 atm multipath fscache raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid10 raid1 raid0 linear md_mod nls_utf8 nls_koi8_r nls_cp1255 nls_iso8859_6 nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_iso8859_1 nls_cp932 nls_cp866 nls_cp864 nls_cp862 nls_cp852 nls_cp850 nls_cp775 nls_cp437 nls_cp1251 nls_cp1250 xts algif_skcipher algif_hash af_alg sha512_generic sha256_generic sha1_generic seqiv jitterentropy_rng drbg pcbc md5 md4 marvell_cesa hmac gf128mul fcrypt ecb des_generic ctr cmac ccm cbc authenc xhci_plat_hcd xhci_pci xhci_hcd uhci_hcd ledtrig_transient ahci_platform uas xfs
2019-05-05 22:09:09 notice kernel[]: jfs f2fs exfat mii aead crypto_null [last unloaded: ifb]
2019-05-05 22:09:09 warning kernel[]: [ 1134.499616] CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G        W       4.4.178-7bc33afbb1b35f5830b2b1b42c9cd8a0-0 #1
2019-05-05 22:09:09 warning kernel[]: [ 1134.499619] Hardware name: Marvell Armada 380/385 (Device Tree)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499621] Backtrace:
2019-05-05 22:09:09 warning kernel[]: [ 1134.499627] [<c001c504>] (dump_backtrace) from [<c001c744>] (show_stack+0x18/0x1c)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499630]  r6:00000000 r5:60000113 r4:c0770d28 r3:00000000
2019-05-05 22:09:09 warning kernel[]: [ 1134.499640] [<c001c72c>] (show_stack) from [<c02ffd7c>] (dump_stack+0x98/0xac)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499646] [<c02ffce4>] (dump_stack) from [<c0029234>] (warn_slowpath_common+0x8c/0xbc)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499649]  r6:00000592 r5:bf7755c0 r4:00000000 r3:c0769454
2019-05-05 22:09:09 warning kernel[]: [ 1134.499658] [<c00291a8>] (warn_slowpath_common) from [<c0029308>] (warn_slowpath_null+0x24/0x2c)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499661]  r8:ec41d908 r7:00000002 r6:ec41dbd0 r5:00000004 r4:212c4a20
2019-05-05 22:09:09 warning kernel[]: [ 1134.499678] [<c00292e4>] (warn_slowpath_null) from [<bf7755c0>] (hfsc_dequeue+0x34c/0x508 [sch_hfsc])
2019-05-05 22:09:09 warning kernel[]: [ 1134.499689] [<bf775274>] (hfsc_dequeue [sch_hfsc]) from [<c04ca07c>] (__qdisc_run+0x3c/0x1ac)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499692]  r9:00000040 r8:00000000 r7:00000000 r6:ec978400 r5:00000000 r4:ec41d800
2019-05-05 22:09:09 warning kernel[]: [ 1134.499703] [<c04ca040>] (__qdisc_run) from [<c04a72cc>] (net_tx_action+0x150/0x180)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499706]  r10:00000000 r9:00000002 r8:c075c3c0 r7:00000000 r6:c075c3c0 r5:00000000
2019-05-05 22:09:09 warning kernel[]: [ 1134.499714]  r4:ec41d800
2019-05-05 22:09:09 warning kernel[]: [ 1134.499720] [<c04a717c>] (net_tx_action) from [<c002c7a8>] (__do_softirq+0x114/0x260)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499723]  r8:00000100 r7:ef07e000 r6:40000002 r5:c0760080 r4:c0760088 r3:c04a717c
2019-05-05 22:09:09 warning kernel[]: [ 1134.499734] [<c002c694>] (__do_softirq) from [<c002c934>] (run_ksoftirqd+0x40/0x58)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499737]  r10:00000000 r9:00000002 r8:c0766938 r7:00000001 r6:00000000 r5:ef0209c0
2019-05-05 22:09:09 warning kernel[]: [ 1134.499745]  r4:ef07e000
2019-05-05 22:09:09 warning kernel[]: [ 1134.499752] [<c002c8f4>] (run_ksoftirqd) from [<c004713c>] (smpboot_thread_fn+0x11c/0x198)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499758] [<c0047020>] (smpboot_thread_fn) from [<c0044120>] (kthread+0x110/0x124)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499760]  r10:00000000 r9:00000000 r8:00000000 r7:c0047020 r6:ef0209c0 r5:00000000
2019-05-05 22:09:09 warning kernel[]: [ 1134.499768]  r4:ef020a40 r3:ef05b180
2019-05-05 22:09:09 warning kernel[]: [ 1134.499775] [<c0044010>] (kthread) from [<c0009dd0>] (ret_from_fork+0x14/0x24)
2019-05-05 22:09:09 warning kernel[]: [ 1134.499778]  r7:00000000 r6:00000000 r5:c0044010 r4:ef020a40
2019-05-05 22:09:09 warning kernel[]: [ 1134.499784] ---[ end trace d3d836264513a417 ]---

By “frequent”, I mean several per second. I have so far not noticed any actual problems, except for my /tmp filling up to the point where the router becomes unusable.

Output of uname -a: Linux turris 4.4.178-7bc33afbb1b35f5830b2b1b42c9cd8a0-0 #1 SMP Thu Apr 11 15:11:29 CEST 2019 armv7l GNU/Linux
Turris version: 3.11.4

Does anyone have an idea what’s going on?

PS: The common thread among these messages seems to be net/sched/sch_hfsc.c:1426.


#2

Alright, I may not have found a solution, but a workaround: Turn off QoS. I’m not happy with this workaround, but it will do for now.

I think this might be related: https://bugs.openwrt.org/index.php?do=details&task_id=9&status[0]=&pagenum=45