Turris Omnia 3.11.7 crashes every 2-3 days (Provider reconnect(?))

Dear forum,

I am somewhat desparated / confused - hope that someone can help me.

# cat /etc/turris-version 
3.11.7

Unfortunately my turris is unstable. It crashes 2-3 times in a week. I logged the output before the crash to a remote syslog server. This seems to happen when my (also quiet unstable) uplink VDSL to my ISP reestablishes.

This makes me crazy - my environment is unfortunately sensitive - I am really dependent on a stable device :frowning:

The only special thing I have running is surricata + device detection - but I also deactivated it and observed the same behavior.

Can someone please advise / help? Help is really appreciated.

Thanks.

Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.723161] netifd: page allocation failure: order:5, mode:0x24000c0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.729696] CPU: 1 PID: 1939 Comm: netifd Not tainted 4.4.191-a890a5a94ebb621f8f1720c24d12fef1-0 #1 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.738850] Hardware name: Marvell Armada 380/385 (Device Tree) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.744868] Backtrace:  
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.747424] [<c001c654>] (dump_backtrace) from [<c001c894>] (show_stack+0x18/0x1c) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.755097]  r6:00000000 r5:60000013 r4:c077cea8 r3:00000000 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.760900] [<c001c87c>] (show_stack) from [<c0304004>] (dump_stack+0x98/0xac) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.768229] [<c0303f6c>] (dump_stack) from [<c00b203c>] (warn_alloc_failed+0xec/0x120) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.776249]  r6:00000001 r5:00000005 r4:024000c0 r3:00000000 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.782051] [<c00b1f54>] (warn_alloc_failed) from [<c00b4e9c>] (__alloc_pages_nodemask+0x920/0xa54) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.791203]  r3:ebb4c000 r2:00000000 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.794891]  r6:c076cd5c r5:b6db6db7 r4:00000000 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.799637] [<c00b457c>] (__alloc_pages_nodemask) from [<c00209a8>] (__dma_alloc_buffer+0x34/0x18c) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.808788]  r10:00011000 r9:00000001 r8:ef1da108 r7:024000c0 r6:00000005 r5:00011000 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.816773]  r4:ffffffff 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.819409] [<c0020974>] (__dma_alloc_buffer) from [<c0020cb8>] (__dma_alloc+0x1b8/0x22c) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.827690]  r9:00000001 r8:ef1da108 r7:024000c0 r6:ef133810 r5:00000001 r4:ffffffff 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.835591] [<c0020b00>] (__dma_alloc) from [<c0020d64>] (arm_coherent_dma_alloc+0x38/0x40) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.844046]  r10:00010a00 r9:ef28dc80 r8:c07723a8 r7:ef133810 r6:ef1da010 r5:000000c0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.852030]  r4:00000000 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.854670] [<c0020d2c>] (arm_coherent_dma_alloc) from [<c0426d04>] (mvneta_setup_txqs+0x390/0x3d8) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.863821]  r4:ef1da0d0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.866458] [<c0426974>] (mvneta_setup_txqs) from [<c0429280>] (mvneta_open+0x2ac/0x440) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.874652]  r10:0000c018 r9:0000c018 r8:ef28dc80 r7:ef133810 r6:00000000 r5:ef28d800 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.882636]  r4:ef28dc80 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.885272] [<c0428fd4>] (mvneta_open) from [<c04b0044>] (__dev_open+0xb0/0x114) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.892770]  r10:00000000 r9:00000000 r8:00000000 r7:00001002 r6:ef28d830 r5:c06e14fc 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.900754]  r4:ef28d800 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.903390] [<c04aff94>] (__dev_open) from [<c04b0300>] (__dev_change_flags+0x9c/0x160) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.911496]  r7:00001002 r6:00000001 r5:00001003 r4:ef28d800 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.917296] [<c04b0264>] (__dev_change_flags) from [<c04b03e4>] (dev_change_flags+0x20/0x50) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.925838]  r8:00000000 r7:c06e14fc r6:00001002 r5:00000140 r4:ef28d800 r3:00008914 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.933739] [<c04b03c4>] (dev_change_flags) from [<c04c6884>] (dev_ifsioc+0x20c/0x314) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.941758]  r8:ebb4dea8 r7:c06e14fc r6:ebb4dea8 r5:ef28d800 r4:00008914 r3:00008914 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.949658] [<c04c6678>] (dev_ifsioc) from [<c04c6db4>] (dev_ioctl+0x3b4/0x8e4)
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.971338]  r10:00000000 r9:bec6fcf0 r8:eec47ca0 r7:c07940c0 r6:bec6fcf0 r5:fffffdfd 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.979323]  r4:00008914 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.981961] [<c049147c>] (sock_ioctl) from [<c0106a28>] (do_vfs_ioctl+0x4b4/0x6ec) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.989632]  r7:00008914 r6:bec6fcf0 r5:ed45c540 r4:ed45c540 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556148.995434] [<c0106574>] (do_vfs_ioctl) from [<c0106c9c>] (SyS_ioctl+0x3c/0x64) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.002844]  r10:00000000 r9:ebb4c000 r8:00000006 r7:00008914 r6:bec6fcf0 r5:ed45c540 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.010829]  r4:ed45c540 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.013466] [<c0106c60>] (SyS_ioctl) from [<c0009d00>] (ret_fast_syscall+0x0/0x48) 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.021137]  r8:c0009ec4 r7:00000036 r6:00001003 r5:00001003 r4:0003b564 r3:bec6fce8 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.029114] Mem-Info: 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.031489] active_anon:120605 inactive_anon:233 isolated_anon:0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.031489]  active_file:51848 inactive_file:24327 isolated_file:3 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.031489]  unevictable:0 dirty:0 writeback:0 unstable:0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.031489]  slab_reclaimable:6395 slab_unreclaimable:2820 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.031489]  mapped:7906 shmem:63456 pagetables:593 bounce:0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.031489]  free:31195 free_pcp:46 free_cma:0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.065405] Normal free:31444kB min:3504kB low:4380kB high:5256kB active_anon:347652kB inactive_anon:632kB active_f
ile:195880kB inactive_file:76952kB unevictable:0kB isolated(anon):0kB isolated(file):12kB present:786432kB managed:768540kB mlocked:0kB dirty:0kB writeback:0kB mapped:26684k
B shmem:182256kB slab_reclaimable:25580kB slab_unreclaimable:11280kB kernel_stack:1728kB pagetables:2372kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.110994] lowmem_reserve[]: 0 2048 2048 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.115151] HighMem free:93404kB min:256kB low:552kB high:852kB active_anon:134768kB inactive_anon:300kB active_fil
e:11512kB inactive_file:20356kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:262144kB mlocked:0kB dirty:0kB writeback:0kB mapped:4940kB shm
em:71568kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:56kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pag
es_scanned:0 all_unreclaimable? no 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.158848] lowmem_reserve[]: 0 0 0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.162470] Normal: 2522*4kB (UME) 1938*8kB (UME) 330*16kB (UM) 11*32kB (UM) 4*64kB (U) 0*128kB 0*256kB 0*512kB 0*1
024kB 0*2048kB 0*4096kB = 31480kB 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.176123] HighMem: 802*4kB (UM) 582*8kB (UM) 943*16kB (UM) 649*32kB (UM) 356*64kB (UM) 110*128kB (UM) 36*256kB (U
M) 7*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 93384kB 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.191559] 139631 total pagecache pages 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.195577] 0 pages in swap cache 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.199001] Swap cache stats: add 0, delete 0, find 0/0 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.204323] Free swap  = 0kB 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.207294] Total swap = 0kB 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.210321] 262144 pages RAM
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.207294] Total swap = 0kB 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.210321] 262144 pages RAM 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.213293] 65536 pages HighMem/MovableOnly 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.217620] 4473 pages reserved 
Oct 20 17:46:29 <user.err> waechter waechter kernel: [556149.220862] mvneta f1034000.ethernet eth1: mvneta_setup_txqs: can't create txq=3 
Oct 20 15:46:29 <daemon.notice> waechter waechter netifd: Interface 'wan' is enabled 
Oct 20 15:46:29 <daemon.notice> waechter waechter netifd: Interface 'wan' is setting up now 
Oct 20 17:46:29 <user.alert> waechter waechter kernel: [556149.229527] BUG: Bad page state in process netifd  pfn:2c365 
Oct 20 17:46:29 <user.emerg> waechter waechter kernel: [556149.235304] page:efd30a34 count:-1 mapcount:0 mapping:  (null) index:0x0 
Oct 20 17:46:29 <user.emerg> waechter waechter kernel: [556149.242139] flags: 0x0() 
Oct 20 17:46:29 <user.alert> waechter waechter kernel: [556149.244773] page dumped because: nonzero _count 
Oct 20 17:46:29 <user.warn> waechter waechter kernel: [556149.249426] Modules linked in: qcserial option iptable_nat ath9k usb_wwan rndis_host qmi_wwan pppoe nf_nat_pptp nf_
nat_ipv4 nf_nat_amanda nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_ipv6 nf_conntrack_ipv4 nf_conntrack_amanda ipt_REJECT ipt_MASQUERADE ftdi_sio ebtable_nat ebtable_
filter ebtable_broute cdc_ether ath9k_common armada_thermal xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length
 xt_id xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NFQUEUE xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY u
sbserial usbnet ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda ts_kmp ts_fsm ts_bm thermal_sys pppox ppp_mp
pe ppp_async nfnetlink_queue nfnetlink_log nfnetlink nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_i
rc n
Oct 20 17:46:29 <user.notice> waechter waechter kernel: nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_
proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast nf_conntrack mvsdio iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables hwmon eb
tables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt cdc_wdm ath9k_hw fuse sch_teql sch_tbf sch_sfq sch_red sch_prio 
sch_pie sch_netem sch_htb sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_vlan act_police act_pedit act_nat act_ipt act_gact act_csum act_
bpf act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress ath10k_pci ath10k_core ath mac80211 cfg80211 compat ledtrig_usbdev ledtr
ig_oneshot xt_LED ledtrig_morse ledtrig_heartbeat ledtrig_gpio cryptodev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip
6_tab
Oct 20 17:46:29 <user.notice> waechter waechter kernel: slhc nfsd nfsv3 msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel veth tun snd_compress snd_pcm_oss snd_m
ixer_oss snd_pcm snd_timer snd_rawmidi snd_seq_device snd_hwdep snd input_core soundcore rxkad vfat fat udf crc_itu_t ntfs nfs_layout_nfsv41_files nfsv4 nfs auth_rpcgss oid_
registry lockd sunrpc grace minix isofs hfsplus hfs cramfs configfs cifs autofs4 kafs af_rxrpc dns_resolver dm_crypt dm_mirror dm_region_hash dm_log dm_mod br2684 atm multip
ath fscache raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid10 raid1 raid0 linear md_mod nls_utf8 nls_koi8_r nls_cp1255 nls_iso8859_6 nls_iso8859_2 nl
s_iso8859_15 nls_iso8859_13 nls_iso8859_1 nls_cp932 nls_cp866 nls_cp864 nls_cp862 nls_cp852 nls_cp850 nls_cp775 nls_cp437 nls_cp1251 nls_cp1250 xts algif_skcipher algif_hash
 af_alg sha512_generic sha256_generic sha1_generic seqiv jitterentropy_rng drbg pcbc md5 md4 marvell_cesa hmac gf128mul fcrypt ecb des_generic ctr cmac ccm cbc authenc xhci_
plat_
Oct 20 17:46:29 <user.notice> waechter waechter kernel: ahci_platform uas xfs libcrc32c jfs f2fs exfat mii aead crypto_null 

(…, cutted - if more output is needed, I will post it)

Try update to Turris OS 3.11.8.

Ok, thanks.

Now:

# cat /etc/turris-version
3.11.8

Here are some error messages during boot:

Oct 20 19:03:29 <user.warn> waechter waechter kernel: [   16.073726] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
Oct 20 19:03:29 <user.warn> waechter waechter kernel: [   16.084464] ath10k_pci 0000:02:00.0: Falling back to user helper
Oct 20 19:03:29 <user.err> waechter waechter kernel: [   16.094644] firmware ath10k!pre-cal-pci-0000:02:00.0.bin: firmware_loading_store: map pages failed
Oct 20 19:03:29 <user.warn> waechter waechter kernel: [   16.103749] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2
Oct 20 19:03:29 <user.warn> waechter waechter kernel: [   16.114143] ath10k_pci 0000:02:00.0: Falling back to user helper
Oct 20 19:03:29 <user.err> waechter waechter kernel: [   16.124137] firmware ath10k!cal-pci-0000:02:00.0.bin: firmware_loading_store: map pages failed
Oct 20 19:03:29 <user.warn> waechter waechter kernel: [   16.205303] ath10k_pci 0000:02:00.0: board id is not exist in otp, ignore it
Oct 20 19:03:29 <user.err> waechter waechter kernel: [   16.213945] ath10k_pci 0000:02:00.0: found invalid board magic

I exchanged both Wifi NICs a few years ago with MikroTik cards.

output of lspci

# lspci -v
00:01.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04) (prog-if 00 [Normal decode])
	Flags: fast devsel
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	Prefetchable memory behind bridge: 00000000-000fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
lspci: Unable to load libkmod resources: error -12

00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	Memory behind bridge: e0000000-e02fffff
	Prefetchable memory behind bridge: 00000000-000fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00

00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6820 (rev 04) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	Memory behind bridge: e0300000-e03fffff
	Prefetchable memory behind bridge: 00000000-000fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00

02:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter
	Subsystem: Device 19b6:d03c
	Flags: bus master, fast devsel, latency 0, IRQ 120
	Memory at e0000000 (64-bit, non-prefetchable) [size=2M]
	Expansion ROM at e0200000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 2
	Capabilities: [50] MSI: Enable+ Count=1/8 Maskable+ 64bit-
	Capabilities: [70] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
	Kernel driver in use: ath10k_pci

03:00.0 Network controller: Qualcomm Atheros AR9580 Wireless Network Adapter (rev 01)
	Subsystem: Device 19b6:d016
	Flags: bus master, fast devsel, latency 0, IRQ 109
	Memory at e0300000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at e0320000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit+
	Capabilities: [70] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [300] Device Serial Number 00-00-00-00-00-00-00-00
	Kernel driver in use: ath9k

I will observe the behavior under the current release now and come back to you.