tuomaz
February 25, 2024, 8:27pm
1
Hi
When 6.5.0 became available I upgraded to it. Shortly after I also upgraded the firmware. After that I’ve had random reboots that makes the router unusable. This is a typical stacktrace from the serial console:
Router Turris successfully started.
[ 159.431145] 8<--- cut here ---
[ 159.434214] Unable to handle kernel paging request at virtual address 8477ae28
[ 159.441454] pgd = 18398363
[ 159.444166] [8477ae28] *pgd=00000000
[ 159.447752] Internal error: Oops: 805 [#1] SMP ARM
[ 159.452554] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[ 159.452721] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[ 159.618699] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.147 #0
[ 159.624808] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 159.630740] PC is at handle_percpu_devid_irq+0x30/0x12c
[ 159.635983] LR is at handle_domain_irq+0x5c/0x78
[ 159.640616] pc : [<c0184dfc>] lr : [<c017f2d8>] psr: a0000193
[ 159.646896] sp : c0f01ed0 ip : c45f603c fp : 00000000
[ 159.652132] r10: c0e580a8 r9 : c0e57ccc r8 : f080210c
[ 159.657368] r7 : 00000000 r6 : 00000000 r5 : 00000000 r4 : c0e57cc0
[ 159.663909] r3 : c0e50000 r2 : c0184dec r1 : 00000032 r0 : c1010400
[ 159.670450] Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[ 159.677690] Control: 10c5387d Table: 0369004a DAC: 00000051
[ 159.683447] Register r0 information: slab kmalloc-256 start c1010400 pointer offset 0 size 256
[ 159.692087] Register r1 information: non-paged memory
[ 159.697151] Register r2 information: non-slab/vmalloc memory
[ 159.702824] Register r3 information: non-slab/vmalloc memory
[ 159.708495] Register r4 information: non-slab/vmalloc memory
[ 159.714167] Register r5 information: NULL pointer
[ 159.718881] Register r6 information: NULL pointer
[ 159.723594] Register r7 information: NULL pointer
[ 159.728308] Register r8 information: 0-page vmalloc region starting at 0xf0802000 allocated at of_iomap+0x48/0x64
[ 159.738602] Register r9 information: non-slab/vmalloc memory
[ 159.744274] Register r10 information: non-slab/vmalloc memory
[ 159.750032] Register r11 information: NULL pointer
[ 159.754834] Register r12 information: non-slab/vmalloc memory
[ 159.760592] Process swapper/0 (pid: 0, stack limit = 0x8d871e88)
[ 159.766613] Stack: (0xc0f01ed0 to 0xc0f02000)
[ 159.770981] 1ec0: c0f05404 c0f01f08 c0f85970 f0802100
[ 159.779178] 1ee0: f080210c c01012e4 c0107a74 60000013 ffffffff c0f01f3c 00000000 c0f00000
[ 159.787375] 1f00: c0e580a8 c0100b7c 0009ffbe 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[ 159.795572] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c45f603c c0f01f58
[ 159.803768] 1f40: c0107a70 c0107a74 60000013 ffffffff 00000051 c0192a44 ffffe000 c09ff950
[ 159.811965] 1f60: ffffe000 c015c8b4 efffc21d c0f10514 efffc200 c0adbcdd c0cc9688 000000e7
[ 159.820162] 1f80: c0fd5524 efffc21d 00000000 efffc200 c0f085b4 c0cc9688 c0f08520 c015cbe4
[ 159.828358] 1fa0: c0fa903c c0e0124c ffffffff ffffffff 00000000 c0e006e4 c0f04f08 c0fa9014
[ 159.836555] 1fc0: 00000000 c0e45a54 c0a8b34e 00000000 00000000 c0e00470 00000051 10c0387d
[ 159.844751] 1fe0: 00000000 0fff7000 414fc091 10c5387d 00000000 00000000 00000000 00000000
[ 159.852948] [<c0184dfc>] (handle_percpu_devid_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[ 159.861675] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[ 159.869178] Exception stack(0xc0f01f08 to 0xc0f01f50)
[ 159.874242] 1f00: 0009ffbe 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[ 159.882438] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c45f603c c0f01f58
[ 159.890634] 1f40: c0107a70 c0107a74 60000013 ffffffff
[ 159.895696] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[ 159.903115] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[ 159.911235] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[ 159.919006] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[ 159.926597] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[ 159.934802] Code: e34c30e5 e5946040 e2811001 e594701c (e78c1002)
[ 159.940910] ---[ end trace ed9ed2bbff1c2bef ]---
[ 159.945538] Kernel panic - not syncing: Fatal exception in interrupt
[ 159.951906] CPU1: stopping
[ 159.954624] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 5.15.147 #0
[ 159.962128] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 159.968062] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[ 159.975831] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[ 159.983424] [<c05e844c>] (dump_stack_lvl) from [<c010ce88>] (do_handle_IPI+0xf8/0x12c)
[ 159.991361] [<c010ce88>] (do_handle_IPI) from [<c010ced4>] (ipi_handler+0x18/0x20)
[ 159.998951] [<c010ced4>] (ipi_handler) from [<c0184e44>] (handle_percpu_devid_irq+0x78/0x12c)
[ 160.007500] [<c0184e44>] (handle_percpu_devid_irq) from [<c017f2d8>] (handle_domain_irq+0x5c/0x78)
[ 160.016485] [<c017f2d8>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[ 160.024684] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[ 160.032186] Exception stack(0xc1083f58 to 0xc1083fa0)
[ 160.037249] 3f40: 00088e72 00000000
[ 160.045447] 3f60: 00000001 c0117cc0 ffffe000 c0f04f54 c0f04f98 00000002 00000000 c0f04f08
[ 160.053644] 3f80: c0e580a8 00000000 c460003c c1083fa8 c0107a70 c0107a74 60000013 ffffffff
[ 160.061839] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[ 160.069256] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[ 160.077371] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[ 160.085138] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[ 160.092729] [<c015cbe4>] (cpu_startup_entry) from [<00101790>] (0x101790)
[ 160.099538] Rebooting in 3 seconds..
I am in contact with the Turris tech support and getting good help, but no resolution so far. I want to know if others have the same problem, or am I alone? I think I saw one or two mentions of similar problems in the release thread for 6.5.0.
My Turris Omnia is a 2GB version with an added SSD and removed wifi cards. The router is now running 6.5.2.
Is it a hardware error on my device or kernel related?
tuomaz
February 25, 2024, 8:28pm
2
Another stacktrace:
Router Turris successfully started.
[ 1536.856642] 8<--- cut here ---
[ 1536.859712] Unable to handle kernel paging request at virtual address 5404c13d
[ 1536.866951] pgd = 50018573
[ 1536.869663] [5404c13d] *pgd=00000000
[ 1536.873249] Internal error: Oops: 5 [#1] SMP ARM
[ 1536.877876] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[ 1536.878045] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[ 1537.044021] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.147 #0
[ 1537.050131] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 1537.056063] PC is at handle_domain_irq+0x4c/0x78
[ 1537.060698] LR is at __irq_resolve_mapping+0x44/0x7c
[ 1537.065677] pc : [<c017f2c8>] lr : [<c01872bc>] psr: 20000193
[ 1537.071958] sp : c0f01ed0 ip : c107c03c fp : 00000000
[ 1537.077193] r10: c0e580a8 r9 : c0e57ccc r8 : f080210c
[ 1537.082429] r7 : 0000001d r6 : c100c000 r5 : 00000000 r4 : c0e57cc0
[ 1537.088970] r3 : c0f00000 r2 : 5404c101 r1 : 00000031 r0 : 5404c101
[ 1537.095512] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[ 1537.102751] Control: 10c5387d Table: 0361c04a DAC: 00000051
[ 1537.108508] Register r0 information: non-paged memory
[ 1537.113572] Register r1 information: non-paged memory
[ 1537.118635] Register r2 information: non-paged memory
[ 1537.123697] Register r3 information: non-slab/vmalloc memory
[ 1537.129370] Register r4 information: non-slab/vmalloc memory
[ 1537.135042] Register r5 information: NULL pointer
[ 1537.139755] Register r6 information: slab kmalloc-1k start c100c000 pointer offset 0 size 1024
[ 1537.148394] Register r7 information: non-paged memory
[ 1537.153456] Register r8 information: 0-page vmalloc region starting at 0xf0802000 allocated at of_iomap+0x48/0x64
[ 1537.163750] Register r9 information: non-slab/vmalloc memory
[ 1537.169422] Register r10 information: non-slab/vmalloc memory
[ 1537.175181] Register r11 information: NULL pointer
[ 1537.179981] Register r12 information: non-slab/vmalloc memory
[ 1537.185739] Process swapper/0 (pid: 0, stack limit = 0x3bc7f97f)
[ 1537.191760] Stack: (0xc0f01ed0 to 0xc0f02000)
[ 1537.196128] 1ec0: c0f05404 c0f01f08 c0f85970 f0802100
[ 1537.204325] 1ee0: f080210c c01012e4 c0107a74 60000013 ffffffff c0f01f3c 00000000 c0f00000
[ 1537.212522] 1f00: c0e580a8 c0100b7c 001b9712 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[ 1537.220718] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c107c03c c0f01f58
[ 1537.228915] 1f40: c0107a70 c0107a74 60000013 ffffffff 00000051 c0192a44 ffffe000 c09ff950
[ 1537.237112] 1f60: ffffe000 c015c8b4 efffc21d c0f10514 efffc200 846817c5 c0cc9688 000000e7
[ 1537.245309] 1f80: c0fd5524 efffc21d 00000000 efffc200 c0f085b4 c0cc9688 c0f08520 c015cbe4
[ 1537.253505] 1fa0: c0fa903c c0e0124c ffffffff ffffffff 00000000 c0e006e4 c0f04f08 c0fa9014
[ 1537.261701] 1fc0: 00000000 c0e45a54 846d1856 00000000 00000000 c0e00470 00000051 10c0387d
[ 1537.269898] 1fe0: 00000000 0fff7000 414fc091 10c5387d 00000000 00000000 00000000 00000000
[ 1537.278095] [<c017f2c8>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[ 1537.286300] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[ 1537.293803] Exception stack(0xc0f01f08 to 0xc0f01f50)
[ 1537.298866] 1f00: 001b9712 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[ 1537.307063] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c107c03c c0f01f58
[ 1537.315258] 1f40: c0107a70 c0107a74 60000013 ffffffff
[ 1537.320320] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[ 1537.327739] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[ 1537.335859] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[ 1537.343629] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[ 1537.351221] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[ 1537.359426] Code: 0a00000b e1a0300d e3c33d7f e3c3303f (e592203c)
[ 1537.365534] ---[ end trace d3fff967a6fe4ecb ]---
[ 1537.370161] Kernel panic - not syncing: Fatal exception in interrupt
[ 1537.376529] CPU1: stopping
[ 1537.379245] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 5.15.147 #0
[ 1537.386748] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 1537.392681] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[ 1537.400451] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[ 1537.408044] [<c05e844c>] (dump_stack_lvl) from [<c010ce88>] (do_handle_IPI+0xf8/0x12c)
[ 1537.415983] [<c010ce88>] (do_handle_IPI) from [<c010ced4>] (ipi_handler+0x18/0x20)
[ 1537.423573] [<c010ced4>] (ipi_handler) from [<c0184e44>] (handle_percpu_devid_irq+0x78/0x12c)
[ 1537.432122] [<c0184e44>] (handle_percpu_devid_irq) from [<c017f2d8>] (handle_domain_irq+0x5c/0x78)
[ 1537.441107] [<c017f2d8>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[ 1537.449307] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[ 1537.456809] Exception stack(0xc1083f58 to 0xc1083fa0)
[ 1537.461871] 3f40: 001e8e52 00000000
[ 1537.470068] 3f60: 00000001 c0117cc0 ffffe000 c0f04f54 c0f04f98 00000002 00000000 c0f04f08
[ 1537.478265] 3f80: c0e580a8 00000000 c3fd003c c1083fa8 c0107a70 c0107a74 60000013 ffffffff
[ 1537.486461] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[ 1537.493878] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[ 1537.501992] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[ 1537.509759] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[ 1537.517350] [<c015cbe4>] (cpu_startup_entry) from [<00101790>] (0x101790)
[ 1537.524159] Rebooting in 3 seconds..
tuomaz
February 25, 2024, 8:30pm
3
One more:
Router Turris successfully started.
[ 6358.348015] BUG: scheduling while atomic: swapper/1/0/0x0a000010
[ 6358.354052] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[ 6358.354220] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[ 6358.520228] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.15.147 #0
[ 6358.526339] Hardware name: Marvell Armada 380/385 (Device Tree)
[ 6358.532274] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[ 6358.540046] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[ 6358.547637] [<c05e844c>] (dump_stack_lvl) from [<c0153a54>] (__schedule_bug+0x5c/0x70)
[ 6358.555581] [<c0153a54>] (__schedule_bug) from [<c09f9790>] (__schedule+0x468/0x688)
[ 6358.563350] [<c09f9790>] (__schedule) from [<c09f9d40>] (schedule_idle+0x38/0x7c)
[ 6358.570853] [<c09f9d40>] (schedule_idle) from [<c015c848>] (do_idle+0x16c/0x1e0)
[ 6358.578275] [<c015c848>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[ 6358.585866] [<c015cbe4>] (cpu_startup_entry) from [<00101790>] (0x101790)
[20768.486636] ------------[ cut here ]------------
[20768.491273] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:752 rcu_nmi_exit+0xf8/0x110
[20768.498957] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[20768.499126] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[20768.665104] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.15.147 #0
[20768.672607] Hardware name: Marvell Armada 380/385 (Device Tree)
[20768.678542] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[20768.686312] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[20768.693905] [<c05e844c>] (dump_stack_lvl) from [<c012d0b8>] (__warn+0xa4/0xbc)
[20768.701153] [<c012d0b8>] (__warn) from [<c012d130>] (warn_slowpath_fmt+0x60/0xb8)
[20768.708659] [<c012d130>] (warn_slowpath_fmt) from [<c09f7f44>] (rcu_nmi_exit+0xf8/0x110)
[20768.716774] [<c09f7f44>] (rcu_nmi_exit) from [<c017f2dc>] (handle_domain_irq+0x60/0x78)
[20768.724804] [<c017f2dc>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[20768.733006] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[20768.740509] Exception stack(0xc0f01f08 to 0xc0f01f50)
[20768.745573] 1f00: 01612220 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[20768.753770] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c125403c c0f01f58
[20768.761966] 1f40: c0107a70 c0107a74 60000013 ffffffff
[20768.767027] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[20768.774446] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[20768.782563] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[20768.790333] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[20768.797923] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[20768.806127] ---[ end trace 883932a0dba35dff ]---
[20768.810754] ------------[ cut here ]------------
[20768.815380] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:1000 rcu_nmi_enter+0xac/0xb8
[20768.823143] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[20768.823300] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[20768.989268] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.15.147 #0
[20768.996769] Hardware name: Marvell Armada 380/385 (Device Tree)
[20769.002701] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[20769.010467] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[20769.018057] [<c05e844c>] (dump_stack_lvl) from [<c012d0b8>] (__warn+0xa4/0xbc)
[20769.025303] [<c012d0b8>] (__warn) from [<c012d130>] (warn_slowpath_fmt+0x60/0xb8)
[20769.032808] [<c012d130>] (warn_slowpath_fmt) from [<c09f800c>] (rcu_nmi_enter+0xac/0xb8)
[20769.040922] [<c09f800c>] (rcu_nmi_enter) from [<c0132a0c>] (irq_enter+0x8/0x10)
[20769.048253] [<c0132a0c>] (irq_enter) from [<c017f2a4>] (handle_domain_irq+0x28/0x78)
[20769.056020] [<c017f2a4>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[20769.064221] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[20769.071723] Exception stack(0xc0f01f08 to 0xc0f01f50)
[20769.076787] 1f00: 01612220 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[20769.084983] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c125403c c0f01f58
[20769.093179] 1f40: c0107a70 c0107a74 60000013 ffffffff
[20769.098240] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[20769.105657] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[20769.113773] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[20769.121540] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[20769.129131] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[20769.137333] ---[ end trace 883932a0dba35e00 ]---
[20769.142004] ------------[ cut here ]------------
[20769.146632] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:880 rcu_eqs_exit.constprop.25+0x80/0x84
[20769.155355] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[20769.155512] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[20769.321480] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.15.147 #0
[20769.328981] Hardware name: Marvell Armada 380/385 (Device Tree)
[20769.334914] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[20769.342681] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[20769.350271] [<c05e844c>] (dump_stack_lvl) from [<c012d0b8>] (__warn+0xa4/0xbc)
[20769.357517] [<c012d0b8>] (__warn) from [<c012d130>] (warn_slowpath_fmt+0x60/0xb8)
[20769.365022] [<c012d130>] (warn_slowpath_fmt) from [<c09f7dc4>] (rcu_eqs_exit.constprop.25+0x80/0x84)
[20769.374181] [<c09f7dc4>] (rcu_eqs_exit.constprop.25) from [<c0192a44>] (rcu_idle_exit+0x10/0x18)
[20769.382993] [<c0192a44>] (rcu_idle_exit) from [<c09ff958>] (default_idle_call+0x34/0x3c)
[20769.391109] [<c09ff958>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[20769.398876] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[20769.406467] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[20769.414670] ---[ end trace 883932a0dba35e01 ]---
[24070.060529] 8<--- cut here ---
[24070.063599] Unable to handle kernel paging request at virtual address f0802100
[24070.070839] pgd = f5ab8143
[24070.073552] [f0802100] *pgd=0100a811, *pte=f100c653, *ppte=f100c453
[24070.079840] Internal error: Oops: 8000000f [#1] SMP ARM
[24070.085078] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[24070.085245] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[24070.251222] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.15.147 #0
[24070.258725] Hardware name: Marvell Armada 380/385 (Device Tree)
[24070.264657] PC is at 0xf0802100
[24070.267804] LR is at handle_domain_irq+0x28/0x78
[24070.272438] pc : [<f0802100>] lr : [<c017f2a4>] psr: 60000193
[24070.278719] sp : c0f01ee0 ip : c333003c fp : 00000000
[24070.283955] r10: c0e580a8 r9 : c0e57ccc r8 : f080210c
[24070.289191] r7 : 0000001d r6 : c0f85970 r5 : c0f01f08 r4 : c0f05404
[24070.295732] r3 : 00000001 r2 : 00000000 r1 : 01970db1 r0 : 01970db1
[24070.302273] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[24070.309512] Control: 10c5387d Table: 0486004a DAC: 00000051
[24070.315269] Register r0 information: non-paged memory
[24070.320334] Register r1 information: non-paged memory
[24070.325396] Register r2 information: NULL pointer
[24070.330110] Register r3 information: non-paged memory
[24070.335172] Register r4 information: non-slab/vmalloc memory
[24070.340845] Register r5 information: non-slab/vmalloc memory
[24070.346517] Register r6 information: non-slab/vmalloc memory
[24070.352188] Register r7 information: non-paged memory
[24070.357250] Register r8 information: 0-page vmalloc region starting at 0xf0802000 allocated at of_iomap+0x48/0x64
[24070.367546] Register r9 information: non-slab/vmalloc memory
[24070.373218] Register r10 information: non-slab/vmalloc memory
[24070.378977] Register r11 information: NULL pointer
[24070.383778] Register r12 information: non-slab/vmalloc memory
[24070.389537] Process swapper/0 (pid: 0, stack limit = 0xdde5dc74)
[24070.395559] Stack: (0xc0f01ee0 to 0xc0f02000)
[24070.399928] 1ee0: f080210c c01012e4 c0107a74 60000013 ffffffff c0f01f3c 00000000 c0f00000
[24070.408125] 1f00: c0e580a8 c0100b7c 01970db0 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[24070.416322] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c333003c c0f01f58
[24070.424518] 1f40: c0107a70 c0107a74 60000013 ffffffff 00000051 c0192a44 ffffe000 c09ff950
[24070.432715] 1f60: ffffe000 c015c8b4 efffc21d c0f10514 efffc200 36e88710 c0cc9688 000000e7
[24070.440912] 1f80: c0fd5524 efffc21d 00000000 efffc200 c0f085b4 c0cc9688 c0f08520 c015cbe4
[24070.449109] 1fa0: c0fa903c c0e0124c ffffffff ffffffff 00000000 c0e006e4 c0f04f08 c0fa9014
[24070.457305] 1fc0: 00000000 c0e45a54 36ed8883 00000000 00000000 c0e00470 00000051 10c0387d
[24070.465502] 1fe0: 00000000 0fff7000 414fc091 10c5387d 00000000 00000000 00000000 00000000
[24070.473699] [<c017f2a4>] (handle_domain_irq) from [<c0f01f3c>] (0xc0f01f3c)
[24070.480686] Code: 00000000 00000000 00000000 00000000 (00000001)
[24070.486795] ---[ end trace 883932a0dba35e02 ]---
[24070.491423] Kernel panic - not syncing: Fatal exception
[24070.496660] CPU1: stopping
[24070.499378] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D W 5.15.147 #0
[24070.506883] Hardware name: Marvell Armada 380/385 (Device Tree)
[24070.512816] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[24070.520587] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[24070.528180] [<c05e844c>] (dump_stack_lvl) from [<c010ce88>] (do_handle_IPI+0xf8/0x12c)
[24070.536118] [<c010ce88>] (do_handle_IPI) from [<c010ced4>] (ipi_handler+0x18/0x20)
[24070.543708] [<c010ced4>] (ipi_handler) from [<c0184e44>] (handle_percpu_devid_irq+0x78/0x12c)
[24070.552257] [<c0184e44>] (handle_percpu_devid_irq) from [<c017f2d8>] (handle_domain_irq+0x5c/0x78)
[24070.561242] [<c017f2d8>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[24070.569444] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[24070.576947] Exception stack(0xc1083f58 to 0xc1083fa0)
[24070.582010] 3f40: 01669db6 00000000
[24070.590207] 3f60: 00000001 c0117cc0 ffffe000 c0f04f54 c0f04f98 00000002 00000000 c0f04f08
[24070.598404] 3f80: c0e580a8 00000000 c345403c c1083fa8 c0107a70 c0107a74 60000013 ffffffff
[24070.606598] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[24070.614017] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[24070.622136] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[24070.629907] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[24070.637498] [<c015cbe4>] (cpu_startup_entry) from [<00101790>] (0x101790)
[24070.644307] Rebooting in 3 seconds..
peci1
February 26, 2024, 2:23am
4
Obviously it is swapping. If you have it enabled, disable it. If not, add a few gigs of swap on the ssd.
tuomaz
February 26, 2024, 7:53am
5
Thanks. I have a super clean and minimal 6.5.2 install. Have not customized anything. Why would swap be enabled and why would it need more memory than the available 2GB?
I’ll check if it is enabled and if I can add some gigs of swap on the SSD.
tuomaz
February 26, 2024, 8:03am
6
To me it looks like swap is not enabled:
root@turris:~# cat /proc/swaps
Filename Type Size Used Priority
root@turris:~#
root@turris:~# cat /proc/meminfo
MemTotal: 2063488 kB
MemFree: 1839204 kB
MemAvailable: 1806324 kB
Buffers: 1084 kB
Cached: 92880 kB
SwapCached: 0 kB
Active: 33440 kB
Inactive: 131584 kB
Active(anon): 296 kB
Inactive(anon): 91444 kB
Active(file): 33144 kB
Inactive(file): 40140 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 1310720 kB
HighFree: 1161724 kB
LowTotal: 752768 kB
LowFree: 677480 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 71100 kB
Mapped: 19252 kB
Shmem: 20680 kB
KReclaimable: 4772 kB
Slab: 21276 kB
SReclaimable: 4772 kB
SUnreclaim: 16504 kB
KernelStack: 1392 kB
PageTables: 880 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1031744 kB
Committed_AS: 121080 kB
VmallocTotal: 245760 kB
VmallocUsed: 8076 kB
VmallocChunk: 0 kB
Percpu: 232 kB
root@turris:~#
1 Like
tuomaz
February 26, 2024, 8:17am
7
I’ve now added 8G of swap. Will be interesting to see if it will survive for longer now.
Still, something must be wrong on the software side if it needs more memory than 2GB in a clean setup?
1 Like
peci1
February 26, 2024, 9:10am
8
Sometimes, not all memory can be used. Applications define the minimum allocation size, so if the memory is very fragmented, it can easily happen that you have e.g. 500 MB free RAM, but no chunk that would satisfy the minimum size requirement. This usually happens with faulty drivers.
I agree it should not be happening on an almost clean install.
tuomaz
February 26, 2024, 9:20pm
9
It ran for 41318 seconds (about 11 hours), then it rebooted with this in the serial log:
[ 6938.802390] Adding 8388604k swap on /dev/sda1. Priority:-2 extents:1 across:8388604k SS
[41317.899482] 8<--- cut here ---
[41317.902551] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[41317.910660] pgd = 85640beb
[41317.913373] [00000008] *pgd=00000000
[41317.916959] Internal error: Oops: 5 [#1] SMP ARM
[41317.921587] Modules linked in: ath9k ath9k_common xt_connlimit qcserial pppoe ppp_async nf_conncount iptable_nat ath9k_hw ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT usb_wwan pppox ppp_generic nf_nat_ftp nf_nat nf_flow_table nf_conntrack_netlink nf_conntrack_ftp nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY usbserial ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress ums_alauda slhc sch_cake nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables exfat ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among
[41317.921755] ebt_802_3 crc_ccitt compat br_netfilter at24 sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_gpio xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel nls_utf8 seqiv jitterentropy_rng drbg md5 hmac ghash_arm_ce cmac nls_iso8859_1 nls_cp437 uas leds_turris_omnia turris_omnia_mcu ledtrig_transient ahci fsl_mph_dr_of ehci_fsl xfs vfat fat
[41318.087732] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.147 #0
[41318.093841] Hardware name: Marvell Armada 380/385 (Device Tree)
[41318.099773] PC is at irq_enter_rcu+0xc/0x40
[41318.103971] LR is at handle_domain_irq+0x28/0x78
[41318.108603] pc : [<c01329d0>] lr : [<c017f2a4>] psr: 60000193
[41318.114884] sp : c0f01ed0 ip : c3d6a03c fp : 00000000
[41318.120120] r10: c0e580a8 r9 : c0e57ccc r8 : f080210c
[41318.125356] r7 : 0000001d r6 : c100c000 r5 : 00000000 r4 : c0e57cc0
[41318.131897] r3 : 00000000 r2 : 00000000 r1 : 00000001 r0 : 02d4b14f
[41318.138438] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[41318.145678] Control: 10c5387d Table: 0401004a DAC: 00000051
[41318.151435] Register r0 information: non-paged memory
[41318.156499] Register r1 information: non-paged memory
[41318.161562] Register r2 information: NULL pointer
[41318.166275] Register r3 information: NULL pointer
[41318.170989] Register r4 information: non-slab/vmalloc memory
[41318.176662] Register r5 information: NULL pointer
[41318.181376] Register r6 information: slab kmalloc-1k start c100c000 pointer offset 0 size 1024
[41318.190015] Register r7 information: non-paged memory
[41318.195077] Register r8 information: 0-page vmalloc region starting at 0xf0802000 allocated at of_iomap+0x48/0x64
[41318.205371] Register r9 information: non-slab/vmalloc memory
[41318.211043] Register r10 information: non-slab/vmalloc memory
[41318.216802] Register r11 information: NULL pointer
[41318.221603] Register r12 information: non-slab/vmalloc memory
[41318.227361] Process swapper/0 (pid: 0, stack limit = 0x105ae6e5)
[41318.233382] Stack: (0xc0f01ed0 to 0xc0f02000)
[41318.237750] 1ec0: c0f05404 c0f01f08 c0f85970 f0802100
[41318.245947] 1ee0: f080210c c01012e4 c0107a74 60000013 ffffffff c0f01f3c 00000000 c0f00000
[41318.254144] 1f00: c0e580a8 c0100b7c 02d4b14e 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[41318.262340] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c3d6a03c c0f01f58
[41318.270537] 1f40: c0107a70 c0107a74 60000013 ffffffff 00000051 c0192a44 ffffe000 c09ff950
[41318.278734] 1f60: ffffe000 c015c8b4 c015cbe4 c0f10514 60000013 693fbf73 00000051 000000e7
[41318.286930] 1f80: c0fd5524 efffc21d 00000000 efffc200 c0f085b4 c0cc9688 c0f08520 c015cbe4
[41318.295127] 1fa0: c0fa903c c0e0124c ffffffff ffffffff 00000000 c0e006e4 c0f04f08 c0fa9014
[41318.303323] 1fc0: 00000000 c0e45a54 693ab0e0 00000000 00000000 c0e00470 00000051 10c0387d
[41318.311520] 1fe0: 00000000 0fff7000 414fc091 10c5387d 00000000 00000000 00000000 00000000
[41318.319717] [<c01329d0>] (irq_enter_rcu) from [<c017f2a4>] (handle_domain_irq+0x28/0x78)
[41318.327834] [<c017f2a4>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[41318.336037] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[41318.343540] Exception stack(0xc0f01f08 to 0xc0f01f50)
[41318.348603] 1f00: 02d4b14e 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[41318.356800] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c3d6a03c c0f01f58
[41318.364995] 1f40: c0107a70 c0107a74 60000013 ffffffff
[41318.370057] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[41318.377476] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[41318.385596] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[41318.393366] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[41318.400957] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[41318.409162] Code: ea231c3a e1a0300d e3c33d7f e3c3303f (e5931008)
[41318.415270] ---[ end trace efa97f6a1723576e ]---
[41318.419897] Kernel panic - not syncing: Fatal exception
[41318.425133] CPU1: stopping
[41318.427850] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 5.15.147 #0
[41318.435353] Hardware name: Marvell Armada 380/385 (Device Tree)
[41318.441286] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[41318.449056] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[41318.456649] [<c05e844c>] (dump_stack_lvl) from [<c010ce88>] (do_handle_IPI+0xf8/0x12c)
[41318.464587] [<c010ce88>] (do_handle_IPI) from [<c010ced4>] (ipi_handler+0x18/0x20)
[41318.472177] [<c010ced4>] (ipi_handler) from [<c0184e44>] (handle_percpu_devid_irq+0x78/0x12c)
[41318.480729] [<c0184e44>] (handle_percpu_devid_irq) from [<c017f2d8>] (handle_domain_irq+0x5c/0x78)
[41318.489713] [<c017f2d8>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[41318.497914] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[41318.505416] Exception stack(0xc1083f58 to 0xc1083fa0)
[41318.510478] 3f40: 02805bb2 00000000
[41318.518676] 3f60: 00000001 c0117cc0 ffffe000 c0f04f54 c0f04f98 00000002 00000000 c0f04f08
[41318.526873] 3f80: c0e580a8 00000000 c40f803c c1083fa8 c0107a70 c0107a74 60000013 ffffffff
[41318.535068] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[41318.542486] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[41318.550600] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[41318.558367] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[41318.565957] [<c015cbe4>] (cpu_startup_entry) from [<00101790>] (0x101790)
[41318.572767] Rebooting in 3 seconds..
Try to downgrade and let us know. My guess would be damaged memory.
tuomaz
February 27, 2024, 6:52am
11
Damaged memory would be sad
What would be the best way to downgrade?
I noticed the kernel in 6.5.2 is compiled without the memtest (kernel docs ) feature. Is it possible for me to compile my own kernel with memtest enabled and use? I also need to set the memtest kernel boot parameter.
memtest= [KNL,X86,ARM,M68K,PPC,RISCV] Enable memtest
Format: <integer>
default : 0 <disable>
Specifies the number of memtest passes to be
performed. Each pass selects another test
pattern from a given set of patterns. Memtest
fills the memory with this pattern, validates
memory contents and reserves bad memory
regions that are detected.
schnapps list
schnapps rollback X
reboot
tuomaz
February 28, 2024, 8:01am
13
I reflashed from an USB drive with an old medkit using reset mode “4 LEDS”. I got one reboot pretty quick even with 6.4.4 but now it has an uptime of over 14 hours. In the serial log I do have these errors though:
[11964.166795] ------------[ cut here ]------------
[11964.171432] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:753 rcu_nmi_exit+0xd8/0x110
[11964.179115] Modules linked in: sctp udp_tunnel ip6_udp_tunnel
[11964.184881] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.147 #0
[11964.190990] Hardware name: Marvell Armada 380/385 (Device Tree)
[11964.196925] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[11964.204687] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[11964.212279] [<c05e844c>] (dump_stack_lvl) from [<c012d0b8>] (__warn+0xa4/0xbc)
[11964.219527] [<c012d0b8>] (__warn) from [<c012d130>] (warn_slowpath_fmt+0x60/0xb8)
[11964.227032] [<c012d130>] (warn_slowpath_fmt) from [<c09f7f24>] (rcu_nmi_exit+0xd8/0x110)
[11964.235147] [<c09f7f24>] (rcu_nmi_exit) from [<c017f2dc>] (handle_domain_irq+0x60/0x78)
[11964.243177] [<c017f2dc>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[11964.251378] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[11964.258881] Exception stack(0xc0f01f08 to 0xc0f01f50)
[11964.263945] 1f00: 00c40c56 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[11964.272142] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c33b603c c0f01f58
[11964.280337] 1f40: c0107a70 c0107a74 60000013 ffffffff
[11964.285399] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[11964.292818] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[11964.300935] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[11964.308705] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[11964.316296] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[11964.324499] ---[ end trace 85cc7a5fb09f7a35 ]---
[11964.329126] ------------[ cut here ]------------
[11964.333751] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:1000 rcu_nmi_enter+0xac/0xb8
[11964.341514] Modules linked in: sctp udp_tunnel ip6_udp_tunnel
[11964.347277] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 5.15.147 #0
[11964.354778] Hardware name: Marvell Armada 380/385 (Device Tree)
[11964.360710] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[11964.368475] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[11964.376065] [<c05e844c>] (dump_stack_lvl) from [<c012d0b8>] (__warn+0xa4/0xbc)
[11964.383309] [<c012d0b8>] (__warn) from [<c012d130>] (warn_slowpath_fmt+0x60/0xb8)
[11964.390814] [<c012d130>] (warn_slowpath_fmt) from [<c09f800c>] (rcu_nmi_enter+0xac/0xb8)
[11964.398927] [<c09f800c>] (rcu_nmi_enter) from [<c0132a0c>] (irq_enter+0x8/0x10)
[11964.406259] [<c0132a0c>] (irq_enter) from [<c017f2a4>] (handle_domain_irq+0x28/0x78)
[11964.414026] [<c017f2a4>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[11964.422227] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[11964.429729] Exception stack(0xc0f01f08 to 0xc0f01f50)
[11964.434792] 1f00: 00c40c56 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[11964.442989] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c33b603c c0f01f58
[11964.451184] 1f40: c0107a70 c0107a74 60000013 ffffffff
[11964.456246] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[11964.463663] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[11964.471778] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[11964.479545] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[11964.487136] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[11964.495338] ---[ end trace 85cc7a5fb09f7a36 ]---
[11964.499992] ------------[ cut here ]------------
tuomaz
February 28, 2024, 11:19am
14
Now it crashed with:
[64865.337745] Internal error: Oops - undefined instruction: 0 [#1] SMP ARM
[64865.344476] Modules linked in: sctp udp_tunnel ip6_udp_tunnel
[64865.350244] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 5.15.147 #0
[64865.357747] Hardware name: Marvell Armada 380/385 (Device Tree)
[64865.363678] PC is at rcu_nmi_enter+0x78/0xb8
[64865.367962] LR is at rcu_nmi_enter+0x68/0xb8
[64865.372241] pc : [<c09f7fd8>] lr : [<c09f7fc8>] psr: 60000193
[64865.378521] sp : c1083f08 ip : c107c03c fp : 00000000
[64865.383757] r10: c0e580a8 r9 : c0e57ccc r8 : f080210c
[64865.388993] r7 : 0000001d r6 : c100c000 r5 : c1082000 r4 : eede06c0
[64865.395534] r3 : 00000001 r2 : 00000000 r1 : 00000000 r0 : 03df2aa7
[64865.402075] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment none
[64865.409316] Control: 10c5387d Table: 033ac04a DAC: 00000051
[64865.415073] Register r0 information: non-paged memory
[64865.420137] Register r1 information: NULL pointer
[64865.424851] Register r2 information: NULL pointer
[64865.429564] Register r3 information: non-paged memory
[64865.434626] Register r4 information: non-slab/vmalloc memory
[64865.440298] Register r5 information: non-slab/vmalloc memory
[64865.445970] Register r6 information: slab kmalloc-1k start c100c000 pointer offset 0 size 1024
[64865.454609] Register r7 information: non-paged memory
[64865.459671] Register r8 information: 0-page vmalloc region starting at 0xf0802000 allocated at of_iomap+0x48/0x64
[64865.469965] Register r9 information: non-slab/vmalloc memory
[64865.475637] Register r10 information: non-slab/vmalloc memory
[64865.481397] Register r11 information: NULL pointer
[64865.486198] Register r12 information: non-slab/vmalloc memory
[64865.491956] Process swapper/1 (pid: 0, stack limit = 0x5cec192b)
[64865.497978] Stack: (0xc1083f08 to 0xc1084000)
[64865.502346] 3f00: c0e57cc0 00000000 c100c000 c0132a0c c0e57cc0 c017f2a4
[64865.510543] 3f20: c0f05404 c1083f58 c0f85970 f0802100 f080210c c01012e4 c0107a74 60000013
[64865.518740] 3f40: ffffffff c1083f8c 00000000 c1082000 c0e580a8 c0100b7c 03df2aa6 00000000
[64865.526936] 3f60: 00000001 c0117cc0 ffffe000 c0f04f54 c0f04f98 00000002 00000000 c0f04f08
[64865.535133] 3f80: c0e580a8 00000000 c107c03c c1083fa8 c0107a70 c0107a74 60000013 ffffffff
[64865.543330] 3fa0: 00000051 c09f9d40 ffffe000 c09ff950 ffffe000 c015c8b4 10c0387d c0f10514
[64865.551526] 3fc0: 0000406a b9b29dca 00000000 00000098 00000051 10c0387d c1083ff8 0000406a
[64865.559723] 3fe0: 414fc091 00000000 00000000 c015cbe4 010d806a 00101790 00000000 00000000
[64865.567920] [<c09f7fd8>] (rcu_nmi_enter) from [<c0132a0c>] (irq_enter+0x8/0x10)
[64865.575254] [<c0132a0c>] (irq_enter) from [<c017f2a4>] (handle_domain_irq+0x28/0x78)
[64865.583025] [<c017f2a4>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[64865.591226] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[64865.598729] Exception stack(0xc1083f58 to 0xc1083fa0)
[64865.603792] 3f40: 03df2aa6 00000000
[64865.611988] 3f60: 00000001 c0117cc0 ffffe000 c0f04f54 c0f04f98 00000002 00000000 c0f04f08
[64865.620185] 3f80: c0e580a8 00000000 c107c03c c1083fa8 c0107a70 c0107a74 60000013 ffffffff
[64865.628380] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[64865.635800] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[64865.643918] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[64865.651688] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[64865.659280] [<c015cbe4>] (cpu_startup_entry) from [<00101790>] (0x101790)
[64865.666090] Code: e3a03001 e1a0200d e0055002 e5952004 (eafffff4)
[64865.672198] ---[ end trace 85cc7a5fb09f7a39 ]---
[64865.676826] Kernel panic - not syncing: Fatal exception
[64865.682063] CPU0: stopping
[64865.684779] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D W 5.15.147 #0
[64865.692282] Hardware name: Marvell Armada 380/385 (Device Tree)
[64865.698216] [<c010ec68>] (unwind_backtrace) from [<c010aa44>] (show_stack+0x10/0x14)
[64865.705985] [<c010aa44>] (show_stack) from [<c05e844c>] (dump_stack_lvl+0x40/0x4c)
[64865.713578] [<c05e844c>] (dump_stack_lvl) from [<c010ce88>] (do_handle_IPI+0xf8/0x12c)
[64865.721517] [<c010ce88>] (do_handle_IPI) from [<c010ced4>] (ipi_handler+0x18/0x20)
[64865.729107] [<c010ced4>] (ipi_handler) from [<c0184e44>] (handle_percpu_devid_irq+0x78/0x12c)
[64865.737656] [<c0184e44>] (handle_percpu_devid_irq) from [<c017f2d8>] (handle_domain_irq+0x5c/0x78)
[64865.746642] [<c017f2d8>] (handle_domain_irq) from [<c01012e4>] (gic_handle_irq+0x7c/0x90)
[64865.754842] [<c01012e4>] (gic_handle_irq) from [<c0100b7c>] (__irq_svc+0x5c/0x78)
[64865.762344] Exception stack(0xc0f01f08 to 0xc0f01f50)
[64865.767408] 1f00: 041accb6 00000000 00000001 c0117cc0 ffffe000 c0f04f54
[64865.775605] 1f20: c0f04f98 00000001 00000000 c0f04f08 c0e580a8 00000000 c3aa003c c0f01f58
[64865.783802] 1f40: c0107a70 c0107a74 60000013 ffffffff
[64865.788863] [<c0100b7c>] (__irq_svc) from [<c0107a74>] (arch_cpu_idle+0x38/0x3c)
[64865.796280] [<c0107a74>] (arch_cpu_idle) from [<c09ff950>] (default_idle_call+0x2c/0x3c)
[64865.804394] [<c09ff950>] (default_idle_call) from [<c015c8b4>] (do_idle+0x1d8/0x1e0)
[64865.812160] [<c015c8b4>] (do_idle) from [<c015cbe4>] (cpu_startup_entry+0x18/0x1c)
[64865.819751] [<c015cbe4>] (cpu_startup_entry) from [<c0e0124c>] (start_kernel+0x738/0x76c)
[64865.827956] Rebooting in 3 seconds..
This does not look good
mbehun
March 6, 2024, 9:22am
15
This issue seems to be related to new DDR3 training code in the new firmware. It seems that Marvell, the manufacturer of the Armada 385 SoC in Turris Omnia, has broken their DDR3 training code several years ago and has been unable to fix the issue still.
I have been working on a solution where both the old and new DDR3 training implementations will be built into the U-Boot bootloader, and users experiencing these issues will be able to tell U-Boot to use the old code via one command and a reboot.
It should be available today, I will give information on how to try it later.
6 Likes
tuomaz
March 6, 2024, 9:06pm
16
Thank you very much for all your efforts on this error. I’m looking forward to try the fix.
A question: this does not seem to affect all users, what could be the reason for this?
1 Like
mbehun
March 7, 2024, 8:59am
17
@tuomaz the DDR training is different for every board, it can train a little bit differently even for each boot of the device. It depends on the DDR chips (each DDR chip can be a little bit different internally), on how they are soldered onto the board, on the traces of each PCB.
Could you please try switching branch to hbl with switch-branch hbl? After this wait some time (like 15 minutes) because it schedules a pkgupdate to update all packages. If you already are on the hbl branch, just do a pkgupdate.
Afterwards, install the omnia-uboot packages with opkg install omnia-uboot. Chech whether you have the newest version, with md5sum /usr/share/omnia/uboot-devel. It should print the checksum e2cb89f4a975952073e71beba2b5fa18.
If this checksum is correct, do a nor-update -d to update U-Boot. The new U-Boot supports old DDR training, you can select it from TurrisOS with
fw_setenv omnia_ddr3_training old
reboot
Alternatively, if you have serial connection available to the board, you can also reboot the board into U-Boot and execute the following commands in U-Boot command line
env set omnia_ddr3_training old
env save
reset
If you have the serial connection, please send the output of U-Boot after doing this.
tuomaz
March 7, 2024, 9:18pm
18
Hi. I followed your instructions above, using the tools from within Turris OS. Now my omnia wont boot as it did:
BootROM - 1.73
Booting from SPI flash
U-Boot SPL 2024.01-OpenWrt-r20293+119-4432454037 (Mar 07 2024 - 04:53:41 +0000)
High speed PHY - Version: 2.0
MiniPCIe/mSATA card detection... mSATA
WWAN slot configuration... PCIe+USB2.0
Detected Device ID 6820
board SerDes lanes topology details:
| Lane # | Speed | Type |
--------------------------------
| 0 | 6 | SATA0 |
| 1 | 5 | USB3 HOST0 |
| 2 | 5 | PCIe1 |
| 3 | 5 | USB3 HOST1 |
| 4 | 5 | PCIe2 |
| 5 | 0 | SGMII2 |
--------------------------------
High speed PHY - Ended Successfully
Using old DDR3 training implementation
DDR3 Training Sequence - Ver TIP-1.29.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
Trying to boot from SPI
U-Boot 2024.01-OpenWrt-r20293+119-4432454037 (Mar 07 2024 - 04:53:41 +0000), Build: jenkins-TurrisOS-packages-hbl-omnia-768
SoC: MV88F6820-A0 at 1600 MHz
DRAM: 2 GiB (800 MHz, 32-bit, 2T, ECC not enabled)
Core: 84 devices, 28 uclasses, devicetree: separate
WDT: Started watchdog@20300 with servicing every 1000ms (60s timeout)
MMC: mv_sdh: 0
Loading Environment from SPIFlash... SF: Detected s25fl164k with page size 256 Bytes, erase size 4 KiB, total 8 MiB
OK
Model: Turris Omnia
MCU type: STM32
MCU version: b5a8a24e007eb72be16aeb3fff6f03ec647023e4/67eddc9540526d0a9d9660f7a7867af9a28a68d6
RAM size: 2048 MiB
Board version: unknown
Serial Number: 0000000B0000EC80
Disabling MCU watchdog... disabled
Regdomain set to **
pcie1.0: Link down
pcie2.0: Link down
Net: eth0: ethernet@70000, eth1: ethernet@30000, eth2: ethernet@34000, eth3: lan0, eth4: lan1, eth5: lan2, eth6: lan3, eth7: lan4
Hit any key to stop autoboot: 0
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
BOOTP broadcast 4
BOOTP broadcast 5
BOOTP broadcast 6
BOOTP broadcast 7
BOOTP broadcast 8
BOOTP broadcast 9
BOOTP broadcast 10
BOOTP broadcast 11
BOOTP broadcast 12
BOOTP broadcast 13
BOOTP broadcast 14
BOOTP broadcast 15
BOOTP broadcast 16
BOOTP broadcast 17
BOOTP broadcast 18
Abort
Wrong Image Format for bootm command
ERROR: can't get kernel image!
Did I do something wrong?
mbehun
March 8, 2024, 10:11am
19
@tuomaz do you know what was your previous version of U-Boot?
Anyway, try to do the following: when U-Boot prints Hit any key to stop autoboot:, hit any key, it wil get you to U-Boot command prompt. Then execute the following commands:
env default -a
env set omnia_ddr3_training old
env save
reset
Does this help?
tuomaz:
cat /proc/meminfo
Interesting, i have the same values at the swap part with the omnia, but with the MOX the swap part is there?
