This has been happening ever since upgrade to 5.0. The router works just fine and then all of a sudden eth2 just goes down:
[34513.745994] mvneta f1034000.ethernet eth2: bad rx status 0f810000 (crc error), size=1520
[34515.390512] mvneta f1034000.ethernet eth2: bad rx status 0f810000 (crc error), size=1520
[34515.522810] mvneta f1034000.ethernet eth2: bad rx status 0f810000 (crc error), size=1520
[34515.819669] mvneta f1034000.ethernet eth2: bad rx status 0f810000 (crc error), size=1520
[34518.418313] mvneta f1034000.ethernet eth2: Link is Down
[34518.478812] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[34518.595627] mvneta f1034000.ethernet eth2: PHY [f1072004.mdio-mii:01] driver [Marvell 88E1510]
[34518.595836] mvneta f1034000.ethernet eth2: configuring for phy/sgmii link mode
The CRC errors happen a lot as well but I don’t know if they are an issue or just a red herring. Like I’ve said, my setup used to work perfectly fine before and the only thing that changed is upgrade to Omnia 5.
After some combination of ip link set dev eth2 down, ip link set dev eth2 up, /etc/init.d/odhcpd restart and unplugging and plugging the network cable the connection recovers.
This used to be just mild annoyance happening perhaps a few times a week but over couple of days this has been happening nearly every couple hours.
Does anyone have ideas how to troubleshoot it? Or better yet how to fix it?
At one point I’ve caught what looks like kernel bug:
[omnia]# ip link set eth2 up
RTNETLINK answers: Out of memory
[omnia]# dmesg -c
[278663.224062] ip: page allocation failure: order:5, mode:0x14000c0(GFP_KERNEL), nodemask=(null)
[278663.224079] ip cpuset=/ mems_allowed=0
[278663.224089] CPU: 0 PID: 9690 Comm: ip Not tainted 4.14.292 #0
[278663.224091] Hardware name: Marvell Armada 380/385 (Device Tree)
[278663.224109] [<c011004c>] (unwind_backtrace) from [<c010b3c4>] (show_stack+0x10/0x14)
[278663.224119] [<c010b3c4>] (show_stack) from [<c07e1858>] (dump_stack+0x94/0xa8)
[278663.224129] [<c07e1858>] (dump_stack) from [<c01e73d4>] (warn_alloc+0xd0/0x190)
[278663.224135] [<c01e73d4>] (warn_alloc) from [<c01e81f0>] (__alloc_pages_nodemask+0xd5c/0xf0c)
[278663.224141] [<c01e81f0>] (__alloc_pages_nodemask) from [<c01158e0>] (__dma_alloc_buffer.constprop.5+0x30/0x180)
[278663.224146] [<c01158e0>] (__dma_alloc_buffer.constprop.5) from [<c0115b1c>] (simple_allocator_alloc+0x1c/0x30)
[278663.224150] [<c0115b1c>] (simple_allocator_alloc) from [<c0115614>] (__dma_alloc+0x164/0x2b4)
[278663.224154] [<c0115614>] (__dma_alloc) from [<c0115798>] (arm_coherent_dma_alloc+0x34/0x3c)
[278663.224164] [<c0115798>] (arm_coherent_dma_alloc) from [<c061295c>] (mvneta_setup_txqs+0x2a4/0x458)
[278663.224170] [<c061295c>] (mvneta_setup_txqs) from [<c0612b64>] (mvneta_open+0x54/0x2b4)
[278663.224180] [<c0612b64>] (mvneta_open) from [<c06c8134>] (__dev_open+0xbc/0x128)
[278663.224186] [<c06c8134>] (__dev_open) from [<c06c842c>] (__dev_change_flags+0xe8/0x190)
[278663.224192] [<c06c842c>] (__dev_change_flags) from [<c06c84ec>] (dev_change_flags+0x18/0x48)
[278663.224201] [<c06c84ec>] (dev_change_flags) from [<c06d8b0c>] (do_setlink+0x3f4/0xa20)
[278663.224207] [<c06d8b0c>] (do_setlink) from [<c06dcb1c>] (rtnl_newlink+0x3c0/0x800)
[278663.224212] [<c06dcb1c>] (rtnl_newlink) from [<c06d8650>] (rtnetlink_rcv_msg+0x260/0x2ac)
[278663.224220] [<c06d8650>] (rtnetlink_rcv_msg) from [<c06fccb4>] (netlink_rcv_skb+0x8c/0xfc)
[278663.224225] [<c06fccb4>] (netlink_rcv_skb) from [<c06fc454>] (netlink_unicast+0x15c/0x22c)
[278663.224229] [<c06fc454>] (netlink_unicast) from [<c06fc8d8>] (netlink_sendmsg+0x3b4/0x414)
[278663.224236] [<c06fc8d8>] (netlink_sendmsg) from [<c06a7498>] (___sys_sendmsg+0x21c/0x250)
[278663.224242] [<c06a7498>] (___sys_sendmsg) from [<c06a7d2c>] (__sys_sendmsg+0x50/0x88)
[278663.224250] [<c06a7d2c>] (__sys_sendmsg) from [<c0107940>] (ret_fast_syscall+0x0/0x54)
[278663.224252] Mem-Info:
[278663.224260] active_anon:69336 inactive_anon:5905 isolated_anon:0
[278663.224260] active_file:36661 inactive_file:61510 isolated_file:0
[278663.224260] unevictable:0 dirty:28 writeback:12 unstable:0
[278663.224260] slab_reclaimable:4575 slab_unreclaimable:7047
[278663.224260] mapped:7362 shmem:6508 pagetables:665 bounce:0
[278663.224260] free:60004 free_pcp:40 free_cma:0
[278663.224266] Node 0 active_anon:277344kB inactive_anon:23620kB active_file:146644kB inactive_file:246040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:29448kB dirty:112kB writeback:48kB shmem:26032kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[278663.224273] Normal free:238188kB min:16384kB low:20480kB high:24576kB active_anon:173148kB inactive_anon:124kB active_file:123112kB inactive_file:148892kB unevictable:0kB writepending:124kB present:786432kB managed:766476kB mlocked:0kB kernel_stack:2688kB pagetables:2660kB bounce:0kB free_pcp:160kB local_pcp:0kB free_cma:0kB
[278663.224274] lowmem_reserve[]: 0 2048 2048
[278663.224285] HighMem free:1828kB min:256kB low:1656kB high:3056kB active_anon:104128kB inactive_anon:23496kB active_file:23532kB inactive_file:97244kB unevictable:0kB writepending:4kB present:262144kB managed:262144kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[278663.224286] lowmem_reserve[]: 0 0 0
[278663.224291] Normal: 10429*4kB (UME) 10448*8kB (UME) 4436*16kB (UME) 1050*32kB (UME) 135*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 238516kB
[278663.224310] HighMem: 126*4kB (UM) 51*8kB (U) 50*16kB (U) 11*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2128kB
[278663.224329] 104669 total pagecache pages
[278663.224331] 0 pages in swap cache
[278663.224333] Swap cache stats: add 0, delete 0, find 0/0
[278663.224335] Free swap = 0kB
[278663.224336] Total swap = 0kB
[278663.224337] 262144 pages RAM
[278663.224338] 65536 pages HighMem/MovableOnly
[278663.224340] 4989 pages reserved
[278663.224350] mvneta f1034000.ethernet eth2: mvneta_setup_txqs: can't create txq=1
[278663.231893] ------------[ cut here ]------------
[278663.231899] WARNING: CPU: 0 PID: 9690 at arch/arm/mm/dma-mapping.c:903 __arm_dma_free.constprop.3+0x108/0x150
[278663.231901] Freeing invalid buffer e0778000
[278663.231903] Modules linked in: sch_cake ath9k ath9k_common qcserial pppoe ppp_async option ath9k_hw ath10k_pci ath10k_core ath usb_wwan snd_usb_audio rndis_host qmi_wwan pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute cfg80211 cdc_ether at24 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_quota xt_pkttype xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT xt_NFLOG xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial usbnet ums_usbat ums_sddr55 ums_sddr09 ums_karma ums_jumpshot ums_isd200 ums_freecom ums_datafab ums_cypress
[278663.231997] ums_alauda ts_fsm ts_bm snd_usbmidi_lib slhc nvmem_core nfnetlink_log nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat cdc_wdm br_netfilter fuse sch_tbf sch_ingress
[278663.232072] sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred configs i2c_mux_pca954x i2c_mux ledtrig_oneshot ledtrig_heartbeat ledtrig_gpio cryptodev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsd nfs msdos ip_gre gre ifb sit ip6_tunnel tunnel6 tunnel4 ip_tunnel rpcsec_gss_krb5 auth_rpcgss oid_registry veth tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm snd_timer
[278663.232154] snd_mixer_oss snd_hwdep snd_compress snd soundcore ntfs lockd sunrpc grace hfsplus hfs cifs autofs4 dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax multipath raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid10 raid1 raid0 linear md_mod nls_utf8 nls_koi8_r nls_cp1255 nls_iso8859_6 nls_iso8859_2 nls_iso8859_15 nls_iso8859_13 nls_cp932 nls_cp866 nls_cp864 nls_cp862 nls_cp852 nls_cp850 nls_cp775 nls_cp1251 nls_cp1250 xts algif_skcipher algif_hash af_alg sha512_generic sha256_generic seqiv jitterentropy_rng drbg pcbc md5 md4 hmac gf128mul ecb cts ctr cmac ccm cbc authenc vfat fat nls_iso8859_1 nls_cp437 uas ledtrig_transient xfs libcrc32c exfat mii
[278663.232252] CPU: 0 PID: 9690 Comm: ip Not tainted 4.14.292 #0
[278663.232254] Hardware name: Marvell Armada 380/385 (Device Tree)
[278663.232261] [<c011004c>] (unwind_backtrace) from [<c010b3c4>] (show_stack+0x10/0x14)
[278663.232268] [<c010b3c4>] (show_stack) from [<c07e1858>] (dump_stack+0x94/0xa8)
[278663.232277] [<c07e1858>] (dump_stack) from [<c012dfb4>] (__warn+0xe4/0x100)
[278663.232283] [<c012dfb4>] (__warn) from [<c012e018>] (warn_slowpath_fmt+0x48/0x6c)
[278663.232289] [<c012e018>] (warn_slowpath_fmt) from [<c0115364>] (__arm_dma_free.constprop.3+0x108/0x150)
[278663.232296] [<c0115364>] (__arm_dma_free.constprop.3) from [<c061251c>] (mvneta_cleanup_txqs+0x1c8/0x290)
[278663.232302] [<c061251c>] (mvneta_cleanup_txqs) from [<c0612ad4>] (mvneta_setup_txqs+0x41c/0x458)
[278663.232307] [<c0612ad4>] (mvneta_setup_txqs) from [<c0612b64>] (mvneta_open+0x54/0x2b4)
[278663.232314] [<c0612b64>] (mvneta_open) from [<c06c8134>] (__dev_open+0xbc/0x128)
[278663.232320] [<c06c8134>] (__dev_open) from [<c06c842c>] (__dev_change_flags+0xe8/0x190)
[278663.232325] [<c06c842c>] (__dev_change_flags) from [<c06c84ec>] (dev_change_flags+0x18/0x48)
[278663.232331] [<c06c84ec>] (dev_change_flags) from [<c06d8b0c>] (do_setlink+0x3f4/0xa20)
[278663.232337] [<c06d8b0c>] (do_setlink) from [<c06dcb1c>] (rtnl_newlink+0x3c0/0x800)
[278663.232343] [<c06dcb1c>] (rtnl_newlink) from [<c06d8650>] (rtnetlink_rcv_msg+0x260/0x2ac)
[278663.232348] [<c06d8650>] (rtnetlink_rcv_msg) from [<c06fccb4>] (netlink_rcv_skb+0x8c/0xfc)
[278663.232353] [<c06fccb4>] (netlink_rcv_skb) from [<c06fc454>] (netlink_unicast+0x15c/0x22c)
[278663.232357] [<c06fc454>] (netlink_unicast) from [<c06fc8d8>] (netlink_sendmsg+0x3b4/0x414)
[278663.232363] [<c06fc8d8>] (netlink_sendmsg) from [<c06a7498>] (___sys_sendmsg+0x21c/0x250)
[278663.232368] [<c06a7498>] (___sys_sendmsg) from [<c06a7d2c>] (__sys_sendmsg+0x50/0x88)
[278663.232374] [<c06a7d2c>] (__sys_sendmsg) from [<c0107940>] (ret_fast_syscall+0x0/0x54)
[278663.232377] ---[ end trace 2f633faf3e43dbdf ]---
[278663.232384] ------------[ cut here ]------------