Issues with some devices after 3.11.13 upgrade

Software SW bugs discussion
1

Hi all,

I’m not 100% sure but it looks like that I started having some issues in my network (TO based) after 3.11.13 upgrade. What is happening:

  • my daughter has Sonos One speaker with Alexa enabled. Suddenly Alexa stopped working. When we tried to reset the device and enable Alexa again, the authentication flow between Sonos and Alexa from the device doesn’t work
  • LG TV cannot connect to the LG Content store claiming that the network is not stable. It has IP address properly assigned and it’s connected to the network
  • On the same LG TV I found that the HbbTv stopped working too
  • some devices connected over wifi sometimes claim - no internet access even when internet works fine for other device connected to the same wifi network
  • The Cloud Backup stopped working with “Failed to connect to the ssbackup server” error message

I cannot see anything in the logs and I’m running out of ideas. I tried to change the upstream DNS server but that didn’t help. It’s also more difficult to debug the issue as these devices like Sonos speaker, LG TV do not have proper logging and the error messages are too generic.

Any idea on what to check / what could be wrong? Thanks a lot for any hint.

Radek

2

First thing I’d try is to reset to 3.11.12 and see if problems are gone.

3

thanks - I’ll try it later today - I totally forgot about the rollback option.

4

Rollback didn’t help so it seems to be only coincidence with 3.11.13. I tried to uninstall Pakon, disabled DNSSEC and still no luck. From the LG TV some applications (like Youtube) work. The only observation is that LG applications are hosted on Amazon AWS as well as Amazon Alexa.

I’m not network expert - but I tried to use tcpdump on traffic from the LG TV (192.168.12.90) and it seems that there is more requests from the TV and just a few responses. I’m not sure if the traffic can be blocked by the firewall on the router or by ISP …

root@omnia:/etc/config# tcpdump -i br-lan host 192.168.12.90 and host not 192.168.12.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
21:51:08.262335 IP 192.168.12.90.50065 > 192.168.13.1.53: 8602+ A? cz.lgrecommends.lgappstv.com. (46)
21:51:13.267578 IP 192.168.12.90.50065 > 192.168.13.1.53: 7353+ A? cz.lgrecommends.lgappstv.com. (46)
21:51:13.791518 IP 192.168.12.90.50065 > 192.168.13.1.53: 19114+ A? cz.lgrecommends.lgappstv.com. (46)
21:51:14.262822 IP 192.168.12.90.50065 > 192.168.13.1.53: 32521+ A? cz.lgrecommends.lgappstv.com. (46)
21:51:15.139702 IP 192.168.13.1.53 > 192.168.12.90.50065: 32521 3/0/0 CNAME cz-lgrecommends-lgappstv-com.aws-prd.net., A 34.253.86.226, A 52.210.163.69 (132)
21:51:15.143519 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [S], seq 2793692832, win 29200, options [mss 1460,sackOK,TS val 649135 ecr 0,nop,wscale 7], length 0
21:51:15.146061 IP 192.168.13.1.53 > 192.168.12.90.50065: 7353 3/0/0 CNAME cz-lgrecommends-lgappstv-com.aws-prd.net., A 34.253.86.226, A 52.210.163.69 (132)
21:51:15.159498 IP 192.168.13.1.53 > 192.168.12.90.50065: 19114 3/0/0 CNAME cz-lgrecommends-lgappstv-com.aws-prd.net., A 34.253.86.226, A 52.210.163.69 (132)
21:51:15.159746 IP 192.168.13.1.53 > 192.168.12.90.50065: 8602 3/0/0 CNAME cz-lgrecommends-lgappstv-com.aws-prd.net., A 34.253.86.226, A 52.210.163.69 (132)
21:51:15.190985 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [S.], seq 206132339, ack 2793692833, win 26847, options [mss 1460,sackOK,TS val 3078164749 ecr 649135,nop,wscale 8], length 0
21:51:15.193770 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], ack 1, win 229, options [nop,nop,TS val 649140 ecr 3078164749], length 0
21:51:15.194701 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [P.], seq 1:518, ack 1, win 229, options [nop,nop,TS val 649140 ecr 3078164749], length 517
21:51:15.241876 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [.], ack 518, win 110, options [nop,nop,TS val 3078164813 ecr 649140], length 0
21:51:15.250146 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [P.], seq 1:1461, ack 518, win 110, options [nop,nop,TS val 3078164819 ecr 649140], length 1460
21:51:15.250401 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [P.], seq 1461:2921, ack 518, win 110, options [nop,nop,TS val 3078164819 ecr 649140], length 1460
21:51:15.250644 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [P.], seq 2921:4023, ack 518, win 110, options [nop,nop,TS val 3078164819 ecr 649140], length 1102
21:51:15.253092 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], ack 1449, win 251, options [nop,nop,TS val 649146 ecr 3078164819], length 0
21:51:15.253094 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], ack 1461, win 251, options [nop,nop,TS val 649146 ecr 3078164819], length 0
21:51:15.253095 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], ack 2909, win 274, options [nop,nop,TS val 649146 ecr 3078164819], length 0
21:51:15.253097 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], ack 2921, win 274, options [nop,nop,TS val 649146 ecr 3078164819], length 0
21:51:15.253098 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], ack 4023, win 296, options [nop,nop,TS val 649146 ecr 3078164819], length 0
21:51:15.259790 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [P.], seq 518:652, ack 4023, win 296, options [nop,nop,TS val 649146 ecr 3078164819], length 134
21:51:15.301411 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [P.], seq 4023:4321, ack 652, win 114, options [nop,nop,TS val 3078164872 ecr 649146], length 298
21:51:15.305642 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [P.], seq 652:2214, ack 4321, win 319, options [nop,nop,TS val 649151 ecr 3078164872], length 1562
21:51:15.376862 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [.], ack 652, win 118, options [nop,nop,TS val 3078164935 ecr 649146,nop,nop,sack 1 {2100:2214}], length 0
21:51:15.390312 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4321, win 319, options [nop,nop,TS val 649160 ecr 3078164935], length 1448
21:51:15.650419 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4321, win 319, options [nop,nop,TS val 649186 ecr 3078164935], length 1448
21:51:16.170428 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4321, win 319, options [nop,nop,TS val 649238 ecr 3078164935], length 1448
21:51:17.210537 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4321, win 319, options [nop,nop,TS val 649342 ecr 3078164935], length 1448
21:51:19.290568 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4321, win 319, options [nop,nop,TS val 649550 ecr 3078164935], length 1448
21:51:23.460563 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4321, win 319, options [nop,nop,TS val 649967 ecr 3078164935], length 1448
21:51:25.320425 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [P.], seq 4321:4358, ack 652, win 118, options [nop,nop,TS val 3078174878 ecr 649146,nop,nop,sack 1 {2100:2214}], length 37
21:51:25.320680 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [F.], seq 4358, ack 652, win 118, options [nop,nop,TS val 3078174878 ecr 649146,nop,nop,sack 1 {2100:2214}], length 0
21:51:25.323585 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [F.], seq 2214, ack 4359, win 319, options [nop,nop,TS val 650153 ecr 3078174878], length 0
21:51:25.358546 IP ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443 > 192.168.12.90.40158: Flags [.], ack 652, win 118, options [nop,nop,TS val 3078174930 ecr 649146,nop,nop,sack 1 {2100:2215}], length 0
21:51:25.361448 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 650157 ecr 3078174930], length 1448
21:51:25.620823 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 650183 ecr 3078174930], length 1448
21:51:26.140761 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 650235 ecr 3078174930], length 1448
21:51:27.180806 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 650339 ecr 3078174930], length 1448
21:51:29.260910 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 650547 ecr 3078174930], length 1448
21:51:33.430909 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 650964 ecr 3078174930], length 1448
21:51:41.771207 IP 192.168.12.90.40158 > ec2-34-253-86-226.eu-west-1.compute.amazonaws.com.443: Flags [.], seq 652:2100, ack 4359, win 319, options [nop,nop,TS val 651798 ecr 3078174930], length 1448
5

really strange - I played more - I changed MTU on WAN and LAN to 1482 and all the services started to work. When I changed it back some of them worked but not all so I changed it back to 1482. I wonder if ISP made any changes.

6

Do you have a xDSL subscriber line? If so enable clamping of TCP MSS to Path MTU in the firewall for the WAN

config zone
	option name 'wan'
	option mtu_fix '1'

restart firewall.

7

I’m not on xDSL - using a wireless connection (not sure about technology now) from Rio Media (they were bought by Nej.cz and that’s kind of suspicion for changes in the ISP side)

I had MSS clamping enabled already before. Here’s the wan zone configuration:

config zone
        option name 'wan'
        option network 'wan6 wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
8

Just looked up Nej.cz to get an idea of your upstream connectivity but it is not clear how your upstream connectivity is set up (least the manual section only showing set-top boxes and modems METALICKÁ GW ELCON – EAD 100 and OPTICKÁ GW ELCON – FOS 100) , maybe you could elaborate?

  • ISP → ISP box (which one) → Wlan → TO?
  • ISP → LTE modem in TO?
9

Well, they do not provide many details even when I log in to their system … They even claim that they do not have any service in my area.

The topology is:

TO (WAN Ethernet port) -> (ISP) Wireless/router (I believe kind of Mikrotik) connected over ethernet cable -> (ISP) Wireless AP in Zdiby -> not sure how they connect further.

10

That is a tricky one with the ISP box connecting to the ISP’s AP, potentially with some sort of encryption (at least one would hope)…

802.11 specifies MTU max. 2312 bytes for Wlan but then all kind of header, encryption and so on take a bite from it. And you would not know whether the ISP box sets a specific MTU.

Common Linux tool to check the path MTU from a host is tracepath (not sure whether available on TOS3.x) and mturoute for Windows nodes - if you want to investigate.

Else, perhaps best to check with your ISP support about the issue, since it does not seem inflicted by the TO hardware or the TOS3.x (well there could be a bug that TOS4.x may or may not remedy), or stay with the solution you already discovered.

Suspect some issue with packet fragmentation which could be analysed from a packet dump by a tool like wireshark

11

To close the thread - I got confirmation from ISP that my location run on backup connection and MTU was lower on the backup line …