SSH logging and diagnosis?

Software SW help
1

Context:

I have port 22 open on the firewall for ssh access from work. It open only to connections from the WAN IP address at work. And that has worked reliably for many years.

Recently at work we swapped ISPs. So have a new WAN IP. So Putty, which I use to connect of course rejects me:

image

Easy fixed I figure. I get our new wan IP by running from my desktop at work:

 curl http://ifconfig.me/ip

and then at home I go to Luci, the Firewall and find the rule and I edit it to update the source IP address in the rule to the new one I got from work.

I go to work, and I still get the error above.

I check it again and double check by surfing to:

What Is My IP Address - See Your Public Address - IPv4 & IPv6

It sees the same. I message myself the IP and at home I double check and yes it’s perfectly right.

I reboot the Omnia for good measure, and check again and it’s right.

I go back to work and I still get this connection refusal.

So back home I check logs and I can’t find any ssh log. I haven’t searched through messages yet, or any other logs, but what I’m after is a to configure ssh logging as needed and to see an explicit ssh log if possible in which I can see the connection attempt if possible with the IP it came from and diagnose the failure - this may involve firewall configuration to log the source IP of rejected connection attempts etc.

I haven’t as yet done exhaustive reading on how to do all that, which I’ll tackle in coming days I imagine, but I figured I’d ask quickly here in case someone has something to share that saves me a lot of time as I’m not particularly well versed on firewall configs and logging for example and have leant on Luci to configure all that in past and not had any issues I guess.

I am very comfortable at the bash prompt too mind you, no issue there.

3

Plase start the ssh client with option “-v” and another time with “-vv” and append the output into your post. To fetch the output into a handsome log file, You can add the option “-E ”.

Please also notice the SSH manual pages, depending on the platform and client You use:

4

Thanks for that. Totally forgot to check for client verbosity. I think because I started with putty, but by the by. Using ssh under WSL Ubuntu the anonymised -vv trace is:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "remote_server.com" port 22
debug1: Connecting to remote_server.com [remote_server.ip] port 22.
debug1: Connection established.
debug1: identity file /home/me/.ssh/id_rsa type -1
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/me/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519 type -1
debug1: identity file /home/me/.ssh/id_ed25519-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519_sk type -1
debug1: identity file /home/me/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/me/.ssh/id_xmss type -1
debug1: identity file /home/me/.ssh/id_xmss-cert type -1
debug1: identity file /home/me/.ssh/id_dsa type -1
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10
kex_exchange_identification: read: Connection reset by peer
Connection reset by remote_server.ip port 22OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "remote_server.com" port 22
debug1: Connecting to remote_server.com [remote_server.ip] port 22.
debug1: Connection established.
debug1: identity file /home/me/.ssh/id_rsa type -1
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/me/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519 type -1
debug1: identity file /home/me/.ssh/id_ed25519-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519_sk type -1
debug1: identity file /home/me/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/me/.ssh/id_xmss type -1
debug1: identity file /home/me/.ssh/id_xmss-cert type -1
debug1: identity file /home/me/.ssh/id_dsa type -1
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10
kex_exchange_identification: read: Connection reset by peer
Connection reset by remote_server.ip port 22

If I read that right I my firewall is good indeed and letting the connection open but something changed at the client side re supported authentication or encryption methods perhaps that requires a change at the remote server side? The remote server being my Omnia :wink:

5

Ok, this gives a hint.
Regarding the key exchange: Maybe there is indeed a mismatch in the algorithms of both systems. Maybe there is some kind of unintended flow between both systems. Try to figure out which endpoint quits the “kex” for what reason.

You should use “-vvv” to trace a connection attempt from within Your own network and one from the offices network. Compare the sections of both log outputs when it comes to the “ssl key exchange” phase.
You can even enhance the log information by configuring the SSHD logging on the turris side. Be careful to not lock yourself out when parametrizing SSHD.

6

Good plan. In the office triple v, double v and single v all produce the same output (WSL Ubuntu under Win 11). But I’ll try it at home (later clearly as, because it fails, I can’t do it from office where I am now).

7

And an anonymised trace of triple v output from my LAN to the Omnia:

$ ssh -vvv root@my_omnia
OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/me/.ssh/config
debug3: kex names ok: [diffie-hellman-group1-sha1]
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/me/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/me/.ssh/known_hosts2'
debug2: resolving "my_omnia" port 22
debug3: resolve_host: lookup my_omnia:22
debug3: ssh_connect_direct: entering
debug1: Connecting to my_omnia [192.168.0.1] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/me/.ssh/id_rsa type 0
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/me/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519 type 3
debug1: identity file /home/me/.ssh/id_ed25519-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519_sk type -1
debug1: identity file /home/me/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/me/.ssh/id_xmss type -1
debug1: identity file /home/me/.ssh/id_xmss-cert type -1
debug1: identity file /home/me/.ssh/id_dsa type -1
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to my_omnia:22 as 'root'
debug3: record_hostkey: found key type ECDSA in file /home/me/.ssh/known_hosts:6
debug3: load_hostkeys_file: loaded 1 keys from my_omnia
debug1: load_hostkeys: fopen /home/me/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp256
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,kex-strict-c-v00@openssh.com
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp256,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:xhs+Ofda3CwB8s3Lmdn5tKMY1XL4zJI1FTck9IFGk48
debug3: record_hostkey: found key type ECDSA in file /home/me/.ssh/known_hosts:6
debug3: load_hostkeys_file: loaded 1 keys from my_omnia
debug1: load_hostkeys: fopen /home/me/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'my_omnia' is known and matches the ECDSA host key.
debug1: Found key in /home/me/.ssh/known_hosts:6
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/me/.ssh/id_rsa RSA SHA256:PK0uUqcUhwWdfEiR/3qh6xVE+FoECBleVGf9gYUE+aE
debug1: Will attempt key: /home/me/.ssh/id_ecdsa 
debug1: Will attempt key: /home/me/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/me/.ssh/id_ed25519 ED25519 SHA256:Evv7IKhcM0j8ib36+a3LefikA7/8gQjsPy0GlaM5qbs
debug1: Will attempt key: /home/me/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/me/.ssh/id_xmss 
debug1: Will attempt key: /home/me/.ssh/id_dsa 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/me/.ssh/id_rsa RSA SHA256:PK0uUqcUhwWdfEiR/3qh6xVE+FoECBleVGf9gYUE+aE
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/me/.ssh/id_rsa RSA SHA256:PK0uUqcUhwWdfEiR/3qh6xVE+FoECBleVGf9gYUE+aE
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:PK0uUqcUhwWdfEiR/3qh6xVE+FoECBleVGf9gYUE+aE
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:PK0uUqcUhwWdfEiR/3qh6xVE+FoECBleVGf9gYUE+aE
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to my_omnia ([192.168.0.1]:22) using "publickey".
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: filesystem
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received RSA key SHA256:nrzFUYEGSj6Vi2VWM3GYxrJEntHKWAaubYUsRxq0yG8
debug3: client_input_hostkeys: received ECDSA key SHA256:xhs+Ofda3CwB8s3Lmdn5tKMY1XL4zJI1FTck9IFGk48
debug3: client_input_hostkeys: received ED25519 key SHA256:brX7fxRLwccb8hpisiTXSOsU7DnhwGVjQo4Lu2n+pQk
debug1: client_input_hostkeys: searching /home/me/.ssh/known_hosts for my_omnia / (none)
debug3: hostkeys_foreach: reading file "/home/me/.ssh/known_hosts"
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key under different name/addr at /home/me/.ssh/known_hosts:4
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key under different name/addr at /home/me/.ssh/known_hosts:5
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key at /home/me/.ssh/known_hosts:6
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key under different name/addr at /home/me/.ssh/known_hosts:11
debug1: client_input_hostkeys: searching /home/me/.ssh/known_hosts2 for my_omnia / (none)
debug1: client_input_hostkeys: hostkeys file /home/me/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 2 new, 18446744073709551615 retained, 2 incomplete match. 0 to remove
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug3: receive packet: type 4
debug1: Remote: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 4
debug1: Remote: /root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env QT_ACCESSIBILITY
debug3: Ignored env COLORTERM
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env XDG_SESSION_PATH
debug3: Ignored env GNOME_DESKTOP_SESSION_ID
debug3: Ignored env GTK_IM_MODULE
debug3: Ignored env LANGUAGE
debug1: channel 0: setting env LC_ADDRESS = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug1: channel 0: setting env LC_NAME = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env CINNAMON_VERSION
debug3: Ignored env XMODIFIERS
debug3: Ignored env DESKTOP_SESSION
debug1: channel 0: setting env LC_MONETARY = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env GTK_MODULES
debug3: Ignored env XDG_SEAT
debug3: Ignored env PWD
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_SESSION_DESKTOP
debug3: Ignored env QT_QPA_PLATFORMTHEME
debug3: Ignored env XDG_SESSION_TYPE
debug3: Ignored env CHROME_CONFIG_HOME
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env XAUTHORITY
debug3: Ignored env XDG_GREETER_DATA_DIR
debug3: Ignored env GDM_LANG
debug3: Ignored env HOME
debug1: channel 0: setting env LC_PAPER = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug1: channel 0: setting env LANG = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env VTE_VERSION
debug3: Ignored env KDEDIR
debug3: Ignored env XDG_SEAT_PATH
debug3: Ignored env GNOME_TERMINAL_SCREEN
debug3: Ignored env CLUTTER_IM_MODULE
debug3: Ignored env XDG_SESSION_CLASS
debug3: Ignored env TERM
debug1: channel 0: setting env LC_IDENTIFICATION = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env USER
debug3: Ignored env GNOME_TERMINAL_SERVICE
debug3: Ignored env DISPLAY
debug3: Ignored env SHLVL
debug1: channel 0: setting env LC_TELEPHONE = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env QT_IM_MODULE
debug1: channel 0: setting env LC_MEASUREMENT = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env XDG_VTNR
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env PAPERSIZE
debug3: Ignored env XDG_RUNTIME_DIR
debug1: channel 0: setting env LC_TIME = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env KDEDIRS
debug1: channel 0: setting env LC_COLLATE = "C"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env GTK3_MODULES
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env PATH
debug3: Ignored env GDMSESSION
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug1: channel 0: setting env LC_NUMERIC = "en_AU.UTF-8"
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

From which I think I’ve workied it out. And the spark of config memories rouses! All I needed was sufficiently informative traces ;-).

I’m not 100% sure but will test later this evening. The thing is for security root password login on the omnia is off. I only support keyed access and only from the source IP of my desktop at work. And that desktop was rebuilt recently. Lost my private key and so the Omia just closes and doens’t offer a login prompt. I just need to enable root login with password for a minut, log in and ttranfer a public key over from the work box (I have rdp access) nd then move over to keyed access. I give this a 95% chance of success but alas lack the time now to do.

8

My theory is wrong! On the omnia I see:

# cat /etc/ssh/sshd_config
# auto-generated config file from /etc/config/sshd

PermitRootLogin yes
AuthorizedKeysFile .ssh/authorized_keys
Subsystem sftp /usr/lib/sftp-server

whcih is odd as there is /etc/config/sshd

# ll /etc/config/sshd
ls: /etc/config/sshd: No such file or directory

for it to be based on. I see:

# netstat -tulpen | grep sshd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      12314/sshd
tcp        0      0 :::22                   :::*                    LISTEN      12314/sshd
# pgrep -a sshd
525 sshd: root@pts/0
12314 /usr/sbin/sshd
# cat /etc/init.d/sshd
#!/bin/sh /etc/rc.common
# Copyright (C) 2006-2011 OpenWrt.org

START=50
STOP=50

USE_PROCD=1
PROG=/usr/sbin/sshd

start_service() {
	for type in rsa ecdsa ed25519
	do
		# check for keys
		key=/etc/ssh/ssh_host_${type}_key
		[ ! -f $key ] && {
			# generate missing keys
			[ -x /usr/bin/ssh-keygen ] && {
				/usr/bin/ssh-keygen -N '' -t $type -f $key 2>&- >&-
			}
		}
	done
	mkdir -m 0700 -p /var/empty

	local lport=$(awk '/^Port / { print $2; exit }' /etc/ssh/sshd_config)
	[ -z "$lport" ] && lport=22

	procd_open_instance
	procd_add_mdns "ssh" "tcp" "$lport"
	procd_set_param command $PROG -D
	procd_close_instance
}

shutdown() {
	local pid

	stop

	# kill active clients
	for pid in $(pidof sshd)
	do
		[ "$pid" == "$$" ] && continue
		[ -e "/proc/$pid/stat" ] && kill $pid
	done
}

But the key is it looks like I never did set PermitRootLogin to prohibit-password, which I commonly do for security. So I’d expect the Omnia to present a login prompt. But it’s not. And so more diagnostics to do.

So I took a look in messages:

# grep sshd /var/log/messages
2024-07-22 16:19:41 info sshd[8763]: Received disconnect from 192.168.0.11 port 58468:11: disconnected by user
2024-07-22 16:19:41 info sshd[8763]: Disconnected from user root 192.168.0.11 port 58468
2024-07-22 16:19:54 info sshd[525]: Accepted publickey for root from 192.168.0.11 port 60090 ssh2: RSA SHA256:PK0uUqcUhwWdfEiR/3qh6xVE+FoECBleVGf9gYUE+aE
2024-07-22 18:37:33 info sshd[5394]: Received signal 15; terminating.
2024-07-22 18:37:33 info sshd[12314]: Server listening on 0.0.0.0 port 22.
2024-07-22 18:37:33 info sshd[12314]: Server listening on :: port 22.
```

which all about me on the LAN, I try to connect from the work machine (RDP, WSL Ubuntu ssh) and get the traces I know but nothing changes in messages. In fact I can do `tail -f /var/log/messages` try to connect and nothing is added. 

Not sure what other diags are available server side (on the Omnia) to detect the connection attempt. Because it does report:

```
debug1: Connection established.
...
Connection reset by remote_server.ip port 22
```

So it does appear to be talking to the Omnia The point of deviaton from a lan connect is:

```
debug1: identity file /home/me/.ssh/id_rsa type -1
```

vs
 
```
debug1: identity file /home/me/.ssh/id_rsa type 0
```

But I expect a logib prompt and am imagining this is the key based login. I hopped onto  abox on my LAN that has no key based configs to connect tot he Omnnia, and sure enough it is presented with a password prompt. So I tried from Powershell:

```
> ssh -vvv root@remote_server.com
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3
debug3: Failed to open file:C:/Users/me/.ssh/config error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> 'C:\\Users\\me/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> 'C:\\Users\\me/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug2: resolving "remote_server.com" port 22
debug3: ssh_connect_direct: entering
debug1: Connecting to remote_server.com [remote_server_IP] port 22.
debug1: Connection established.
debug3: Failed to open file:C:/Users/me/.ssh/id_rsa error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_rsa.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_rsa error:3
debug1: identity file C:\\Users\\me/.ssh/id_rsa type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_rsa-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_rsa-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_dsa.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_dsa error:3
debug1: identity file C:\\Users\\me/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_dsa-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_dsa-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ecdsa error:3
debug1: identity file C:\\Users\\me/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ecdsa-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_ecdsa-cert type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa_sk error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa_sk.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ecdsa_sk error:3
debug1: identity file C:\\Users\\me/.ssh/id_ecdsa_sk type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa_sk-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ecdsa_sk-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ecdsa_sk-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_ecdsa_sk-cert type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519 error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ed25519 error:3
debug1: identity file C:\\Users\\me/.ssh/id_ed25519 type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ed25519-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519_sk error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519_sk.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ed25519_sk error:3
debug1: identity file C:\\Users\\me/.ssh/id_ed25519_sk type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519_sk-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_ed25519_sk-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_ed25519_sk-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_ed25519_sk-cert type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_xmss.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_xmss error:3
debug1: identity file C:\\Users\\me/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/me/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/me/.ssh/id_xmss-cert.pub error:2
debug3: failed to open file:C:/Users/me/.ssh/id_xmss-cert error:3
debug1: identity file C:\\Users\\me/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.6
debug3: recv - from CB(2) ERROR:108, io:00000180BE2F1930
kex_exchange_identification: read: Connection reset
Connection reset by remote_server_IP port 22
```

Its ssh does respect triple v and does yield on clue:

```
debug3: recv - from CB(2) ERROR:108, io:00000180BE2F1930
```

which I see mentioned here:

https://github.com/PowerShell/Win32-OpenSSH/issues/1776

where they would like server side sshd.log ... but not sure how to produce one from the omnia just yet.
9

ssh password login despite of key configuration: You can explicitly ask sshd to try passwd only by using ssh -l

on ssh error 108, the following two threads are pointing to “connection reset due to other reasons” as the root cause (don’t believe every suggestion from the participating people):
https://stackoverflow.com/questions/69394001/how-can-i-fix-kex-exchange-identification-read-connection-reset-by-peer#

https://serverfault.com/questions/1015547/what-causes-ssh-error-kex-exchange-identification-connection-closed-by-remote

Next steps:

  • If not yet done, i would reboot the turris and try and analyze the logs again
  • provide a “controlled parallel sshd” as follows:

controlled parallel sshd
Using the command line and the LUCI local startup menu you can easily configure another sshd instance (see https://man.openbsd.org/sshd.8 for the options). If required, You can even extended options using -o, see https://man.openbsd.org/sshd_config.5)

  • Log in to turris via LUCI Web UI as root
  • Switch to the menu system → startup → local startup
  • put in -p 11111
  • restart the system to have a parallel instance of sshd listening on port 11111
10

Just checking if you mean sshd -p 11111?

Of course I’d then open that port on the firewall. But perhaps another approach is a port forward from another port like 11111 to 22 on the firewall, to test if there’s some block on 22 on the corporate firewall between the client PC and the remote Omnia (if so that would be pretty new, as it’s worked for years on port 22).

11

I am also concerned that I can do this:

~# cat /etc/ssh/sshd_config 
# auto-generated config file from /etc/config/sshd

PermitRootLogin yes
AuthorizedKeysFile .ssh/authorized_keys
Subsystem sftp /usr/lib/sftp-server
~# cat /etc/config/sshd
cat: can't open '/etc/config/sshd': No such file or directory

As in how does one configure sshd when the config file says it’s autogenerated from nothing? A slight point of confusion. But if I can configure sshd I can play with it’s many options for more explicit logging and such too.

I just opened port 22 to the world and could log in from a remote server in Europe that I can access (so I ssh out tot hat and then ssh back in successfully). And I know the ports open to the workplace source, worked for years like that. But it’s failing with:

kex_exchange_identification: read: Connection reset by peer
Connection reset by remote_server.ip port 22

leaving me wanting to work on my sshd configs to see why. Not least logigng confgs and also configs like KexAlgorithms and so on.

12

How old are your Putty and Ubuntu? I had this before multiple times and it turned out that newer server versions and configs didn’t allow for the ciphers the older clients offered… one time I even had to compile a more recent OpenSSH on a client machine because it couldn’t be updated in its entirety at that time.