Hi there
Using SSH jump hosts to reach hosts inside a LAN, without exposing them on the WAN is pretty common, and suggested often.
If you don’t know what I am talking about:
+------------+ +---------------+ +------------------+
| SSH Client | ---> | SSH Jump Host | ---> | SSH Target Server|
+------------+ +---------------+ +------------------+
But I was surprised, how little information is available on how to properly setup an SSH server to function as jump host.
So here is how I did mine.
User Group
I created a group sshjump, this is mostly because, the system already had a group sshlogin to restrict users allowed to login remotely, but I didn’t want to use that for users just jumping trough:
$ groupadd --system sshjump
User Profile
All users jumping trough, will use the same user profilevanhalen. In that way I don’t need to create user profiles for every user. I can simply add or remove their SSH public keys from vanhalen’s authorized key file.
Normally I would create a user without password, but since I didn’t find a way to that on TOS, I just created a random password, which nobody will ever use. I never allow any password login on any SSH server, but it’s just to be able to create the user profile.
$ _random_password=$( openssl rand -base64 48 )
I create the user as follows:
$ useradd --comment "SSH Jump Client" --system \
--password "$_random_password" \
--user-group --groups sshjump \
--no-create-home --shell /bin/false vanhalen
So this this poor vanhalen guy is incapacitated an homeless.
To make things even worse for him, let’s send him in jail:
$ mkdir -p /var/jail
$ chown root:root /var/jail
$ chmod 755 /var/jail
I tried others like /dev/null and /var/empty, but the problem is, the SSH server expects it to be real and owned by root.
Authorized Public Keys
To add users allowed to jumping thru this host, you simply add their SSH public keys to the file /etc/ssh/authorized_keys/vanhalen.
SSH Server Configuration
I usually use a hardened configuration, as recommended by ssh-audit.com, I wont go over all this here. In the common section I just show you the three lines which are relevant for us.
In /etc/ssh/sshd_config:
# OpenSSH_9.8p1, OpenSSL 1.1.1w 11 Sep 2023
PermitRootLogin prohibit-password
AuthenticationMethods publickey
AllowGroups sshlogin sshjump
Then I add a new Match section at the end of /etc/ssh/sshd_config:
# --------------------------------------------------
# Restrictions for Client just junping through here
# --------------------------------------------------
Match Group sshjump
# Allow SSH agent forwarding
AllowAgentForwarding yes
# Allow StreamLocal (Unix-domain socket) forwarding
# Needed for SSH connections multiplexing from the client to the jump host
AllowStreamLocalForwarding yes
# TCP port forwarding is necessary for the SSH jump host functionality
AllowTcpForwarding yes
# Authorized keys file outside of the user's home directory and chroot
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
# Don't allow file system access
ChrootDirectory /var/jail
# Don't allow ports forwarded back to the client, being accessible by other
# clients
# GatewayPorts no
# Don't allow ports forwarded back to the client at all
PermitListen none
# Don't allow clients to access any terminals
PermitTTY no
# Don't allow clients to create (VPN) tunnels
# PermitTunnel no
# Don't allow user-defined environment files on the jump host
# PermitUserEnvironment no
# Don't allow execution of user-defined startup files on the jump host.
PermitUserRC no
# Remove any existing Unix-domain socket file for local or remote port
# forwarding before creating a new one. The default is no.
StreamLocalBindUnlink yes
# Don't allow X11 forwarding
# X11Forwarding no
Test before you Jump
Make sure to check your configuration for errors:
$ /usr/sbin/sshd -t ; echo $?
0
If it looks allrigtht, restart:
$ /etc/init.d/sshd restart
SSH Client Configuration
On the client side in ~/.ssh/config, the jump host configuration has to be at the beginning of the file:
# On top of the file - This is how we connect to our jump host
Host jump.example.net
User vanhalen
ForwardAgent yes
Your various configurations settings can reside in between.
The Match setting takes care of sending all your connections thru the jump host. It needs to be towards the end of your ~/.ssh/config:
# At the end of the file - Use the jump host, for all our hosts,
# except for the jump host itself.
Match host !jump.example.net,*.example.net
ProxyJump jump.example.net
This way you don’t need to change any configurations you already have.
Why and How I am using this myself
I need to reach multiple servers in multiple locations, and it’s just tiresome to setup NAT rules and coming up with random port-numbers.
Also from my experience, changing to a different port than 22 on a IPv4 address, doesn’t change much, you logs fill themselves with SSH login attempts nonetheless. It doesn’t make a big difference nowadays for an attacker to scan all ports for SSH, if the already can scan the whole IPv4 space in about 45 minutes. So changing your SSH ports in 2024, is like trying to sell fax machines (except maybe in Germany),
But all my locations have IPv6, and since IPv6 is to big to scan, I can just open ports and I can still read the important things in my logs. And even better, IPv6 doesn’t need any NAT rules. It just works.
But sometimes, I’m somewhere on the road, where there is no IPv6. For example my mobile provider, hasn’t realized yet, that we are in twenty-first century.
So I was asking myself how do I make my SSH client connect directly to the servers, when I am on site or there I have IPv6, and use the jump host only if I there is no other way?
This is my SSH client configuration which takes care of this:
On top of my ~/.ssh/config:
# IPv4 Jump-Host - only used, if the local client can't use IPv6.
Host ipv4.example.net
AddressFamily inet
User vanhalen
ForwardAgent yes
At the bottom of my ~/.ssh/config:
# Use our IPv4 jump host, if I can't reach the target host by IPv6
Match host !ipv4.example.net,*.exmaple.net !exec "ip route get $(host %h | grep 'IPv6 address' | awk '{print $NF}') &> /dev/null"
ProxyJump ipv4.example.net
Let’s delve into the ip route get command in the exec part of the configuration. This command is used to determine the route information for a given IP address. Here’s a detailed breakdown of that portion:
ip route get $(host %h | grep 'IPv6 address' | awk '{print $NF}') &> /dev/null
-
host %h: This command resolves the hostname (%h) to an IP address using DNS. The output includes information about both IPv4 and IPv6 addresses. -
grep 'IPv6 address': This filters the output to only include lines containing ‘IPv6 address’. This is done to focus on the IPv6 address if it exists. -
awk '{print $NF}': This usesawkto print the last field of the line, which should be the IPv6 address. -
$(...): This is command substitution. The result of the command within the parentheses is substituted into the overall command. -
ip route get ...: This part of the command uses theip route getcommand to determine the routing information for the extracted IPv6 address. It is a way to check if there is a route to the given address. -
&> /dev/null: This redirects both standard output and standard error to/dev/null, effectively suppressing any output. The command is used for its exit status to determine if a route to the IPv6 address exists.
The entire line is enclosed in !exec, which means that the Match block will match if the exit status of the command is non-zero, indicating that there is no valid route to the IPv6 address. This is essentially a way to conditionally match hosts that do not have a valid IPv6 route.
In summary, this part of the configuration is checking if the host has an IPv6 address and if there is a valid route to that address. If not, the Match block will be considered a match, and the specified configuration (in this case, the ProxyJump to ipv4.example.net) will be applied.
