OpenSSH/Cookbook/Proxies and Jump Hosts

A proxy is an intermediary that forwards requests from clients to other servers. Performance improvement, load balancing, security or access control are some reasons proxies are used.

Jump Hosts -- Passing Through a Gateway or Two

It is possible to connect to another host via one or more intermediaries so that the client can act as if the connection were direct.

The main method is to use an SSH connection to forward the SSH protocol through one or more jump hosts, using the ProxyJump directive, to an SSH server running on the target destination host. This is the most secure method because encryption is end-to-end. In addition to whatever other encryption goes on, the end points of the chain encrypt and decrypt each other's traffic. So the traffic passing through the intermediate hosts is always encrypted. But this method cannot be used if the intermediate hosts deny port forwarding.

Using the ProxyCommand option to invoke Netcat as the last in the chain is a variation of this for very old clients. The SSH protocol is forwarded by nc instead of ssh. Attention must also be paid to whether or not the username changes from host to host in the chain of SSH connections. The outdated netcat method does not allow a change of username. Other methods do.

When port forwarding is available the easiest way is to use ProxyJump in the configuration file or -J as a run-time parameter. An example of -J usage is:

$ ssh -J firewall.example.org:22 server2.example.org

The ProxyJump directive (-J) is so useful it has an entire sub-section below.

In older versions -J is not available. In this case the safest and most straightforward way is to use ssh(1)'s stdio forwarding (-W) mode to "bounce" the connection through an intermediate host.

$ ssh -o ProxyCommand="ssh -W %h:%p firewall.example.org" server2.example.org

This approach supports port-forwarding without further tricks.

Even older clients don't support the -W option. In this case ssh -tt may be used. This forces TTY allocation and passes the SSH traffic as though typed, though this is less secure. To connect to server2 via firewall as the jump host:

$ ssh -tt firewall.example.com ssh -tt server2.example.org

That example opens an SSH session to the remote machine. You can also pass commands. For example, to reattach to a remote screen session using screen you can do the following:

$ ssh -tt firewall.example.com ssh -tt server2.example.org screen -dR

The chain can be arbitrarily long and is not limited to just two hosts. The disadvantage of this approach over stdio-forwarding at the network layer with -W or -J is that your session, any forwarded agent, X11 server, and sockets are exposed to the intermediate hosts.

Passing Through One or More Gateways Using ProxyJump

Starting from OpenSSH 7.3, released August 2016, the easiest way to pass through one or more jump hosts is with the ProxyJump directive in ssh_config(5).

Host server2
        HostName 192.168.5.38
        ProxyJump user1@jumphost1.example.org:22
        User fred

Multiple jump hosts can be specified as a comma-separated list. The hosts will be visited in the order listed.

Host server3
        HostName 192.168.5.38
        ProxyJump user1@jumphost1.example.org:22,user2@jumphost2.example.org:2222
        User fred

It also has the shortcut of -J when using it as a run-time parameter.

$ ssh -J user1@jumphost1.example.org:22 fred@192.168.5.38

Multiple jump hosts can be chained in the same way.

$ ssh -J user1@jumphost1.example.org:22,user2@jumphost2.example.org:2222 fred@192.168.5.38

It is not possible to use both the ProxyJump and ProxyCommand directives in the same host configuration. The first one found is used and then the other blocked.

Transiting a Jump Host Which Has Multiple RDomains / Routing Tables

When passing through a jump host which has its relevant interfaces each on a different rdomain(4), it will be necessary to manipulate the routing tables manually. Specifically that means going back to an older method of transit which relies on netcat and uses ProxyCommand instead of ProxyJump so that route(8) can be added in the middle. Here is an example doing so using only run time parameters to pass through to rdomain 1:

$ ssh -o ProxyCommand="ssh user1@jumphost.example.org route -T 1 exec nc %h %p" fred@server.example.org

That configuration can be made persistent by adding it to the client configuration file, ssh_config(5):

Host jump jumphost.example.org
        HostName jumphost.example.org
        User user1
        IdentitiesOnly yes
        IdentityFile ~/.ssh/jump_key

Host server server.example.com
        HostName 10.100.200.44
        User fred
        IdentitiesOnly yes
        IdentityFile ~/.ssh/inside_key
        ProxyCommand ssh jump route -T 1 exec nc %h %p

With those settings, it is then possible to connect to the host on the LAN via the jump host simply with the line ssh server and all the settings will be taken into use.

Otherwise, the recommended way would have been to use ProxyJump. For more on using the older ProxyCommand directive see the section below, ProxyCommand with Netcat.

Conditional Use of Jump Hosts

It is possible to use Match exec to select for difficult patterns or otherwise make complex decisions, such as which network the client is connecting from or the network connectivity of the available network(s). Below are two cases for how to automatically choose when to use ProxyJump to connect via an intermediate host. The first example is for occasions when there is no local IPv6 connectivity when connecting to a remote machine which has only IPv6 . The second case is for occasions when the target machines are on another network.

Match host ipv6only.example.org
        User fred

Match host ipv6only.example.org !exec "route -n get -inet6 %h"
        ProxyJump dualstack.example.org

With those settings, connections to the machine ipv6only.example.org will go via a jump host only if it is not directly accessible via IPv6.

On GNU/Linux systems that capability would be more complicated.

Match host ipv6only.example.org
        User fred

Match host ipv6only.example.org !exec "/sbin/ip route get $(host -t AAAA %h | sed 's/^.* //')"
        ProxyJump dualstack.example.org

The scripts are invoked using the shell named in the $SHELL environment variable. Remember that configuration directives are applied according to the first one to match. So specific rules go near the top and more general ones go at the end.

Another way is to use the nc(1) utility to check for connectivity to the target host directly at the port in question.

Match !host jumphost.example.org !exec "nc -z -w 1 %h %p"
        ProxyJump jumphost.example.org

Since the above assumes that only one jump host is ever used, it might be combined with other Match criteria for more precision.

Match host 192.168.1.* !host jumphostone.example.org !exec "nc -z -w 1 %h %p"
        ProxyJump jumphostone.example.org

Match host 192.168.2.* !host jumphosttwo.example.org !exec "nc -z -w 1 %h %p"
        ProxyJump jumphosttwo.example.org

That would catch all connections going to the network 192.168.1.*, if there is no direct connection, and send them through the first jump host. Likewise it would also send all connections to the network 192.168.2.* through the second jump host if there is no direct connection. But if there is a direct connection the client will proceed without the jump host.

Of course the tests will have to be more complicated if the client machine moves between several different networks with the same numbering.

Using Canonical Host Names Which Are Behind Jump Hosts

Some local area networks (LANs) have their own internal domain name service to assign its own host names. These names are not accessible to systems outside the LAN. So therefore it is not possible to use the -J or ProxyJump option from the outside because the client would not be able to look up the names on the LAN on the other side of the jump host. However, the jump host itself can look these names up, so the ProxyCommand option can be used instead to call an SSH client on the jump host and use its capabilities to look up a name on the LAN.

In the following sequence, the absence of outside DNS records for the inner host is shown. Then a connection is made to the inner host via the jump host using the jump host's SSH client and the -W option. Keys or certificates are the recommended way of connecting, but passwords are used in this example so that the stages are more visible:

$ host fuloong04.localnet.lan
Host fuloong04.localnet.lan not found: 3(NXDOMAIN)

$ host fuloong04
Host fuloong04 not found: 2(SERVFAIL)

$ ssh -o ProxyCommand="ssh -W fuloong04.localnet.lan:22 jumphost.example.org" fred@fuloong04
fred@jumphost.example.org's password: 
fred@fuloong04's password: 
Last login: Sun May 24 04:06:21 2020 from 203.0.113.131
OpenBSD 6.7-current (GENERIC) #44: Sun May 24 04:06:31 MDT 2020

$ host fuloong04.localnet.lan
fuloong04.localnet.lan has address 10.10.10.193

The -W option ensures that the connection is forwarded over the secure channel and just passes through the jump host without being decrypted. The jump host must both be able to do the DNS look up for LAN names as well as have an SSH client available.

Here is what the initial connection looks like as the client collects the host key and asks about it:

$ ssh -o ProxyCommand="ssh -W fuloong04.localnet.lan:22 jumphost.example.org" fred@fuloong04
fred@jumphost.example.org's password: 
The authenticity of host 'fuloong04 (<no hostip for proxy command>)' can't be established.
ECDSA key fingerprint is SHA256:TGH6fbXRswaP7iR1OBrJuhJ+c1JiEvvc2GJ2krqaJy4.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'fuloong04' (ECDSA) to the list of known hosts.
fred@fuloong04's password: 
Last login: Thu May  7 21:06:18 2020 from 203.0.113.131
OpenBSD 6.7-current (GENERIC) #44: Sun May 24 04:06:21 MDT 2020

Notice that while a name must be given in the usual position it will only serve to identify which host keys are to be associated with the connection. After that initial connection, a host key is saved in known_hosts for the name given as destination fuloong04 not the name given in the ProxyCommand option:

$ ssh-keygen -F fuloong04
# Host fuloong04 found: line 55 
fuloong04 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKwZOXY7wP1lCvlv5YC64QvWPxSrhCa0Dn+pK4b97jjeUqGy/wII5zCnuv6WZKCEjSQnKFP+8K9cmSGnIvUisPg= 

$ ssh-keygen -F fuloong04.localnet.lan

The target host name just serves to identify which host keys to expect. It has no association with the actual host name or address passed in the ProxyCommand option and could even be a random string. Above fuloong04 was used by itself but, again, it could be any random string or even overridden by using the HostKeyAlias option.

As usual, these SSH options can be recorded in a permanent shortcut within ssh_config(5) to reduce effort in typing and to avoid errors.

Old Methods of Passing Through Jump Hosts

A note on old methods: These old methods are deprecated because newer versions of OpenSSH provide easier ways to pass through a jump host. See instead the section Passing Through One or More Gateways Using ProxyJump above. However, some long term support versions of various GNU/Linux distributions may keep very old versions of OpenSSH around on purpose. So while these are unlikely to be encountered, there is still a possibility to need these methods for some years to come.

In old versions of OpenSSH, specifically version 7.2 and earlier, passing through one or more gateways is more complex and requires use of stdio forwarding or, prior to 5.4, use of the netcat(1) utility.

Old: Passing Through a Gateway Using stdio Forwarding (Netcat Mode)

Between OpenSSH 5.4 and 7.2 inclusive, a 'netcat mode' can connect stdio on the client to a single port forwarded on the server. This can also be used to connect using ssh(1), but it needs the ProxyCommand option either as a run time parameter or as part of ~/.ssh/config. However, it no longer needs netcat(1) to be installed on the intermediary machine(s). Here is an example of using it in a run time parameter.

$ ssh -o ProxyCommand="ssh -W %h:%p jumphost.example.org" server.example.org

In that example, authentication will happen twice, first on the jump host and then on the final host where it will bring up a shell.

The syntax is the same if the gateway is identified in the configuration file. ssh(1) expands the full name of the gateway and the destination from the configuration file. The following allows the destination host to be reached by entering ssh server in the terminal.

Host server
        Hostname server.example.org
        ProxyCommand ssh -W %h:%p jumphost.example.org

The same can be done for SFTP. Here the destination SFTP server can be reached by entering sftp sftpserver and the configuration file takes care of the rest. If there is a mix up with the final host key, then it is necessary to add in HostKeyAlias to explicitly name which key will be used to identify the destination system.

Host sftpserver
        HostName sftpserver.example.org
        HostKeyAlias sftpserver.example.org
        ProxyCommand ssh -W %h:%p jumphost.example.org

It is possible to add the key for the gateway to the ssh-agent which you have running or else specify it in the configuration file. The option User refers to the user name on the destination. If the user is the same on both the destination and the originating machine, then it does not need to be used. If the user name is different on the gateway, then the -l option can be used in the ProxyCommand option. Here, the user 'fred' on the local machine, logs into the gateway as 'fred2' and into the destination server as 'fred3'.

Host server
        HostName server.example.org
        User fred3
        ProxyCommand ssh -l fred2 -i /home/fred/.ssh/rsa_key -W %h:%p jumphost.example.org

If both the gateway and destination are using keys, then the option IdentityFile in the config is used to point to the gateway's private key and the option IdentityFile specified on the commandline points at the destination's private key.

Host jump
        HostName server.example.org
        IdentityFile /home/fred/.ssh/rsa_key_2
        ProxyCommand ssh -i /home/fred/.ssh/rsa_key -W %h:%p jumphost.example.org

The old way prior to OpenSSH 5.4 used netcat, nc(1).

Host server
        Hostname server.example.org
        ProxyCommand ssh jumphost.example.org nc %h %p

But that should not be used anymore and the netcat mode, provided by -W, should be used instead. The new way does not require netcat at all on any of the machines.

Old: Recursively Chaining Gateways Using stdio Forwarding

The easy way to do this is with ProxyJump which is available starting with OpenSSH 7.3 mentioned above. For older versions, if the route always has the same hosts in the same order, then a straightforward chain can be put in the configuration file. Here three hosts are chained with the destination being given the shortcut machine3.

Host machine1
        Hostname server.example.org
        User fred
        IdentityFile /home/fred/.ssh/machine1_e25519
        Port 2222

Host machine2
        Hostname 192.168.15.21
        User fred
        IdentityFile /home/fred/.ssh/machine2_e25519
        Port 2222
        ProxyCommand ssh -W %h:%p machine1

Host machine3
        Hostname 10.42.0.144
        User fred
        IdentityFile /home/fred/.ssh/machine3_e25519
        Port 2222
        ProxyCommand ssh -W %h:%p machine2

Thus any machine in the chain can be reached with a single line. For example, the final machine can be reached with ssh machine3 and worked with as normal. This includes port forwarding and any other capabilities.

Only hostname and, for second and subsequent hosts, ProxyCommand are needed for each Host directive. If keys are not used, then IdentityFile is not needed. If the user is the same for all hosts, the that can be skipped. And if the port is the default, then Port can be skipped. If using many keys in the agent at the same time and the error "too many authentication" pops up on the client end, it might be necessary to add IdentitiesOnly yes to each host's configuration.

Old: Recursively Chaining an Arbitrary Number of Hosts

Again, the easy way to do this is with ProxyJump should be used when available, which is starting with OpenSSH 7.3. For older versions, it is possible to make the configuration more abstract and allow passing through an arbitrary number of gateways. You can set the user name with -l thanks to the token, but that user name will be used for all host that you connect to or through.

Host */* 
        ProxyCommand ssh %r@$(dirname %h) -W $(basename %h):%p

In this way hosts are separated with a slash (/) and can be arbitrary in number.

$ ssh host1/host2/host3/host4

There are limitations resulting from using the slash as a separator, as there would be with other symbols. However, it allows use of dirname(1) and basename(1) to process the host names.

The following configuration uses sed(1) to allow different port numbers and user names using the plus sign (+) as the delimiter for hosts, a colon (:) for ports, and an percentage sign (%) for user names. The basic structure is ssh -W $() $() and where %h is substituted for the target host name.

Host *+*
        ProxyCommand ssh -W $(echo %h | sed 's/^.*+//;s/^\([^:]*$\)/\1:22/') $(echo %h | sed 's/+[^+]*$//;s/\([^+%%]*\)%%\([^+]*\)$/\2 -l \1/;s/:\([^:+]*\)$/ -p \1/')

The port can be left off for the default of 22 or delimited with a colon (:) for non-standard values.

$ ssh host1+host2:2022+host3:2224

As-is, the colons confound sftp(1), so the above configuration will only work with it using standard ports. If sftp(1) is needed on non-standard ports then another delimiter, such as an underscore (_), can be configured.

Any user name except the final one can be specified for a given host using the designated delimiter, in the above it is a percentage sign (%). The destination host's user name is specified with -l and all others can be joined to their corresponding host name with the delimiter.

$ ssh -l user3 user1%host1+user2%host2+host3

If user names are specified, depending on the delimiter, ssh(1) can be unable to match the final host to an IP number and the key fingerprint in known_hosts. In such cases, it will ask for verification each time the connection is established, but this should not be a problem if either the equal sign (=) or percentage sign (%) is used.

If keys are to be used then load them into an agent, then the client figures them out automatically if agent forwarding with the -A option is used. However, agent forwarding is not needed if the ProxyJump option (-J) is available. It is considered by many to actually be a security flaw and a general misfeature. So use the ProxyJump option instead if it is available. Also, because the default for MaxAuthTries on the server is 6, using keys normally in an agent will limit the number of keys or hops to 6, with server-side logging getting triggered after half that.

Old: ProxyCommand with Netcat

Another way, which was needed for OpenSSH 5.3 and older, is to use the ProxyCommand configuration directive and netcat. The utility nc(1) is for reading and writing network connections directly. It can be used to pass connections onward to a second machine. In this case, destination is the other server reached via the intermediary jumphost.

$ ssh -o 'HostKeyAlias=destination.example.edu' \
        -o 'ProxyCommand ssh %h nc destination.example.edu 22' \
        jumphost.example.org

Keys and different login names can also be used. Using ProxyCommand, ssh(1) will first connect to jumphost and then from there to destination.example.edu. The HostKeyAlias directive is needed to look up the right key for destination.example.edu, because without it the key for jumphost will be tried and that will, of course, fail unless both systems have the same host keys. In this example, the account user2 exists on jumphost.

$ ssh -o 'HostKeyAlias=destination.example.edu' \
        -o 'ProxyCommand ssh -i jumphost_key-rsa -l user2 %h \
                nc destination.example.edu 22' \
        jumphost.example.org

It's also possible to make this arrangement more permanent and reduce typing by editing ssh_config Here a connection is made to destination via jumphost whenever the target host name is used in full:

Host destination.example.org
        ProxyCommand ssh %r@jumphost.example.org nc %h %p

Here a connection is made to destination via jumphost using the shortcut name 'jump'.

Host jump
        HostKeyAlias destination.example.org
        Hostname jumphost.example.org
        User fred
        ProxyCommand ssh %h nc destination.example.org 22

It can be made more general. The first stanza is to ensure that the bastion or jump host is not looping back on itself. The second stanza applies general rules to pass through the bastion or jumphost for all domains ending in .example.edu or for the shortcut my-private-host:

Host jumphost.example.org
        ProxyCommand none

Host *.example.org my-private-host
        ProxyCommand ssh -l %r jumphost.example.org nc %h %p

The same can be done with sftp(1) by passing parameters on to ssh(1). Here is a simple example with sftp(1) where jumphost is the intermediate host used to connect to machine2. The user name is the same for both hosts.

$ sftp -o 'HostKeyAlias=machine2.example.edu' \
        -o 'ProxyCommand=ssh %h nc machine2.example.edu 22' \
        fred@jumphost.example.edu

Here is a more complex example using a key for jumphost but regular password-based login for the SFTP server.

$ sftp -o 'HostKeyAlias=destination.example.edu' \
        -o 'ProxyCommand ssh -i /home/fred/.ssh/jumphost_rsa \
        -l user2 jumphost.example.edu nc destination.example.edu 22'   \
        jumphost.example.edu

If the user accounts names are different on the two machines, that works, too. Here, 'fred' is the account on the second machine which is the final target. The user 'user2' is the account on the intermediary or jump host.

$ ssh -i fred -i /home/fred/.ssh/destination_rsa \
       -o 'HostKeyAlias destination.example.org' \
       -o 'ProxyCommand ssh -i /home/fred/.ssh/jumphost_rsa \
              -l user2 %h nc destination.example.org 22'
       jumphost.example.org

If at all possible, upgrade or back port a newer version of OpenSSH so that the ProxyJump or -J option may be used instead.

Old: ProxyCommand without Netcat, Using Bash's /dev/tcp/ Pseudo-device

On GNU/Linux jump hosts missing even nc(1), but which have the Bash shell interpreter, the pseudo device /dev/tcp can be another option. This makes use of some built-in Bash functionality, along with the cat(1) utility:

$ ssh -o HostKeyAlias=server.example.org \
	-o ProxyCommand="ssh -l fred %h 'exec 3<>/dev/tcp/server.example.com/22; cat <&3 & cat >&3;kill $!'" fred@jumphost.example.com

It is necessary to set the HostKeyAlias to the host name or IP address of the destination because the SSH connection is just passing through the jump host and needs to expect the right key.

The main prerequisite for this method is having a login shell on a GNU/Linux jump host with Bash as the default shell. If another shell is used instead, such as the POSIX shell, then the pseudo device will not exist and an error will occur instead:

sh: 1: cannot create /dev/tcp/server.example.com/22: Directory nonexistent

With ksh(1), a similar error will occur.

ksh: cannot create /dev/tcp/server.example.com/22: No such file or directory

The zsh(1) shell will also be missing the /dev/tcp pseudo device but produce a different error.

zsh:1: no such file or directory: /dev/tcp/server.example.com/22
zsh:1: 3: bad file descriptor
zsh:1: 3: bad file descriptor
zsh:kill:1: not enough arguments

So, again, this is a process specific to the bash(1) shell. Furthermore, this method is specific to Bash on GNU/Linux distros and will not be available with any of the BSDs or Mac OS as a jump host even if bash(1) is present.

The full details of the connection process from the client perspective can be captured to a log file using the '-E option while increasing the verbosity of the debugging information.

$ ssh -vvv -E /tmp/ssh.dev.tcp.log \
	-o HostKeyAlias=server.example.org \
	-o ProxyCommand="ssh -l fred %h 'exec 3<>/dev/tcp/server.example.com/22; cat <&3 & cat >&3;kill $!'" fred@jumphost.example.com

This pseudo device method is included here mostly as a curiosity but can serve as a last resort in some edge cases. As mentioned before, all recent deployments of OpenSSH will have the -J option for ProxyJump instead.

Port Forwarding Through One or More Intermediate Hosts

Tunneling, also called port forwarding, is when a port on one machine mapped to a connection to a port on another machine. In that way remote services can be accessed as if they were local. Or in the case of reverse port forwarding, vice versa. Forwarding can be done directly from one machine to another or via a machine in the middle.

When available, the ProxyJump option is preferable. It works just as easily with port forwarding. Below a tunnel is set up from the localhost to machine2 which is behind a jump host.

$ ssh -L 8900:localhost:80 -J jumphost.example.org machine2

Thus port 8900 on the localhost will actually be a tunnel to port 80 on machine2. It can be as simple as that. As with the usual ProxyJump option, jump hosts can be chained by joining them with commas.

$ ssh -L 8900:localhost:80 -J jumphost1.example.org,jumphost2.localnet.lan machine2

Alternative accounts and ports can be specified that way, too, if needed. This method works best with key- or certificate-based authentication. It is possible to use all the options in this way, such as -X for X11 forwarding. The same with -D for making a SOCKS proxy.

Old: Port Forwarding Via a Single Intermediate Host Without ProxyJump

Below we are setting up a tunnel from the localhost to machine2, which is behind a firewall, machine1. The tunnel will be via machine1 which is publicly accessible and also has access to machine2.

$ ssh -L 2222:machine2.example.org:22 machine1.example.org

Next connecting to the tunnel will actually connect to the second host, machine2.

$ ssh -p 2222 remoteuser@localhost

Here is an example of running rsync(1) between the two hosts using machine1 as an intermediary with the above setup.

$ rsync -av -e "ssh -p 2222"  /path/to/some/dir/   localhost:/path/to/some/dir/

SOCKS Proxy

Basic SSH Dynamic Forwarding

It is possible to connect via an intermediate machine using a SOCKS proxy. SOCKS4 and SOCKS5 proxies are currently supported by OpenSSH. SOCKS5 allows transparent traversal of a firewall or other barrier by an application and can use strong authentication with help of GSS-API. Dynamic application-level port forwarding allows the outgoing port to be allocated on the fly thus creating a proxy at the TCP session level.

Here the web browser can connect to the SOCKS proxy on port 3555 on the local host:

$ ssh -D 3555 server.example.org

Using ssh(1) as a SOCKS5 proxy, or in any other capacity where forwarding is used, you can specify multiple ports in one action:

$ ssh -D 80 -D 8080 -f -C -q -N fred@server.example.org

You'll also want the DNS requests to go via your proxy. So, for example, in recent versions of Firefox, there may be an option "Proxy DNS when using SOCKS v5" to check. Or in older versions, about:config needs network.proxy.socks_remote_dns set to true instead. However, in Chromium, you'll need to launch the browser while adding two run-time options --proxy-server and --host-resolver-rules to point to the proxy and tell the browser not to send any DNS requests over the open network.

It will be similar for other programs that support SOCKS proxies. So, you can tunnel Samba over ssh(1), too.

Going via a jump host is just a matter of using the ProxyJump option:

$ ssh -D 8899 -J jumphost.example.org machine2

Chaining multiple jump hosts can be done this way too. See the earlier section on that for discussion and examples.

Old: SOCKS Proxy Via a Single Intermediate Host

If you want to open a SOCKS proxy via an intermediate host, it is possible:

$ ssh -D 3555 -J user1@middle.example.org user2@server.example.org

On older clients, an extra step is needed.

$ ssh -L 8001:localhost:8002 user1@middle.example.org -t ssh -D 8002 user2@server.example.org

In the second example, the client will see a SOCKS proxy on port 8001 on the local host, which is actually a connection to machine1 and the traffic will ultimately enter and leave the net through machine2. Port 8001 on the local host connects to port 8002 on machine1 which is a SOCKS proxy to machine2. The port numbers can be chosen to be whatever is needed, but forwarding privileged ports still requires root privileges.

SSH Over Tor

There are two ways to use SSH over Tor. One is using the client over Tor, the other is to host the server as an Onion service. Running the client over Tor allows its origin to be hidden. Hosting the server behind Tor allows its location to be hidden. The two methods can be combined.

Tunneling the SSH Client Over Tor with Netcat

Besides using ssh(1) as a SOCKS proxy, it is possible to tunnel the SSH protocol itself over another SOCKS proxy such as Tor. It is anonymity software and a corresponding network that uses relay hosts to conceal a user's location and network activity. Its architecture is intended to prevent surveillance and traffic analysis. Tor can be used in cases where it is important to conceal the point of origin of the SSH client. It can also be used to connect to onion services. Unfortunately, connecting through Tor often comes at the expense of noticeable latency.

On the end point which the client sees, Tor is a regular SOCKS5 proxy and can be used like any other SOCKS5 proxy. So this is tunneling SSH over a SOCKS proxy. The easiest way to do this is with the torsocks utility if available, simply by preceding a SSH command with it:

$ torsocks ssh fred@server.example.org

However, this can also be accomplished by using netcat:

$ ssh -o ProxyCommand="nc -X 5 -x localhost:9050 %h %p" fred@server.example.org

When attempting a connection like this, it is very important that it does not leak information. In particular, the DNS lookup should occur over Tor and not be done by the client itself. Make sure that if VerifyHostKeyDNS is used that it be set to 'no'. The default is 'no' but check to be sure. It can be passed as a run-time argument to remove any doubt or uncertainty.

$ ssh -o VerifyHostKeyDNS=no -o ProxyCommand="nc -X 5 -x localhost:9050 %h %p" server.example.org

Using the netcat-openbsd nc(1) package, this seems not to leak any DNS information. Other netcat packages might or might not be the same. It's also not clear if there are other ways in which this method might leak information. YMMV.

Since these methods proxy through Tor, all of them will also enable connecting to onion services. You can configure SSH to automatically connect to these through Tor, without affecting other types of connections. An entry similar to the following can be added to ssh_config or ~/.ssh/config:

Host *.onion
        VerifyHostKeyDNS no
        ProxyCommand nc -x localhost:9050 -X 5 %h %p

You can further add CanonicalizeHostname yes before any Host declarations so that if you give onion services a nickname, SSH will apply this configuration after determining the hostname is a .onion address.

Providing SSH as an Onion Service

SSH can be served from an .onion address to provide anonymity and privacy for the client and, to a certain extent, the server. Neither will know where the other is located, though a lot of additional precautions must be taken to come close to anonymizing the server itself and at least some level of trusting the users will still be necessary.

The first step in making SSH available over Tor is to set up sshd_config(5) so that the SSH service listens on the localhost address and only on the localhost address.

ListenAddress 127.0.0.1:22

Multiple ListenAddress directives can be used if multiple ports are to be made available. However, any ListenAddress directives provided should bind only to addresses in the 127.0.0.0/8 network or the IPv6 equivalent. Listening to any WAN or LAN address would defeat Tor's anonymity by allowing the SSH server to be identified from its public keys.

The next step in serving SSH over Tor is to set up a Tor client with a hidden service forwarded to the local SSH server. Follow the instructions for installing a Tor client given at the Tor Project web site, but skip the part about the web server if it is not needed. Add the appropriate HiddenServicePort directive to match the address and that sshd(8) is using.

HiddenServicePort 22 127.0.0.1:22

If necessary, add additional directives for additional ports. In newer versions of Tor, a 56-character version 3 onion address can be made by adding HiddenServiceVersion 3 to the Tor configuration in versions that support it.

Be sure that HiddenServiceDir points to a location appropriate for your system. The onion address of the new SSH service will then be in the file hostname inside the directory indicated by the HiddenServiceDir directive and will be accessible regardless of where it is on the net or how many layers of NAT have it buried. To use the SSH service and verify that it is actually available over Tor, see the preceding section on using the SSH client over Tor with Netcat.

Just making it available over Tor is not enough alone to anonymize the server. However, one of the other advantages of this method is NAT punching. If the SSH service is behind several layers of NAT, then providing it as an Onion service allows passing through those layers seamlessly without configuring each router at each layer. This eliminates the need for a reverse tunnel to an external server and works through an arbitrary number of layers of NAT such as are now found with mobile phone modems.

Passing Through a Gateway with an Ad Hoc VPN

Two subnets can be connected over SSH by configuring the network routing on the end points to use the tunnel. The result is a VPN. A drawback is that root access is needed on both hosts, or at least sudo(8) access to ifconfig(8) and route(8). A related more limited and more archaic approach, not presented here but which does not require root at the remote end, would be to use ssh to establish connectivity and then establish a network using PPP and slirp.

Note that there are very few instances where use of a VPN is legitimately called for, not because VPNs are illegal (quite the opposite, indeed data protection laws in many countries make them absolutely compulsory to protect content in transit) but simply because OpenSSH is usually flexible enough to complete most routine sysadmin and operational tasks using normal SSH methods as and when required. This SSH ad-hoc VPN method is therefore needed only very rarely.

Take this example with two networks. One network has the address range 10.0.50.1 through 10.0.50.254. The other has the address range 172.16.99.1 through 172.16.99.254. Each has a machine, 10.0.50.1 and 172.16.99.1 respectively, that will function as a gateway. Local machine numbering starts with 3 because 2 will be used for the tunnel interfaces on each LAN.

              +----10.0.50.1       172.16.99.1----+
              +    10.0.50.2 ===== 172.16.99.2    +
              |                                   |
10.0.50.3-----+                                   +---172.16.99.3
              |                                   |
10.0.50.4-----+                                   +---172.16.99.4
              |                                   |
10.0.50.5-----+                                   +---172.16.99.5
              |                                   |
10.0.50.etc---+                                   +---172.16.99.etc
              |                                   |
10.0.50.254---+                                   +---172.16.99.254

First a tun device is created on each machine, a virtual network device for point-to-point IP tunneling. Then the tun interfaces on these two gateways are then connected by an SSH tunnel. Each tun interface is assigned an IP address.

The tunnel connects machines 10.0.50.1 and 172.16.99.1 to each other, and each are already connected to their own local area network (LAN). Here is a VPN with the client as 10.0.50.0/24, remote as 172.16.99.0/24. First, set on the client:

$ ssh -f -w 0:1 192.0.2.15 true
$ ifconfig tun0 10.1.1.1 10.1.1.2 netmask 255.255.255.252
$ route add 172.16.99.0/24 10.1.1.2

On the server:

$ ifconfig tun1 10.1.1.2 10.1.1.1 netmask 255.255.255.252
$ route add 10.0.50.0/24 10.1.1.1

Troubleshooting an Ad Hoc VPN

There are several possible causes to the following kind of error message:

channel 0: open failed: connect failed: open failed
Tunnel forwarding failed

One possibility is that tunneling has not yet been enabled on the server. The SSH server's default configuration has tunneling turned off, so it must be enabled explicitly using the PermitTunnel configuration directive prior to attempting an ad hoc VPN. Failure to enable tunneling will result in an error like the above when connecting using the -w option in the client. The solution in that case for that is to set PermitTunnel to 'yes' on the server.

Another possibility is that either the remote system or the local system or both lack the necessary network interface pseudo-device. That can happen because either or both accounts lack the privileges necessary to create the tun(4) devices on the fly. Several work-arounds exist, including using a privileged account to make the tun(4) devices on each system in advance. In other words, the solution is to manually create the necessary network interface pseudo-devices for the tunneling.

A third possibility is that the one or both of the accounts do not have proper permissions to access the network interface pseudo-devices. Check and, if necessary, correct group memberships.

 

References