Connecting to a Mongo Replica Set via SSH

Paul Ridgway -

Introduction

When in a previous role, we recently upgraded our MongoDB Java Client Driver to use the latest version (we were a bit behind).

Fortunately there were minimal compatibility changes:

  • There were a few Checked Exceptions that have been removed around the ServerAddress class.

  • A few classes like StringRangeSet disappeared, we just created our own.

  • A load of methods were deprecated - which did not force us to upgrade all usages but required us to have some smarter exception handling, which I will add some details on later.

All in all the result was good - no major code changes were required, no compatibility issues, the new driver has plenty of new features we are yet to use and it also has much better logging from the start of the connection so its easier to see what is going on with the cluster, discovery, etc.

The Problem

Our versions are handled across the board using a Root POM, so updated dependencies cascade downstream to other projects.

I have a couple of experimental projects that rely on the Root POM and some of our internal libraries, so these also required updating.

Unlike our production services, which are completely locked down, I sometimes connect to the AWS Instances running my experiments to debug issues etc. On occasion this involves tunnelling the database connection locally so I can run the code against it.

For example:

ssh test.xxx.net -L 17017:db-node1.xxx.net:27017

With the latest Mongo Driver this seems more difficult (or perhaps I didn't notice before) but it checks the mode of the server:

INFO - Discovered cluster type of REPLICA_SET

And if it is a Replica Set it adds all the peers and potentially removes the 'seed host' specified.

INFO - Adding discovered server db-node3.xxx.net:27017 to client view of cluster
INFO - Adding discovered server db-node1.xxx.net:27017 to client view of cluster
INFO - Adding discovered server db-node2.xxx.net:27017 to client view of cluster
INFO - Server 127.0.1.1:27017 is no longer a member of the replica set.  Removing from client view of cluster.

Tunnelling a single port to a Replica Node over SSH means that after the 'client view' or 'seed host' is removed my local computer ends up trying to talk to hostnames that resolve to Private IPs - and naturally it fails to connect.

The Solution

Fortunately there is an easy way around this.

IP Addresses starting start 127.xxx.xxx.xxx all resolve to the loopback adapter (ie they connect to the computer you are using).

This means that when tunnelling the connection we can map each Replica Node to a different local IP on the default Mongo port:

ssh stats.xxx.net \
    -L 127.0.1.1:27017:db-node1.xxx.net:27017 \
    -L 127.0.1.2:27017:db-node2.xxx.net:27017 \
    -L 127.0.1.3:27017:db-node3.xxx.net:27017

To ensure that anything running locally can connect on these alternate IPs we also have to update the /etc/hosts file:

127.0.1.1       db-node1.xxx.net
127.0.1.2       db-node2.xxx.net
127.0.1.3       db-node3.xxx.net

And it works!

You must be careful of course, with your client able to resolve and connect to the whole Replica Set it knows the Primary Node and can make destructive changes to the data.

When viewing or investigating any non-test data we always inspect a Secondary Node.