Configuring A Production Ready HA Consul Cluster

Create the Directory and System Structure

We can easily try out consul in an unstructured way by using the consul command. This will allow you to test out some functionality. We did this in the last guide to get familiar with the software.

However, we are going to attempt to set up a more reliable system that is easier to manage, so we will be creating some structure to make this work. Complete the following steps on each of your computers (servers and clients).

The first thing we should take care of is creating a user specific to our task. This is a standard case of user privilege separation, so we will run our consul processes with a dedicated user.

Create the user now by typing:

adduser consul

You can skip all of the prompts (You might want to set a password. It will complain otherwise) if you would like.

Next, we will create the configuration hierarchy that will house the different configurations that we will use depending on how we want to start the service. To make this easy, we will make a parent consul.d directory in the /etc config structure and put subdirectories called bootstrapserver, and client under this on each system:

mkdir -p /etc/consul.d/{bootstrap,server,client}

We can put our configurations in each of these later. Each server will probably use, at most, two of these directories, but we will create the structure for consistency on each host.

We also need to create a location where consul can store persistent data between reboots. We will create a directory at /var/consul for this purpose and give it to the consul user so that it can manage the data:

mkdir /var/consul
chown consul:consul /var/consul

With this structure in place, we should be able to get started crafting our configuration files.

Creating the Bootstrap Configuration

The first configuration we need to create is for bootstrapping the cluster. This is not a very common event as it is only necessary for creating the cluster initially. However, we’re going to create the configuration file so that we can quickly get started again in the event that the cluster goes down completely.

You can put this configuration file on only one of your consul servers, or on all of them to give you more options for bootstrapping. We will only be putting it on server1 for this demonstration.

The configuration files are stored in simple JSON, so they’re quite easy to manage. Create the first file in the bootstrap subdirectory:

nano /etc/consul.d/bootstrap/config.json

In this file, we can start off by specifying that when this config is used, consul should start as a server in bootstrap mode, we specify the datacenter where our cluster will live(Consul is datacenter aware and these designations will help you organize your different clusters by datacenter), and we explicitly define the data directory, which will be used by consul to store cluster state information(it should be noted that is your server has more than one IP defined, you will need to add the advertise_addr configuration directive to specify which ip consul should bind to):

    "bootstrap": true,
    "server": true,
    "datacenter": "nyc2",
    "data_dir": "/var/consul"

Next, we want to implement some encryption to the whisper protocol that consul uses. It has this functionality built in using a shared secret system. The secret must be a 16-bit base-64 encoded string. To get a value of appropriate for this value, we will exit the file temporarily.
In the terminal, we can use the consul command to generate a key of the necessary length and encoding. Type:

#> consul keygen

Copy the value that is generated and re-open the configuration file:

nano /etc/consul.d/bootstrap/config.json

Use the copied string as the value for the encrypt parameter, we will also add some additional information to specify the log level and to indicate that we want to use syslog for logging:

    "bootstrap": true,
    "server": true,
    "datacenter": "nyc2",
    "data_dir": "/var/consul",
    "encrypt": "X4SYOinf2pTAcAHRhpj7dA==",
    "log_level": "INFO",
    "enable_syslog": true

Save and close the file once you are finished making the updates.