Using Consul for Service Discovery In Multiple Data Centers – Part 1

1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5.00 out of 5)

Consul configuration in a multi Data Center environment

Below I am going to describe how to use Consul Health Checking across Multiple Data Centers.

A picture is worth a thousand words, the diagram below describes a high level overview of the Consul multi Data Center architecture.

Consul Multi Data Center Diagram

below are the highlights covered in in this article in great detail.

  • OS required for Consul VMs/Docker installation
  • Consul Installation and configuration
  • Consul GEO(Query) configuration used in Consul multiple Data Center setup
  • Consul and Python API’s
  • Dashboard for Consul data (Host, Service Availability)

Consul from HashiCorp is a fantastic Service Discovery and Health Checking tool.

Consul provides the fowling features out of the box:

  • Service Discovery
  • Health Checking (TCP, HTTP, Script, Docker)
  • KV Store
  • Multi Datacenter

Note: Because of limited hardware available for my test, I am using Solaris Kernel Zones in my examples below. using any VM hypervisor, Docker,etc.. should return the same or similar results. I hope to update this in the future with a Docker implementation example.

Architecture / Preparing / Installing the OS(Kernel Zones/VMs)

For the below Consul configuration you will need the below items.

  • 6 Consul servers – 3 per DC (can be VMs or Docker’s)
  • 2 Consul clients – 1 per DC
    Tip: For maximum HA use at least 2 clients per DC
  • Your end-hosts monitored by consul(with no Consul agent)

You Will also need to make sure, the below list of ports are open in your Firewall.

  1. Server RPC (Default 8300). This is used by servers to handle incoming requests from other agents. TCP only.
  2. Serf LAN (Default 8301). This is used to handle gossip in the LAN. Required by all agents. TCP and UDP.
  3. Serf WAN (Default 8302). This is used by servers to gossip over the WAN to other servers. TCP and UDP.
  4. HTTP API (Default 8500). This is used by clients to talk to the HTTP API. TCP only.
  5. HTTPS API (Default 443). This is used by clients to talk to the HTTPS API. TCP only.
  6. DNS Interface (Default 8600). Used to resolve DNS queries. TCP and UDP.

The table below lists names and IP address used in this configuration (feel free to replace them with your needs).

 Name  IP Addrss
 Consul Servers
 Consul Client
 End Host
 Name  IP Address
 Consul Servers
 Consul Client
 End Host

Note: An updated post using the most recent version of Consul (version 1.4.2) is available here, the below configuration will work with consul version 0.9.2 and below.

As mentioned, since I had limited hardware I used Solaris Kernel Zones to create the environment.
The zone configuration and installation are below.
Note: If you are using a different VM hypervisor or Docker you can skip this section, I hope to update this in the future with a Docker implementation.

Solaris zones installation configuration

First, we need to create the Solaris zone configuration, below are the steps to create that for the consul server and client.

Run the below to create the Consul zone configuration.

Next, you need to generate a zone manifest and profile, run the below to do so.

Now, we are ready to install the zone, Install the Consul zones by running the below.

Firewall and NAT(optional)

I also created a NAT rule for all zones(10.10.1.x), used for outgoing traffic to the external interface.
Note: The NAT configuration is absolutely not a requirement for Consul to work, its just a limitation in our network environment.

To configure firewall NAT in BSD or Solaris, create a rule like the below.
cat /etc/firewall/pf.conf

Now, lets enable the SMF firewall service.

We are now ready to move on to the Consul installation.

Consul installation

First, we need to download the latest Consul binary, you do so by running the below (replace with the OS and version you use), I used version 0.9.2.

Consul is a single binary file, installation is simple, just run the below on all zones (server and client).

Consul user and configuration

Next, we need to create the Consul running environment.
Create a Consul user and directory’s outlined below.

To create a consul user with the necessary directory’s. run the below on all Consul servers and clients.

Next, we need to generate a Consul secret join key, the key will be used for all cluster communications.
Just run the below to do so.

Finally we need to create the Consul config.json. you do so by running the below.
Consul config.json for the Consul Servers
Consul Server DC1 – First node config.json

Consul Server DC2 – First node config.json

Note1: The above config.json is for the first node. Make sure to replace the below fields on the two other nodes, nodes two and three.

  1. bind_addr
  2. node_name

Tip: The performance keyword is by default set to 5. the reason for that (to my understanding) is to accommodate AWS t2.tiny configurations, for maximum performance set this to 1.

Next, to start the consul servers, just run the below.
First on the DC1 3 nodes. then, once up, run on the 3 DC2 nodes.
Tip: You can remove the nohup to run in the foreground.

Note: The above startup enables the Web UI. if you don’t like the Web UI on the Consul servers just remove the -ui option.

Now, Lets move to the Consul client configuration.
Consul config.json for the Consul Clients
Consul Client DC1 – First node config.json

Consul Client DC2 – First node config.json

Note: The Address property under services, can be used to replace the DNS reply address for this service lookup.

To start the consul client, just run the below.
Tip: You can remove the nohup to run in the foreground.

If all done correctly, you should now have a working Consul cluster.
To access the Web UI , just go to any Consul server or client, port 8500.
For example would bring you to the below screen, pick your DC and continue to node and services selection.

Pick you your Data Center

List of Consul services

A failed Consul services

To continue reading part two, on how to configure Consul for Multi Data Center click here.

Like what you’re reading? give it a thumbs up by rating the article.

You might also like – realted to Docker Kubernetes / micro-services

Leave a Reply

Notify of