Using Consul for Service Discovery in Multiple Data Centers – Part 2

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 5.00 out of 5)

Consul configuration in a multi Data Center environment

This is part two on how to configure Consul in a multi Data Center environment. Click here for part one.

Note: An updated post using the most recent version of Consul (version 1.4.2) is available here, the below configuration will work with consul version 0.9.2 and below.

Consul Multi Data Center Diagram

Verify your cluster configuration

In the first part we successfully completed creating two separate data centers stand alone Consul environments.

Below, we are first gong to verify the Consul environment as working properly, we will then move-on to configure Consul GEO prepared query for multi Data Center use, then, we will finally complete the Consul setup with a few API examples, using Curl or Python. as a bonus I added a Consul Availability Dashboard.

To test/verify your cluster, just run the below DNS checks.
Tip: Consul should return 3 address.

To return Data Center specific information.
For DC1 just run the below.

For DC2 just run the below.

To lookup a specific service.
Tip: If omitting the DC keyword, it will always return the local client DC.

So far we have been configuring two stand alone working Consul clusters. however it will only return local results, meaning if your client has joint dc1, you will get back dc1 results.

In order to return any available DC results, we will have to configure Consul GEO auto failover.
In the next section I will show you how to configure Consul GEO failover.

Consul Multi Data Center Configuration

To use Consul GEO failover, you will have to create something referred to as a prepared query.

The Consul prepared query gives you the capability to do an API or DNS lookup, based on the query keyword. prepared query’s also gives you the capability to cascade data centers, more is explained below.

There are many options available to use in a prepared query, you can basically use any reg-ex style to manipulate the dns/api data returned.

In our example below, we create a prepared query matching lookups with the Service name, in our example service db1 will be looked-up and matched, first matching dc1, then if failed return dc2. an example lookup is below.

To create the prepared query run the below.

Tip: The name.full matches any service name.

To verify the prepared query was added to the system, just run the below, the results should look similar to the below output.

Other prepared query options are to set the Datacenters lookup order, or max Data Center hops, an example is below.

For a full list of options please check the Consul API query docs.

Finally, we are now ready for the fun part. to test the HA across Data Centers, follow the below.

The assumption is that both DC’s you configured run a DB like MySQL or similar service. in our example its MySQL on port 3306, feel free to replace with your choice of db/application.

Below is an example service lookup using the query keyword, instead of the service keyword across data centers, the first lookup reaches/returns DC1, and the second lookup reaches/returns DC2, since DC1 is down.

As you can see it returns the local IP Address
Tip: If you like the return to be a different IP Address, you can replace that by adding the Address keyword.

Now, stop the local DB or Consul agent, run the exact same lookup again, and here you go, it should/will return the remote db address, like the example below.

Using Python to access Consul Data

Till now we ware only using regular DNS tools like dig or nslookup. in the next section I will show you how to use Python to return the same results.

To simple return DNS results, we can use one of the Python DNS modules. a simple example is below.

Running the above Python script will return two lines, one line will show just the IP address, and one line will show the IP Address and the port.
In our case it should look like the below.

All of the examples above was using regular built-in Python modules, next I will show you how you can use the more specific Python Consul Module.

To use Python Consul Module, we must first install the Python Module, you do so by running the below.

There are so many ways to manipulate Consul data with the Python module, it will be hard to cover all of them, I will include just a few examples below.

Returning Consul node results.

Disabling a Consul service.

Below is a partial list of options, just uncomment what you are trying to use.

For the full list of the python module options please click here.

Using Consul data for a data center Dashboard – server availability

One of the benefits of Consul is the health checks.
I am using Consul health checks, as a method to provide server availability per Data Center.

The below Availability Dashboard uses Consul to dynamically update server available, and was build using the Gentellela template, available here.
Availability Dashboard screen capture is below

I hope you enjoyed reading the Consul Multi Data Center setup and configuration, give it a thumbs up by rating the article or by just providing feedback.

You might also like – realted to Docker Kubernetes / micro-services.

Leave a Reply

Notify of