Updated: Using Consul For Service Discovery In Multiple Data Centers Version 1.4 – part 2

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)

Configuring Consul for Service Discovery in a Multiple Data Centers

This is part two on Configuring Consul for Service Discovery in a Multiple Data Centers using version 1.4, to see part 1 click here.

Note: You might also want to read the original post which was using version 0.9.2, by going here part 1 and here part 2.

In the first part I successfully showed how to create a two separate data center stand alone Consul environment.

For a quick recap from part 1 I included below the Consul Multi Data Center Layout.
Consul Multi Data Center layout used in this article

Below, we are first gong to verify the Consul environment is working properly, we will then move-on to configure Consul GEO prepared query for multi Data Center use, then, we will finally complete the Consul setup with a few API examples, using Curl or Python.

To test/verify your cluster, just run the below DNS checks.
Tip: Consul should return 3 address (if using a 3 master node cluster).

To return Data Center specific information.
For DC1 just run the below.

For DC2 just run the below.

To lookup a specific service.
Tip: If omitting the DC keyword, it will always return the local client DC.

So far we have been configuring two stand alone working Consul clusters (with shared serf results). however it will only return local results, meaning if your client has joint dc1, you will get back dc1 results.

In order to return DC specific results, we will have to configure Consul GEO auto fail-over.
In the next section I will show you how to configure Consul GEO failover.

Consul Multi Data Center Configuration

To use Consul GEO failover, you will have to create something referred to as a prepared query.

The Consul prepared query gives you the capability to use an API or DNS lookup, based on the query keyword. prepared query’s also gives you the capability to cascade data centers, more is explained below.

There are many options available to use in a prepared query, you can basically use any reg-ex style to manipulate the dns/api data returned.

In our example below, we create a prepared query matching lookups with the Service name, in our example service, rad-6789 will be looked-up and matched. first matching dc1, then if failed return dc2. an example lookup is below.

Creating a prepared query

To create the prepared query run the below.

Tip: The name.full matches any service name.

To verify the prepared query was added to the system and is working, just run the below, the results should look similar to the output below.

Other prepared query options are to set the Datacenters lookup order, or max Data Center hops, an example is below.

For a full list of options please check the Consul API query docs.

Finally, we are now ready for the fun part. to test the HA service across Data Centers, follow the below to see that in action.

The assumption is that both DC’s you configured run a service. in our example I am using the Solaris RAD service which listens on port 6789, in the original post available here I used a MySQL DB listening on port 3306, feel free to replace with your choice of db/application.

Note: The above configuration uses the Consul TCP port check, there are a number of other options, including using your own script.

Below is an example service lookup using the query keyword, instead of the service keyword, which works across data centers, the first lookup reaches/returns DC1, and the second lookup reaches/returns DC2, since DC1 is down.

As you can see it returns the local IP Address
Tip: If you like the return to be a different IP Address, you can replace that by adding the Address keyword.

Now, stop the local service or Consul agent, run the exact same lookup again, and here you go, it should/will return the remote rad-6789 address, like the example below.

Note: You can also run the query with out the SRV keyword to return just the IP Address.

Working RAD service (TCP check).

DC2 node status.

Failed RAD service (TCP check).

Using Python to access Consul Data

Till now we ware only using regular DNS tools like dig or nslookup. in the next section I will show you how to use Python to return the same results.

To simple return DNS results, we can use one of the Python DNS modules. a simple example is below.

Running the above Python script will return two lines, one line will show just the IP address, and one line will show the IP Address and the port.
In our case it should look like the below.

All of the examples above was using regular built-in Python modules, next I will show you how you can use the more specific Python Consul Module.

To use Python Consul Module, we must first install the Python Module, you do so by running the below.

There are so many ways to manipulate Consul data with the Python module, it will be hard to cover all of them, I will include just a few examples below.

Returning Consul node results.

Disabling a Consul service.

Below is a partial list of options, just uncomment what you are trying to use.

For the full list of the python module options please click here.

I hope you enjoyed reading the Consul Multi Data Center setup and configuration, give it a thumbs up by rating the article or by just providing feedback.

You might also like – realted to Docker Kubernetes / micro-services.

Leave a Reply

Notify of