Using CoreOS, Minio(S3) To Build A Highly Available Private Docker Registry – Part 1

1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5.00 out of 5)

High Availability For A Private Docker Registry – Architecture

Below is an update on how to build a Private Docker Registry, the provisos article is available here – Gotchas / Tips Creating Your Own Private Docker Registry With Self Signed Certificate.

High Availability Docker Registry Digram

One of the consideration when configuring your private Docker (Cloud) environment (and many times and overlooked component), is the high availability of the Docker Registry. the Docker Registry in most cases stores all your images, code, etc… down time of your private Docker Registry can have a critical effect on your business process success.

There are are multiple options how to configure HA for a private Docker Registry, Commercially available options like Enterprise Docker, or CoreOS, as well as open source free alternatives.

Below I will show you, one of many options you can configure your own Private Docker Registry with a High Availability Design.

The fowling components will be used in the configuration:

  1. Installation of Container OS (CoreOS)
  2. Docker Swarm Configuration
  3. Installation and configuration of a 4 node HA Minio/S3 setup (
  4. Load Balance / Reverse Proxy your Docker Registry

A few additional notes about the below configuration.

  • The configuration below was configured using Virtualbox, but the same apply to a physical hardware setup.
  • There are a number of of other S3(Object Store) solutions like Openio, Scality, Ceph, etc.. but not free… 😉 , plus in my testing Minio just worked awesome.

Lets jump right in to configuration.

Docker OS Installation / configuration

You can install and configure Docker on almost any Linux distro, but there are those Linux distros doing just one thing.. run containers in the most optimal way, usually build from the ground up.

In my configuration below I will be using CoreOS, as find it to be an optimal option, however feel free to use what serves you best. to be fair, there are a few other options, like RancherOS mentioned here Managing Docker On Ubuntu 17.04 Using Rancher Or Portainer.

Docker OS Installation

First lets download the CoreOS ISO, I used the Alpha channel. you can download the latest from here or all channels from here.

As mentioned above I used Virtualbox for the CoreOS installation.
If using Virtualbox, below are a few helpful considerations.

  1. First lets create a private virtualbox network switch, you do so by going to File > Preferences… > Network.
    Click, Add new NAT network.
  2. Click on Port Forwarding
  3. Once done – should look something like this.
  4. Now, select your gust VM, click Settings > Network
    Modify Network to Nat Network
  5. While in Settings > USB un-check Enable USB Controller we will not use that
  6. Also, under Audio un-check Enable Audio
  7. Under under System Make sure to give the VM at least 2048Mb memeory
  8. Last, under Storage Select Optical drive (which is empty), select the CoreOS iso you have downloaded.

We are now ready to boot the first VBox (or Physical hardware).

CoreOS configuration and install to disk

After the OS boots it will look something like the below.

First, we need to generate a random password, do so by running the below.

Next, we need to generate an ssh key for latter use.

Next we need to get an etcd key, you do so, by going to, like the below.
Tip: You only need this if you will be using etcd

Save the etcd key output to be added in the cloud_config.yml

Notes on the below cloud_config.yml.

  • The first line in cloud_config.yml must start with #cloud-config
  • Replace usera passwd with the openssl commend output
  • Replace the ssh-rsa key with the ssh key output.
  • To work as root just run sudo su – or sudo… your command
  • The 3 vLan tags are absolutely not a requirement, feel free to modify at your need

Tip: The simplest way to get the passwd or ssh key content in the cloud_config.yml file is, while in vi just read the file, something like the below.

I usually create a cloud_config.yml locally on my host (or parent host in a VBox config), then from the new guest I scp the newly generated cloud_config.yml, something like the below.

The table below lists names and IP address used in this configuration (feel free to replace them with your needs).

CoreOS Cluster IP Address
 Name  IP Addrss

A full example of the first node cloud_config.yml is below, make sure to replace the IP Address like the table above for each of the 3 nodes.

Tip: You can validate the cloud config by going to the CoreOS validate page.

Part of the process of installing CoreOS, it will fetch the latest release and install that.
For that to work we need to set a proxy (if you are behind a proxy), you do so by running the below.

Finally we are ready to install the OS to disk, do so by running the below.

Tip: To modify configuration data(what was specified in cloud_config.yml) after the OS is installed, you will need to do the modification in /var/lib/coreos-install/user_data since /, usr, etc… are read only. the network configuration is located in /etc/systemd/network/

General Helpful Tips

After the reboot login to the VBox guest (or physical).
If the Vbox network was configured correctly the below should work.

To verify the system working as expected, lets check a few things
The below should all check out green.

A few examples related to etcd (etcd2) verification.

To test etcd2 keys.

If everything checks out (green) works properly, we can move on to the next section, Docker Swarm configuration, then finally to the Docker registry setup, read all of this and more in part 2.

Like what you’re reading? give it a thumbs up by rating the article.

You might also like – Other articles related to Docker Kubernetes / micro-services.

Leave a Reply

Notify of