Using CoreOS, Minio(S3) To Build A Highly Available Private Docker Registry – Part 2

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

High Availability For A Private Docker Registry – Architecture

This is part two on how to configure Docker Registry in a High Availability configuration. To read part one click here.

High Availability Docker Registry Digram

Docker Swarm Configuration on CoreOS

To configure Docker Swarm you will need minimum 3 nodes, below I will use the names/ip’s listed in part 1.

To initialize the Docker Swarm cluster, just run the below on the first node coreos1.

Next, run the below, on the two other nodes (coreos2 and coreos3).
Replace the token with your Docker Swarm token.

For fail-over to work properly, in recent versions, run the below on node2 and node3, this is needed to allow the other two node become manger in case the first node fails.

Now, lets verify the cluster functionality.

Next, create a Docker Swarm network.

Next, create Docker Swarm service.

Now, verify the cluster network and service.

Docker helpful examples

A Docker Swarm busybox example.

Verify the ps busybox Docker Swarm.

Running the busybox as part of Docker Swarm.

Scale the busybox service as part of Docker Swarm.

Destroying the Docker Swarm cluster

Single node to leave the cluster.

Last node to leave the cluster – on coreos1.

Installing Minio Object Store

To install/configure the Docker Registry in a High Availability architecture, we need to configure the back-end storage to be shared. One of the many options available is using AWS S3 as the back-end storage.

One of the free open source vendors created an S3 alike option. in my configuration I will be using the Minio back-end object store(available from minio.io), I found it to be an awesome option for an S3 alike back-end storage.
Note: Just for extras, I will also use an NFS back end shared directory store, helping the fail-over file system always available.

One of the requirements for a Distributed Minio is running a minimum of a 4 node cluster (i.e. 4 docker instances), since I started my configuration with only 4 VBox VM’s I will continue that route(with only 3 nodes) but I will configure 4 Docker instances to meet the requirements.
Note: Idle in a production environment you should have 4 physical nodes, or there are a few other options like two nodes with 2 or 4 physical disks each, for the full requirements, please visit the Distributed Minio Documentation.

Enough talk, lets get to work.

A few notes on the Minio setup.

  1. Minio like S3 requiters an access and secret key. I am using minio and minio123 as these keys, please make sure to change them in your environment
  2. Minio by default listens to port 9000, in order not to have any issues if more then one Minio instance land on the same host you should make sure each node uses there own port, I used 9001, 9002, and so on
  3. I used directory /mnt/docker_reg/minio on all Minio instances. for instances 1 I will be using /mnt/docker_reg/minio/f1 and for instances 2 it will be /mnt/docker_reg/minio/f2 and so on..
  4. Also, normally /mnt/docker_reg/minio/ would be a local mount, for good speed an SSD would make sense. in my case I used /mnt/docker_reg/minio/ as an NFS client mount, then f1, f2,.. are sub directory’s
  5. Lastly, I ran in to issues with the official distributed Minio image. until this bug-fix is integrated in the official Minio image I am using the nitisht/minio:swarm4 image which works great
  6. Update: In my most recent testing the minio/minio:edge Docker image also worked without any issues, I updated the docker-compose-secrets.yaml.

One additional note: All Minio participating need to have their system clock synced, this was add as part of the cloud-config in part 1, if you are not using a cloud-config(CoreOS), you can create a schedule job with systemd something like the below.

Create two files in /etc/systemd/system/, something like the below.
date.service file.

date.timer file.

Now lets enable and load this in the system

Verify the timer working by running the below. out should look something like the below.

Next, create a Docker Stack file called docker-compose-secrets.yaml, like the one below.

Tip: Make sure to specfiy version 3.2+ as prior to that attachable: true was not supported, and you will run in to network issues latter.

The highlight lines in the configuration above are added to address placement constrains, if you don’t mind Minio instances jumping around while nodes go up/down you can just omit those lines.

Lets create the S3 secret keys (You don’t wont these in your Docker file for security reasons)

We are now going to add labels to the system
The labels are used as placement constrains, this will ensure that each Minio Docker instance stays on this node.
Verify you labels, default is empty unless you already added some labels…

Create constrain labels.

Verify new labels are added to the system.

Next, lets deploy the stack, you do so by running the below.

Check the minio_stack Docker logs, give it some time to come up…

Last login to the UI to create your bucket, enter your access keys

Once logged in, click on the + sign, like the screen shut below, and create a bucket called docker.

Next allow read/write privilege for * by cliking on the 3 dots left side on the bucket name – (of curse you should adjust this production to your need).

Docker Registry Installation and configuration

Redis installation for Cache

Before we start the Docker Registry installation, we need to install and configure Redis for caching (Redis is optional).

To install Redis Cache is simple, just run the below on every node running latter the Registry. or you can use separate nodes for that if you like.

Docker Registry Setup Using Redis Cache

Now, we are ready to install the Docker Registry.
To install the Docker Registry create a file called docker-registry-config.yml like the below.

For a complete reference of all Docker registry attributes click here

Next, run the below on all nodes running a Minio instance.

Tip: You can run the Docker Registry on as little as two nodes if that is enough for your workload, each Registry will access the local Minio instance.

To verify Docker Registry is working and using Redis Cache.
Run the below in the in one windows.

In another window run the below. this will pull hello-world image, then push that in your private Docker Registry, you should node a bunch of activity in the Redis window.

Another way to verify Redis working is, by looking in the Docker Registry logs, you should see something like the below, look for the words redis.
docker logs 148cc9088407|less

Now, lets verify the object storage working.
see the below screen capture

Important Note: The above configuration is not using SSL nor is it using any Authentication/Authorization, token for access control. these are things defiantly required in any production configuration, Click here – to configure a secure registry using SSL, Tokens and LDAP /a>.

Another way how to run Docker Registry

If you are using CoreOS, you can also use fleetctl to create the Docker Registry, you will do so by creating a file called registry.service with the below content.

Next, run create the service for all nodes

Verify the service working.

To remove/destroy the unit just run the below.

Docker Registry SSL and Authentication

I hope to update this soon.. once I have a chance. in the mean time you can check this out Gotchas / Tips Creating Your Own Private Docker Registry With Self Signed Certificate

Docker Registry High Availability And Load Balancing

The last piece of the Docker Registry needed for High Availability is adding a load balancer in front of all Docker Registry instances.

There are many options you can use for Load Balancing, the simplest option might be using something like Traefik or Consul and especially if you already have such a setup in place.
You can see details how to – Using Traefik As a Load Balancer / HTTP Reverse Proxy For Micro-Services or Using Consul for Service Discovery In Multiple Data Centers – Part 1.

Other widely used options are, using Nginx or HAproxy.

Helpful Docker alias I use

In CoreOS you do so by adding the below to the write_files: section.

In part 3, I discuss Securing A Private Docker Registry By Using An SSL, Tokens, LDAP

I hope you enjoyed reading the High Availability Docker Registry, give it a thumbs up by rating the article or by just providing feedback.

You might also like – Other articles related to Docker Kubernetes / micro-services.

Leave a Reply

avatar
300
  Subscribe  
Notify of