DevTech101

DevTech101
1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 3.00 out of 5)
Loading...

Configuring a 3 node(master) Kubernetes cluster

Below I am going to describe the process to install/configure a 3 node Kubernetes cluster. the configuration will contain 3 Kubernetes master nodes (i.e. controller, api, scheduler, proxy), I will also add 1 worker node(additional workers can be added as needed).

Note: Most of the examples below are still relevant for the overall kubernetes configuration. However, some of the YAML file examples below are outdated and will only work with kubernetes v1.8.x. For a copy of the most recent CoreOS examples kubernetes version v.1.13.x+, is available on my GitHub repository by going here. You can Also generate your own by using the generator also available on my GitHub repository by going here. A more up-to-date write up is also available Installing a 3 master node Kubernetes cluster on CentOs.

Why not use Tectonic? Two reasons for that.
  1. While Tectonic is great, I am still learning how Kubernetes works, and the only way to understand is, by creating your own cluster from scratch.
  2. Tectonic comes with a 10 node license limitation.
I divided the configuration into parts outlined below (still working progress). Note: An up-to-date example is available on my GitHub project page, or generate your own Kubernetes configuration with the Kubernetes generator available here on my GitHub page.

This is part 1 – Initial setup – getting CoreOS and prepare SSL certificates

What items, versions are in the configuration below, and what not. Items versions used.
  1. Openssl or Cfssl to generate certificates.
  2. CoreOS: 1576.1.0
  3. Kubernetes/kubelet/kubectl: 1.8.1
  4. Etcd: version 3
  5. Flannel: version 0.90-0.30
  6. Default Container engine Rkt: version 1.29.0 (no Docker is being used)
  7. Use SSL for all communication if possible
Items not included.
  1. Matchbox configuration
Note: This setup is still in progress, I will be updating on further development. With that said, let’s jump right in.

OS configuration – CoreOS preparation

Lets start with the OS, I opted in using CoreOS which can be download form here – CoreOS ISO images. The reason for selecting CoreOS is obvious, CoreOS runs the process in a container(rkt) and has out of the box excellent Kubernetes support plus many more great features. A few notes about recent CoreOS development.
  1. CoreOS used to have this so called cloud-config configuration file, in the recent versions they moved to something they call container linux config (ignition config). If you are migrating from a cloud config to a container linux config, you might find migrating to ignition and ignition config refrence helpful.
  2. To create an ignition config you will need the ct utility, which you can download from here and a helpfull video how to use it from here
  3. The ct utility (transpiler) will generate an ignition config which you can then use to configure your nodes.
An example on how to use the ct utility is below,
ct -in-file n1-ign.yaml -platform vagrant-virtualbox -out-file n1-ign.ign
ct -in-file n2-ign.yaml -platform vagrant-virtualbox -out-file n2-ign.ign
ct -in-file n3-ign.yaml -platform vagrant-virtualbox -out-file n3-ign.ign
Note: You can skip and jump directly to a complete ct/ignition ready files for a 3 Node cluster. Click here for Node 1, Node 2 and Node 3. In my configuration below I am using 3 Virtualbox VM’s, to see how to configure those VM’s you can read this.. Note: that using physical hardware would apply the same or similar process and can be fully automated with matchbox. Last, CoreOS has also a project called matchbox, matchbox is an HTTP and gRPC service that renders signed Ignition configs, cloud-configs, network boot configs, and metadata to machines to create Container Linux clusters. matchbox helps fully automate the Ignition config. For more details you can check out CoreOS matchbox documentation here.

Prepare SSL certificates by using Openssl or Cfssl

Since Several components of Kubernetes require SSL, to simplify the certificate creation I created only two certificates.
  1. CA certificate
  2. General certificate – used for all components
  3. Optional create worker certificates
Since the certificate is shared i.e. re-used by all components i.e. Etcd, Docker, Rkt, Kubelet and all Manifests, etc.. the certificate has to contain all the names and ip’s, more is described below. The Kubernetes node names, IP address are in the table below.
CoreOS Cluster IP Address
 Name  IP Addrss
 coreos1  10.0.2.11/24
 coreos2  10.0.2.12/24
 coreos3  10.0.2.13/24

Generating the Certificates by using Cfssl

Cfssl Reference Since many OS’s do not include the cfssl application, the installation steps are below.
# Use the below apt-get
apt-get install cfssl

# Or use the below
mkdir ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin
# default json files for ca and client (I modified the files below)
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json
Now, create the ca-config.json json files. (I modified the expire time)
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "server": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "etcd-node": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "etcd-proxy": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
Next, create the ca-csr.json file – used for CA configuration.
{
    "CN": "Etcd",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "New York",
            "O": "Company1",
            "OU": "Ops",
            "ST": "NY"
        }
    ]
}
Next, create the etcd-node-csr.json – used for etcd and all other components. Note: Replace with your domain
{
    "CN": "etcd-node",
    "hosts": [
        "0.0.0.0",
        "10.0.2.11",
        "10.0.2.12",
        "10.0.2.13",
        "10.0.2.1",
        "10.0.2.2",
        "10.3.0.1",
        "127.0.0.1",
        "localhost",
        "coreos1",
        "coreos2",
        "coreos3",
        "coreos1.local",
        "coreos2.local",
        "coreos3.local",
        "coreos1.domain.com",
        "coreos2.domain.com",
        "coreos3.domain.com",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster.local",
        "kube-apiserver",
        "kube-admin",
        "domain.com"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "New York",
            "O": "Company1",
            "OU": "Ops",
            "ST": "NY"
        }
    ]
}
Create the below if you would like to use a separate key for the kub-proxy communication. Note: Replace with your domain
{
    "CN": "etcd-proxy",
    "hosts": [
        "0.0.0.0",
        "10.0.2.11",
        "10.0.2.12",
        "10.0.2.13",
        "10.0.2.1",
        "10.0.2.2",
        "10.3.0.1",
        "127.0.0.1",
        "localhost",
        "coreos1",
        "coreos2",
        "coreos3",
        "coreos1.local",
        "coreos2.local",
        "coreos3.local",
        "coreos1.domain.com",
        "coreos2.domain.com",
        "coreos3.domain.com",
        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster.local",
        "kube-apiserver",
        "kube-admin",
        "domain.com"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "US",
            "L": "New York",
            "O": "Company1",
            "OU": "Ops",
            "ST": "NY"
        }
    ]
}
Now, just run the below which will generate the certificate and keys used latter in the kubernetes configuration.
# Creat CA
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

# Create Node Certficate
cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=etcd-node \
  etcd-node-csr.json | cfssljson -bare etcd-node

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=etcd-proxy \
  etcd-proxy-csr.json | cfssljson -bare etcd-proxy
You should now have all the files below.
# ls -l
-rw-r--r-- 1 root root 1102 Oct 17 15:04 ca-config.json
-rw-r--r-- 1 root root  256 Oct 17 15:04 ca-csr.json
-rw------- 1 root root 1675 Oct 17 16:23 ca-key.pem
-rw-r--r-- 1 root root  989 Oct 17 16:23 ca.csr
-rw-r--r-- 1 root root 1285 Oct 17 16:23 ca.pem
-rw-r--r-- 1 root root  879 Oct 17 16:22 etcd-node-csr.json
-rw------- 1 root root 1675 Oct 17 16:23 etcd-node-key.pem
-rw-r--r-- 1 root root 1476 Oct 17 16:23 etcd-node.csr
-rw-r--r-- 1 root root 1826 Oct 17 16:23 etcd-node.pem
-rw-r--r-- 1 root root  880 Oct 17 16:22 etcd-proxy-csr.json
-rw------- 1 root root 1675 Oct 17 16:23 etcd-proxy-key.pem
-rw-r--r-- 1 root root 1476 Oct 17 16:23 etcd-proxy.csr
-rw-r--r-- 1 root root 1830 Oct 17 16:23 etcd-proxy.pem
-rwxr-xr-x 1 root root  404 Oct 17 15:03 run.sh

Generating the Certificates by using OpenSSL

Since most linux distros include Openssl, I am including steps to use that below. Create your config file cert.conf Note: Replace with your domain
[req] 
default_bits       = 2048
prompt             = no
default_md         = sha256
distinguished_name = dn
req_extensions     = v3_req
x509_extensions    = v3_ca
 
[ dn ]
C                  = US
ST                 = NY
L                  = New York
O                  = Company1
OU                 = Ops
CN                 = etcd-node
 
[ v3_ca ]
keyUsage = critical,keyCertSign, cRLSign
basicConstraints = critical,CA:TRUE
subjectKeyIdentifier = hash

[ v3_req ]
keyUsage = critical,digitalSignature, keyEncipherment, nonRepudiation
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = critical,CA:FALSE
subjectKeyIdentifier = hash
subjectAltName = @alt_names
 
[ alt_names ]
DNS.1              = domain.com
DNS.2              = coreos1.domain.com
DNS.3              = coreos2.domain.com
DNS.4              = coreos3.domain.com
DNS.5              = localhost
DNS.6              = coreos1
DNS.7              = coreos2
DNS.8              = coreos3
DNS.9              = coreos1.local
DNS.10             = coreos2.local
DNS.11             = coreos3.local
DNS.12             = kubernetes
DNS.13             = kubernetes.default
DNS.14             = kubernetes.default.svc
DNS.15             = kubernetes.default.svc.cluster.local
DNS.16             = kube-apiserver
DNS.17             = kube-admin
IP.1               = 127.0.0.1
IP.2               = 10.0.2.11
IP.3               = 10.0.2.12
IP.4               = 10.0.2.13
IP.5               = 10.3.0.1
IP.6               = 10.0.2.1
IP.7               = 10.0.2.2
IP.8               = 0.0.0.0
email              = admin@domain.com
Run the below to generate the certificates CA and general.
# Generate the CA private key.
openssl genrsa -out ca-key.pem 2048
 
# Generate the CA certificate.
sed -i 's/^CN.*/CN                 = Etcd/g' cert.conf
openssl req -x509 -new -extensions v3_ca -key ca-key.pem -days 3650 \
-out ca.pem \
-subj '/C=US/ST=New York/L=New York/O=domain.com/CN=Etcd' \
-config cert.conf

# Generate the server/client private key.
openssl genrsa -out etcd-node-key.pem 2048
 
# Generate the server/client certificate request.
sed -i 's/^CN.*/CN                 = etcd-node/g' cert.conf
openssl req -new -key etcd-node-key.pem \
-newkey rsa:2048 -nodes -config cert.conf \
-subj '/C=US/ST=New York/L=New York/O=domain.com/CN=etcd-node' \
-outform pem -out etcd-node-req.pem -keyout etcd-node-req.key
 
# Sign the server/client certificate request.
openssl x509 -req -in etcd-node-req.pem -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out etcd-node.pem -days 3650 -extensions v3_req -extfile cert.conf
You should now have the below files.
-rw-r--r-- 1 root root 1679 Oct 17 16:20 ca-key.pem
-rw-r--r-- 1 root root 1269 Oct 17 16:20 ca.pem
-rw-r--r-- 1 root root   17 Oct 17 16:20 ca.srl
-rw-r--r-- 1 root root 1569 Oct 17 16:20 cert.conf
-rw-r--r-- 1 root root 1679 Oct 17 16:20 etcd-node-key.pem
-rw-r--r-- 1 root root 1622 Oct 17 16:20 etcd-node-req.pem
-rw-r--r-- 1 root root 1789 Oct 17 16:20 etcd-node.pem
-rwxr-xr-x 1 root root  925 Oct 17 16:17 run.sh
You are now ready to move to the next step, configuring Etcd – in part 2. You might also like – Other articles related to Docker Kubernetes / micro-service.
Like what you’re reading? please provide feedback, any feedback is appreciated.
0 0 votes
Article Rating
Subscribe
Notify of
guest
2 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Dan
Dan
August 22, 2018 5:05 am

Good Morning

Would it be possible to provide an example of “3.Optional create worker certificates”

This guide has been really useful in understanding how to secure K8s in my own test environment, had been struggling to find any relevant information on-line previously.

2
0
Would love your thoughts, please comment.x
()
x
%d bloggers like this: