Installing a Kubernetes 1.11 Cluster On CentOS 7.5(1804) The Manual Way – Install/configure the Kubernetes VM – Part 2

1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 5.00 out of 5)
Loading...

Installing, configuring 3 node Kubernetes(master) cluster on CentOS 7.5 – configure the Kubernetes VM’s

In Part 1 I described how to install and configure the bear-metal OS hosting all the Kubernetes VM’s, below I am continuing with with the installation and configuration of all Kubernetes VM/masters.

This is Part 2 – Installing the Kubernetes VM’s.

Installing the Kubernetes VM’s

  • Install CentOS 7.5(1804) / (CentOS-7-x86_64-Minimal-1804.iso), on 5 Virtual Box VM’s.
  • 3 VM’s will be configured as Masters, and 2 will be configured as worker nodes.
    • The host names, IP Address’s used in my configuration are below (feel free to come up with your own configuration schema).

      Configure each VM with the below resources.

      1. 1 Virtual CPU is fine.
      2. At least 2Gb of RAM.
      3. At least 12Gb HDD.
      4. Assign Network Switch to network port.

      Set the below on each Kubernetes VM (master and Workers).
      Disable SE Linux by Running the below.

      setenforce 0
      
      # Save the config after a reboot
      sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
      

      Disable swap

      swapoff -a
      
      # Save the config after a reboot by commenting in /etc/fstab
      #/dev/mapper/centos-swap swap                    swap    defaults        0 0
      

      If you are behind a firewall or corporate proxy, add your proxy to /etc/yum.conf and /etc/environment (for an example check out in part 1).

      modprobe br_netfilter
      echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
      echo '1' > /proc/sys/net/ipv4/ip_forward
      
      # Run sysctl to verify
      sysctl -p
      

      Install Docker packages

      yum install -y yum-utils device-mapper-persistent-data lvm2
      yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
      yum install -y docker-ce
      

      Add kubernetes repo

      cat < /etc/yum.repos.d/kubernetes.repo
      [kubernetes]
      name=Kubernetes
      baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
      enabled=1
      gpgcheck=1
      repo_gpgcheck=1
      gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
              https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
      EOF
      

      Install kubernetes and other related packages

      yum install -y kubelet kubeadm kubectl flannel epel-release conntrack jq vim bash-completion lsof screen git net-tools && yum update -y
      

      Since we are not going to use kubeadm for our configuration comment out all entry’s in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf like the below example.

      cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
      # Note: This dropin only works with kubeadm and kubelet v1.11+
      #[Service]
      #Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
      #Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
      # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
      #EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
      # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
      # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
      #EnvironmentFile=-/etc/sysconfig/kubelet
      #ExecStart=
      #ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
      

      Disable the system firewall by running the below, it would be partially managed by kubernetes.

      systemctl disable firewalld.service
      systemctl stop firewalld
      

      Note: There are other options to deal with firewall rules, for example just enabling the ports required by kubernetes.

      Lastly before continuing reboot each vm instances.

      Configuring etcd – kubernetes SSL certificates

      In order for all communication in kubernetes to be secure we need to create certificates.
      Note: In order to keep things the simplest possible, I will be using the same certificate key for all components. in production you might consider breaking those out in your production environment.

      Creating Kubernetes SSL certificates

      Create a cert.conf file with the content below. add other host names/ ip address as needed, also make sure to replace with your domain name.

      [req] 
      default_bits       = 2048
      prompt             = no
      default_md         = sha256
      distinguished_name = dn
      req_extensions     = v3_req
      x509_extensions    = v3_ca
       
      [ dn ]
      C                  = US
      ST                 = NY
      L                  = New York
      O                  = Company1
      OU                 = Ops
      CN                 = etcd-node
       
      [ v3_ca ]
      keyUsage = critical,keyCertSign, cRLSign
      basicConstraints = critical,CA:TRUE
      subjectKeyIdentifier = hash
      
      [ v3_req ]
      keyUsage = critical,digitalSignature, keyEncipherment, nonRepudiation
      extendedKeyUsage = clientAuth, serverAuth
      basicConstraints = critical,CA:FALSE
      subjectKeyIdentifier = hash
      subjectAltName = @alt_names
       
      [ alt_names ]
      DNS.1              = kubernetes
      DNS.2              = kubernetes.default
      DNS.3              = kubernetes.default.svc
      DNS.4              = kubernetes.default.svc.cluster.local
      DNS.5              = kube-apiserver
      DNS.6              = kube-admin
      DNS.7              = localhost
      DNS.8              = domain.com
      DNS.9              = kmaster1
      DNS.10              = kmaster2
      DNS.11              = kmaster3
      DNS.12              = kmaster1.local
      DNS.13              = kmaster2.local
      DNS.14              = kmaster3.local
      DNS.15              = kmaster1.domain.com
      DNS.16              = kmaster2.domain.com
      DNS.17              = kmaster3.domain.com
      DNS.18              = knode1
      DNS.19              = knode2
      DNS.20              = knode3
      DNS.21              = knode1.domain.com
      DNS.22              = knode2.domain.com
      DNS.23              = knode3.domain.com
      
      IP.1              = 127.0.0.1
      IP.2              = 0.0.0.0
      IP.3              = 10.3.0.1
      IP.4              = 10.3.0.10
      IP.5              = 10.3.0.50
      IP.6              = 172.20.0.1
      IP.7              = 172.20.0.2
      IP.8              = 172.20.0.11
      IP.9              = 172.20.0.12
      IP.10              = 172.20.0.13
      IP.11              = 172.20.0.51
      IP.12              = 172.20.0.52
      IP.13              = 172.20.0.53
      
      email              = admin@example.com
      

      Next, Copy the below and create a file called run.sh in the same directory as cert.conf file.

      # Generate the CA private key
      openssl genrsa -out ca-key.pem 2048
      sed -i 's/^CN.*/CN                 = Etcd/g' cert.conf
      
      # Generate the CA certificate.
      openssl req -x509 -new -extensions v3_ca -key ca-key.pem -days 3650 \
      -out ca.pem \
      -subj '/C=US/ST=New York/L=New York/O=example.com/CN=Etcd' \
      -config cert.conf
      
       # Generate the server/client private key.
      openssl genrsa -out etcd-node-key.pem 2048
      sed -i 's/^CN.*/CN                 = etcd-node/g' cert.conf
      
      # Generate the server/client certificate request.
      openssl req -new -key etcd-node-key.pem \
      -newkey rsa:2048 -nodes -config cert.conf \
      -subj '/C=US/ST=New York/L=New York/O=example.com/CN=etcd-node' \
      -outform pem -out etcd-node-req.pem -keyout etcd-node-req.key
      # Sign the server/client certificate request.
      openssl x509 -req -in etcd-node-req.pem \
      -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
      -out etcd-node.pem -days 3650 -extensions v3_req -extfile cert.conf
      

      Next, make the file executable, and run to create the certificates. output would look something like the below.

      chmod +x run.sh
      
      ./run.sh 
      Generating RSA private key, 2048 bit long modulus
      ......................+++
      ................................................+++
      e is 65537 (0x10001)
      Generating RSA private key, 2048 bit long modulus
      ..................................+++
      ........................................................................+++
      e is 65537 (0x10001)
      Signature ok
      subject=/C=US/ST=NY/L=New York/O=Company1/OU=Ops/CN=etcd-node
      Getting CA Private Key
      

      To verify the certificate, run the below.

      openssl x509 -in etcd-node.pem -text -noout
      

      Once completed you should be left with the below list of files.

      ca-key.pem  ca.pem  ca.srl  cert.conf  etcd-node-key.pem  etcd-node-req.pem  etcd-node.pem
      

      From the list of above files we only need the ca.pem, ca-key.pem, etcd-node.pem and etcd-node-key.pem.

      Create the directory if not exist, and copy the certificate files.

      mkdir -p /etc/kubernetes/ssl
      ln -s /etc/kubernetes/ssl /etc/kubernetes/pki
      cp ca.pem ca-key.pem etcd-node.pem etcd-node-key.pem /etc/kubernetes/ssl
      

      Note: I named the most used key certificate as etcd-node-(key).pem, since its the first SSL certificate being used, feel free to rename at your own desire(just remember to update all references).

      Etcd installation and configuration

      Add to /etc/environment

      ETCDCTL_SSL_DIR=/etc/kubernetes/ssl
      ETCD_SSL_DIR=/etc/kubernetes/ssl
      ETCDCTL_CA_FILE=/etc/kubernetes/ssl/ca.pem
      ETCDCTL_CERT_FILE=/etc/kubernetes/ssl/etcd-node.pem
      ETCDCTL_KEY_FILE=/etc/kubernetes/ssl/etcd-node-key.pem
      ETCDCTL_ENDPOINTS="https://172.20.0.11:2379,https://172.20.0.12:2379,https://172.20.0.13:2379"
      ETCD_ENDPOINTS="https://172.20.0.11:2379,https://172.20.0.12:2379,https://172.20.0.13:2379"
      FLANNELD_ETCD_ENDPOINTS="https://172.20.0.11:2379,https://172.20.0.12:2379,https://172.20.0.13:2379"
      FLANNELD_IFACE="enp0s3"
      FLANNELD_ETCD_PREFIX="/coreos.com/network"
      

      If you are behind a corporate firewall or proxy, set the below in /etc/systemd/system/docker.service.d/http-proxy.conf (create the directory if not exist).

      Environment="HTTP_PROXY=http://your_proxy_ip_:1234/"
      Environment="HTTPS_PROXY=http://your_proxy_ip_:1234/"
      Environment="NO_PROXY=.domain.com,127.0.0.1,localhost,kmaster1,kmaster1.domain.com,kmaster2,kmaster2.domain.com,kmaster3,kmaster3.domain.com,172.20.0.12,172.20.0.11,172.20.0.13"
      

      Install etcd on the 3 master servers

      Download the latest etcd and configure

      mkdir -p /var/lib/etcd
      curl -# -LO https://github.com/coreos/etcd/releases/download/v3.3.8/etcd-v3.3.8-linux-amd64.tar.gz
      tar xf etcd-v3.3.8-linux-amd64.tar.gz
      chown -Rh root:root etcd-v3.3.8-linux-amd64
      find etcd-v3.3.8-linux-amd64 -xdev -type f -exec chmod 0755 '{}' \;
      cp etcd-v3.3.8-linux-amd64/etcd* /usr/bin/
      

      Create etcd service, like the below.
      cat /etc/systemd/system/etcd.service
      Note: Replace on each each master the below values with the proper name/ip.

      1. –name=kmaster1
      2. –listen-peer-urls=https://172.20.0.11:2380
      3. –advertise-client-urls=https://172.20.0.11:2379
      4. –initial-advertise-peer-urls=https://172.20.0.11:2380
      5. –listen-client-urls=https://172.20.0.11:2379,https://127.0.0.1:2379,https://127.0.0.1:4001

      The example below is from master1.

      [Unit]
      Description=Etcd Server
      Documentation=https://coreos.com/etcd/docs/latest/
      After=network.target
      After=network-online.target
      Wants=network-online.target
      
      [Service]
      ExecStart=/usr/bin/etcd \
      --name=kmaster1 \
      --cert-file=/etc/kubernetes/pki/etcd-node.pem \
      --key-file=/etc/kubernetes/pki/etcd-node-key.pem \
      --peer-cert-file=/etc/kubernetes/pki/etcd-node.pem \
      --peer-key-file=/etc/kubernetes/pki/etcd-node-key.pem \
      --trusted-ca-file=/etc/kubernetes/pki/ca.pem \
      --peer-trusted-ca-file=/etc/kubernetes/pki/ca.pem \
      --initial-advertise-peer-urls=https://172.20.0.11:2380 \
      --listen-peer-urls=https://172.20.0.11:2380 \
      --listen-client-urls=https://172.20.0.11:2379,https://127.0.0.1:2379,https://127.0.0.1:4001 \
      --advertise-client-urls=https://172.20.0.11:2379 \
      --initial-cluster-token=etcd-token \
      --initial-cluster=kmaster1=https://172.20.0.11:2380,kmaster2=https://172.20.0.12:2380,kmaster3=https://172.20.0.13:2380 \
      --data-dir=/var/lib/etcd \
      --initial-cluster-state=new
      Restart=always
      RestartSec=5s
      User=etcd
      
      [Install]
      WantedBy=multi-user.target
      

      Create an etcd user

      useradd etcd
      chown -R etcd:etcd /var/lib/etcd
      

      Add network configuration on all 3 masters, by running the below.

      cat <  /etc/sysctl.d/k8s.conf
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      net.bridge.bridge-nf-call-arptables = 1
      EOF
      
      # Reload stack
      sysctl --system
      

      Start and Enable etcd on all 3 masters.

      systemctl enable etcd && systemctl start etcd
      
      # Verify if etcd started.
      systemctl status etcd
      OR
      journalctl -u etcd
      

      Verify if etcd works across all nodes.

      etcdctl member list
      7c8d40e4de52c1b9: name=kmaster2 peerURLs=https://172.20.0.12:2380 clientURLs=https://172.20.0.12:2379 isLeader=false
      96e236888999c3ca: name=kmaster3 peerURLs=https://172.20.0.13:2380 clientURLs=https://172.20.0.13:2379 isLeader=true
      9e191dbdf076e744: name=kmaster1 peerURLs=https://172.20.0.11:2380 clientURLs=https://172.20.0.11:2379 isLeader=false
      

      Note: The isLeader=true will automatically get elected/updated based on last elected etcd master (in the case above its kmaster3).

      Create flannel network key in etcd, by running the below.

      /usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.20.0.0/20", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }
      

      Verify that the key got created.

      etcdctl get /coreos.com/network/config
      { "Network": "10.20.0.0/20", "SubnetLen": 24, "Backend": { "Type": "vxlan", "VNI": 1 } }
      

      In Part 3 will continue configuring Flannel and Docker.

      You might also like – Other related articles to Docker and Kubernetes / micro-service.

      Like what you’re reading? please provide feedback, any feedback is appreciated.

9
Leave a Reply

avatar
3000
1 Comment threads
8 Thread replies
1 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Eli Kleinmancamer Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
camer
Guest
camer

Hello Eli, I have installed last version of etcd (v3.3.12) but I have configured kmaster1 as you said but to start by systemctl I got “etcd.service: main process exited, code=exited, status=1/FAILURE”. However I can start “etcd” by command successfully. Outpout for journalctl -u etcd.service : Apr 04 06:45:10 localhost.localdomain systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE Apr 04 06:45:10 localhost.localdomain systemd[1]: Unit etcd.service entered failed state. Apr 04 06:45:10 localhost.localdomain systemd[1]: etcd.service failed. Apr 04 06:45:16 localhost.localdomain systemd[1]: etcd.service holdoff time over, scheduling restart. Apr 04 06:45:16 localhost.localdomain systemd[1]: Stopped Etcd Server. Apr 04 06:45:16 localhost.localdomain systemd[1]: Started Etcd Server.… Read more »