1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 1.00 out of 5)

Configuring RBAC, TLS Node Bootstrapping On An Existing Kubernetes(1.11) Cluster.

Below is a continuation to my previous post(S) part 1-6 on how to configure Kubernetes 3 Master Node cluster. In the post below I am going to show you.
  1. How to enable and configure RBAC on an your existing kubernetes cluster.
  2. how to automatically bootstrap a Kubernetes worker Node, when SSL/TLS is being used.
Please check out the full series to see how to configure a 3 node Kubernetes master, the links are below. This is Part 7 – Enabling / Configuring RBAC, TLS Node bootstrapping.

Enabling RBAC in your Kubernetes cluster.

A very important aspect of every Kubernetes deployment is security. One of the security features Kubernetes added doing the years is, Role-based access control (RBAC). it was in beta since version 1.6, but is now very stable. Note: The RBAC configuration was left out of the Kubernetes examples in the series (part 1-6) configuring Kubernetes. Below I am going to show you how to enable and configure RBAC on your existing Kubernetes cluster Note: The below configuration assumes you are using a recent Kubernetes version. So lets jump right in. All you need to do to enable RBAC on your Kubernetes cluster is, turn on RBAC on you kube-api Nodes. since almost every movement in Kubernetes passes the API server, that is the place to enable RBAC. However, to successful configure RBAC there are still a number of of configuration changes required, all outlined below. Note: Using SSL and turning on RBAC is not the only security configuration to secure a Kubernetes implementation, there are a number of other configurations you should consider for example. additional authentication/authorization like LDAP,etc.. First, lets verify if our cluster supports RBAC, you do so by running the below, output should look something like the below.
kubectl api-versions|grep authorization
As mentioned above, turning on RBAC is as as simple as adding the below in your Kubernetes api server. but in order to allow to administer your cluster, you will have to create and allow a service account, all outlined below.. First, create a service account (below i am using admin as the account), feel free to use whatever you like.
kubectl create serviceaccount admin --namespace kube-system

# Verify the new account
kubectl get serviceAccounts -n kube-system admin -o yaml
In Kubernetes there are two type’s of permission(s) assignments.
  1. Full Cluster wide
  2. Namespace only
We will be using the pre-defined cluster-admin role, which grants cluster wide access. Note: In the recent version(s) of Kubernetes is included a list of pre-defined roles, in older versions you might need to create the role your self. You can check all available roles by running the below, we will be using the cluster-admin role.
kubectl get clusterroles
The below is only necessary if you followed this series part 1-6. When using SSL Kubernetes uses as the identification / authentication the cn= as the user account, meaning if cn=usera, then usera will be the user who we need to grant access wrights. In our case I used for the cn=, CN=etcd-node. Now, If you like you can keep the name as CN=etcd-node, and keep that name to grand the required access below. but what if you wont to change that. You can to re-generate the certificates by using the steps in Part 2. Then copy (scp) all the *pem files that get generate to /etc/kubernetes/ssl (make sure to stop all your master and worker nodes before doing so). I decided to modify/update the certificate definitions with the user name admin(like the below), then redistributed(scp) all the certificates. Note: I changed the cn= to be cn=admin and the CA CN to be CN=Kube-CA. Below is an updated certificate generating script you can use.
# Generate the CA private key
openssl genrsa -out ca-key.pem 2048
sed -i 's/^CN.*/CN                 = Kube-CA/g' cert.conf

# Generate the CA certificate.
openssl req -x509 -new -extensions v3_ca -key ca-key.pem -days 3650 \
-out ca.pem \
-subj '/C=US/ST=New York/L=New York/' \
-config cert.conf

 # Generate the server/client private key.
openssl genrsa -out etcd-node-key.pem 2048
sed -i 's/^CN.*/CN                 = admin/g' cert.conf

# Generate the server/client certificate request.
openssl req -new -key etcd-node-key.pem \
-newkey rsa:2048 -nodes -config cert.conf \
-subj '/C=US/ST=New York/L=New York/' \
-outform pem -out etcd-node-req.pem -keyout etcd-node-req.key
# Sign the server/client certificate request.
openssl x509 -req -in etcd-node-req.pem \
-CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out etcd-node.pem -days 3650 -extensions v3_req -extfile cert.conf
Below is the output of the new generated certificate. Notice, I left the file name the same as it would require to many changes. Notice the CN=Kube-CA and CN=admin.
openssl x509 -in /etc/kubernetes/ssl/etcd-node.pem -text -noout
        Version: 3 (0x2)
        Serial Number:
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=NY, L=New York, O=Company1, OU=Ops, CN=Kube-CA
            Not Before: Aug 16 18:40:00 2018 GMT
            Not After : Aug 13 18:40:00 2028 GMT
        Subject: C=US, ST=NY, L=New York, O=Company1, OU=Ops, CN=admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
            X509v3 Subject Key Identifier:
            X509v3 Subject Alternative Name:
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:kube-apiserver, DNS:kube-admin, DNS:localhost,, DNS:kmaster1, DNS:kmaster2, DNS:kmaster3, DNS:kmaster1.local, DNS:kmaster2.local, DNS:kmaster3.local,,,, DNS:knode1, DNS:knode2, DNS:knode3,,,, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:, IP Address:,
    Signature Algorithm: sha256WithRSAEncryption
We now need to assign the cluster admin role to the new admin service account we created. Note: If you didn’t re-do your certificates, then use etcd-node instead of admin below.
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--namespace=kube-system \

# If you left the certificate created in part2 run the below.
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--namespace=kube-system \
Note: Without running the clusterrolebinding, the kube-api server will not come online after enabling RBAC(below). To verify the cluster role got created as expected, just run the below.
kubectl get clusterrolebinding cluster-admin-binding -o yaml
Now, lets enable RBAC in you api server, you do so by adding the below.
cat /etc/kubernetes/manifests/kube-apiserver.yaml
- apiserver   <<<<<---- add in this section of the file
... [snip]
- --authorization-mode=Node,RBAC
... [snip]
Also, replace the below line.
cat /etc/kubernetes/manifests/kube-apiserver.yaml
# From
    - --runtime-config=extensions/v1beta1/networkpolicies=true,extensions/v1beta1=true

# To
    - --runtime-config=extensions/v1beta1/networkpolicies=true,extensions/v1beta1=true,
Note: Make sure to create the below Role and RoleBinding, this will allow the kube-proxy running on the worker nodes to generate the proper iptable rules for clusterIP / service IP access. Create the ClusterRoleBinding.
cat <
Create the cluster Role.
cat <
To make sure the proper iptable rules got generated run the below and look for your service ip’s.
iptables -t nat -L

# Example of iptable rule with service ip.
iptables -t nat -L|grep 10.3
KUBE-MARK-MASQ  tcp  -- !             /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  anywhere                /* default/kubernetes:https cluster IP */ tcp dpt:https
KUBE-MARK-MASQ  udp  -- !            /* kube-system/kube-dns:dns cluster IP */ udp dpt:domain
Make sure your API server(s) are restarted properly. In many instances you will need a full master node restart, otherwise you might see many errors and the node will not work as expected. Now is a good idea to updated your coredns configuration to work with RBAC. Below is an updated coredns.yaml which includes RBAC as part of the config.
apiVersion: v1
kind: ServiceAccount
  name: coredns
  namespace: kube-system
  labels: "true" Reconcile
kind: ClusterRole
  labels: rbac-defaults Reconcile
  name: system:coredns
- apiGroups:
  - ""
  - endpoints
  - services
  - pods
  - namespaces
  - list
  - watch
kind: ClusterRoleBinding
  annotations: "true"
  labels: rbac-defaults EnsureExists
  name: system:coredns
  kind: ClusterRole
  name: system:coredns
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
apiVersion: v1
kind: ConfigMap
  name: coredns
  namespace: kube-system
  labels: EnsureExists
  Corefile: |
    .:53 {
        kubernetes cluster.local {
            pods insecure
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30 cluster.local
apiVersion: extensions/v1beta1
kind: Deployment
  name: coredns
  namespace: kube-system
    k8s-app: kube-dns "true" Reconcile "CoreDNS"
  replicas: 2
    type: RollingUpdate
      maxUnavailable: 1
      k8s-app: kube-dns
        k8s-app: kube-dns
      annotations: 'docker/default'
      serviceAccountName: coredns
        - key:
          effect: NoSchedule
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      - name: coredns
        image: coredns/coredns:1.2.1
        imagePullPolicy: IfNotPresent
            memory: 170Mi
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
          allowPrivilegeEscalation: false
            - NET_BIND_SERVICE
            - all
          readOnlyRootFilesystem: true
      dnsPolicy: Default
        - name: config-volume
            name: coredns
            - key: Corefile
              path: Corefile
apiVersion: v1
kind: Service
  name: kube-dns
  namespace: kube-system
  annotations: "9153" "true"
    k8s-app: kube-dns "true" Reconcile "CoreDNS"
    k8s-app: kube-dns
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
Just apply the configuration by running the below.
kubectl apply -f coredns.yaml
Next, lets move on to Node bootstrapping

Auto SSL/TLS Node bootstrapping

If you run a kubernetes cluster with SSL enabled, one of the issues you might run in-to (sooner or latter), are generating Node SSL certificates to work with the masters as well as certificate rotation, more is explained below. When we initially configure our kubernetes cluster, we used/generated our own CA certificate. we also generated a certificate containing a list of Alt valid Names and Alt valid IP’s . For example in our case we used kmaster1-kmaster3 as well as knode1-knode3. Now what happens in day2, if we would like to add Worker Node4(knode4), we would normally have to re-generated all certificates to achieve that. Also, we would like to streamline / automate the creation of new additional worker nodes with SSL fully working. To address this issues, kubernetes created a set of additional roles and process. the full process is outlined below. Note: The process below assumes you are using a kubernetes version of 1.10+, since there are many enhancements/changes in the recent versions. However, I will try to highlight some of them key point changes.

Creating the bootstrap account(s)

Lets begin by creating a bootstrap service account (if it doesn’t exist).
kubectl create serviceaccount kubelet-bootstrap --namespace kube-system
Now lets assign the node bootstrap role to this user, you do so by running the below.
kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \

Configuring / generating Node token(s)

Let me try to explain the process of joining a new Node to the cluster without having a pre- generated certificate.
  • New Node comes online.
  • Connects to one of the Master API servers with SSL.
  • Authenticates by using a token.
  • Creates a certificate requests with his Node name to the API server.
  • Master API server validates the Token and certificate request
  • Master API servers (auto) approves certificate request and signs (by the Master controller-manager)
  • Master API signs the Node certificate request (by the Master controller-manager)
  • Master API hands back the signed certificate to the Node
  • The new Node creates a new Node client certificate file
  • The Node creates a permanent kubeconfig.yaml file, and comes online
So now that you have a better understating of the process, lets try to implement the changes required to makes this happen. Note: In order for a new Node to communicate to the Master API server to request a certificate he will still need the Master CA certificate. First, lets generated a token that we will be using as the initial (temporary) authentication, you do so by running the below.
head -c 16 /dev/urandom | od -An -t x | tr -d ' '
Next, take the output and lets generated our csv token file which we will be using latter for initial authentication. An example file would look like the below – save that in token-bootstrap-auth.csv.
cat /etc/kubernetes/ssl/token-bootstrap-auth.csv
Next, we have to modify our kube-apiserver.yaml Make sure to add Node first in the –authorization-mode line.
cat /etc/kubernetes/manifests/kube-apiserver.yaml
- apiserver <<<<<<<<<<-------- add to this section
... [snip]
- --authorization-mode=Node,RBAC
- --token-auth-file=/etc/kubernetes/ssl/token-bootstrap-auth.csv
... [snip]
Next add the below two lines to your config.yaml.
cat /etc/kubernetes/config.yaml
... [snip]
... <<-- add below the above line
RotateCertificates: true
ServerTLSBootstrap: true
... [snip]

Create a Node bootstrap file

On the new Worker Node, we need to create a bootstrap-kubeconfig.yaml, this file will be used at the Node initial bootstrap process i.e. to request a certificate. Note: The token line below correspond and shuld contain to the same tokan generated on the master in token-bootstrap-auth.csv.
apiVersion: v1
kind: Config
- name: local
    certificate-authority: /etc/kubernetes/ssl/ca.pem
- context:
    cluster: local
    user: tls-bootstrap-token-user
  name: tls-bootstrap-token-user@kubernets
current-context: tls-bootstrap-token-user@kubernets
preferences: {}
- name: tls-bootstrap-token-user
    token: 3af4fcd12f67ff3418fa0f1f0a43bb62
We are almost done. The last thing we need to modify is the kubelet.service file.
cat /etc/systemd/system/kubelet.service
... [snip]
--kubeconfig=/etc/kubernetes/ssl/kubeconfig.yaml \  <<<<<---- add below this line
--bootstrap-kubeconfig=/etc/kubernetes/ssl/bootstrap-kubeconfig.yaml \
--cert-dir=/etc/kubernetes/ssl \
Make sure to remove the /etc/kubernetes/ssl/kubeconfig.yaml (if you have one), by running the below, otherwise the bootstarp wont kick-in.
rm /etc/kubernetes/ssl/kubeconfig.yaml
Now we are ready for prime time. Reload the kubelet service and start the process.
systemctl daemon-reload

# Get startup debug info output
journalctl -u kubelet -f &

systemctl enable kubelet && systemctl start kubelet.
To verify all is working as expected, head over to the master and run the below.
kubectl get csr --all-namespaces -o wide
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-Wnmy0oFiycWKaT8LsWRv1EN7m333dv-ZVYz-Bd2X14w   1m        kubelet-bootstrap   Approved,Issued

# And now
kubectl describe csr node-csr-Wnmy0oFiycWKaT8LsWRv1EN7m333dv-ZVYz-Bd2X14w
Name:               node-csr-Wnmy0oFiycWKaT8LsWRv1EN7m333dv-ZVYz-Bd2X14w
CreationTimestamp:  Wed, 22 Aug 2018 15:05:09 -0400
Requesting User:    kubelet-bootstrap
Status:             Approved,Issued
         Common Name:    system:node:knode3
         Serial Number: 
         Organization:   system:nodes
You can also check for the new node, something like the below.
kubectl get nodes --all-namespaces -o wide
kmaster1   Ready         22d       v1.11.1           CentOS Linux 7 (Core)   3.10.0-862.9.1.el7.x86_64    docker://18.6.0
kmaster2   Ready         21d       v1.11.1           CentOS Linux 7 (Core)   3.10.0-862.9.1.el7.x86_64    docker://18.6.0
kmaster3   Ready         21d       v1.11.1           CentOS Linux 7 (Core)   3.10.0-862.9.1.el7.x86_64    docker://18.6.0
knode1     Ready         9d        v1.11.2           CentOS Linux 7 (Core)   3.10.0-862.9.1.el7.x86_64    docker://18.6.0
knode2     Ready         9d        v1.11.2           CentOS Linux 7 (Core)   3.10.0-862.9.1.el7.x86_64    docker://18.6.0
knode3     Ready         1d        v1.11.2           CentOS Linux 7 (Core)   3.10.0-862.11.6.el7.x86_64   docker://18.6.0

# OR
kubectl get all  --all-namespaces -o wide
NAMESPACE     NAME                                     READY     STATUS    RESTARTS   AGE       IP            NODE
... [snip]
kube-system   pod/kube-proxy-knode3                    1/1       Running   7          1d   knode3
... [snip]
If you are running a cluster with an older kubernetes version, you might need to add the below entry’s which are now obsolete in version 1.11.
# cat /etc/systemd/system/kubelet.service - instead of the consfig.yaml entry's
By default kubernetes certificates are generated with a 1 year expiration, to update / change that for a longer/shorter period just set the below.
cat /etc/kubernetes/manifests/kube-controller-manager.yaml
# Change from the default of 1 year (8760h0m0s).
... [snip]
- ./hyperkube
- controller-manager
... [snip] <<<<<<<-------------- add below this
Kubernetes pre- 1.9 might require additional steps outlined below, not tested but feel free to try.
kubectl create clusterrolebinding node-client-auto-approve-csr \ \

kubectl create clusterrolebinding node-client-auto-renew-crt \ \

# Required auto-renewal (might not be needed after 1.7), for sure not required in latest 1.11.
kubectl create clusterrolebinding node-client-auto-approve-csr --clusterrole=approve-node-client-csr --group=system:bootstrappers
kubectl create clusterrolebinding node-client-auto-renew-crt --clusterrole=approve-node-client-renewal-csr --group=system:nodes
kubectl create clusterrolebinding node-server-auto-renew-crt --clusterrole=approve-node-server-renewal-csr --group=system:nodes

Generating a user certificate for cluster access

It is a good idea to generate a certificate for an admin (or regular) user to use while administrating the cluster. below are outlined the steps to do so. First lets prepare our user certifcate reqest. Note: The example below uses a user account of usera.
openssl genrsa -out usera.pem 2048
openssl req -new -key usera.pem -out usera.csr -subj "/CN=usera"

crt_out=`cat usera.csr |base64 |tr -d '\n'`

cat usera_auth-req.yaml
cat << EOF > usera_auth-req.yaml
kind: CertificateSigningRequest
  name: user-request-usera
  - system:authenticated
  request: $crt_out
  - digital signature
  - key encipherment
  - client auth
Reqest the user usera certfricate to be signed by the cluster
kubectl apply -f usera_auth-req.yaml 

# Output shuld show the below. created
Next, Approve the usera certfricate reqest, you do so by running the below.
kubectl certificate approve user-request-usera

# Output shuld show the below. approved
Last, sign the certificate reqest, you do so by running the below.
kubectl get csr user-request-usera -o jsonpath='{.status.certificate}' | base64 -d > usera.crt
To verify the newly created certfciate, just run the below.
kubectl get csr
# Expected out shuld be somthing like the below.
NAME                 AGE       REQUESTOR          CONDITION
user-request-usera   1m        system:unsecured   Approved,Issued
To assign privleges to this user (usera), run the below. (by default he will have no privlages assigned). Note: The below will assign full cluster admin rights to usera, this might not be the desired results you like. you shuld rely only assign to the user the role he rely needs (use kubectl get clusterroles for a list of roles).
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
To set and use the newly created certficate for usera, just run the below. Note: You can always set KUBECONFIG=your_new_kubeconfig_config_file (in our case ~/.kube/config), and kubectl will use that.
kubectl --kubeconfig ~/.kube/config config set-cluster usera --insecure-skip-tls-verify=true --server=
kubectl --kubeconfig ~/.kube/config config set-credentials usera --client-certificate=usera.crt --client-key=usera.pem --embed-certs=true
kubectl --kubeconfig ~/.kube/config config set-context usera --cluster=usera --user=usera
kubectl --kubeconfig ~/.kube/config config use-context usera
Helpful commends to check your Kubernetes roles and bindings.
kubectl get roles
kubectl get rolebindings
kubectl get clusterroles
kubectl get clusterrolebindings
kubectl get clusterrole system:node -o yaml
On-line helpful resource Effective RBAC – Jordan Liggitt, Red Hat Other On-line resource defiantly helpful – maybe a bit outdated with the most recent kubernetes versions. In Part 8 I will show you how to install / Configuring Helm, Prometheus, Alertmanager, Grafana and Elasticsearch. You might also like – Other related articles to Docker and Kubernetes / micro-service. Like what you’re reading? please provide feedback, any feedback is appreciated.
0 0 votes
Article Rating
Notify of
Inline Feedbacks
View all comments
Would love your thoughts, please comment.x
%d bloggers like this: