Enabling, Configuring RBAC, TLS Node Bootstrapping On An Existing Kubernetes Cluster – Part 7

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)

Configuring RBAC, TLS Node Bootstrapping On An Existing Kubernetes(1.11) Cluster.

Below is a continuation to my previous post(S) part 1-6 on how to configure Kubernetes 3 Master Node cluster.

In the post below I am going to show you.

  1. How to enable and configure RBAC on an your existing kubernetes cluster.
  2. how to automatically bootstrap a Kubernetes worker Node, when SSL/TLS is being used.

Please check out the full series to see how to configure a 3 node Kubernetes master, the links are below.

This is Part 7 – Enabling / Configuring RBAC, TLS Node bootstrapping.

Enabling RBAC in your Kubernetes cluster.

A very important aspect of every Kubernetes deployment is security.
One of the security features Kubernetes added doing the years is, Role-based access control (RBAC). it was in beta since version 1.6, but is now very stable.
Note: The RBAC configuration was left out of the Kubernetes examples in the series (part 1-6) configuring Kubernetes.

Below I am going to show you how to enable and configure RBAC on your existing Kubernetes cluster
Note: The below configuration assumes you are using a recent Kubernetes version.

So lets jump right in.

All you need to do to enable RBAC on your Kubernetes cluster is, turn on RBAC on you kube-api Nodes. since almost every movement in Kubernetes passes the API server, that is the place to enable RBAC.

However, to successful configure RBAC there are still a number of of configuration changes required, all outlined below.

Note: Using SSL and turning on RBAC is not the only security configuration to secure a Kubernetes implementation, there are a number of other configurations you should consider for example. additional authentication/authorization like LDAP,etc..

First, lets verify if our cluster supports RBAC, you do so by running the below, output should look something like the below.

As mentioned above, turning on RBAC is as as simple as adding the below in your Kubernetes api server. but in order to allow to administer your cluster, you will have to create and allow a service account, all outlined below..

First, create a service account (below i am using admin as the account), feel free to use whatever you like.

In Kubernetes there are two type’s of permission(s) assignments.

  1. Full Cluster wide
  2. Namespace only

We will be using the pre-defined cluster-admin role, which grants cluster wide access.

Note: In the recent version(s) of Kubernetes is included a list of pre-defined roles, in older versions you might need to create the role your self.

You can check all available roles by running the below, we will be using the cluster-admin role.

The below is only necessary if you followed this series part 1-6.
When using SSL Kubernetes uses as the identification / authentication the cn= as the user account, meaning if cn=usera, then usera will be the user who we need to grant access wrights.
In our case I used for the cn=, CN=etcd-node.

Now, If you like you can keep the name as CN=etcd-node, and keep that name to grand the required access below. but what if you wont to change that.

You can to re-generate the certificates by using the steps in Part 2.
Then copy (scp) all the *pem files that get generate to /etc/kubernetes/ssl (make sure to stop all your master and worker nodes before doing so).

I decided to modify/update the certificate definitions with the user name admin(like the below), then redistributed(scp) all the certificates.
Note: I changed the cn= to be cn=admin and the CA CN to be CN=Kube-CA.
Below is an updated certificate generating script you can use.

Below is the output of the new generated certificate. Notice, I left the file name the same as it would require to many changes.
Notice the CN=Kube-CA and CN=admin.

We now need to assign the cluster admin role to the new admin service account we created.
Note: If you didn’t re-do your certificates, then use etcd-node instead of admin below.

Note: Without running the clusterrolebinding, the kube-api server will not come online after enabling RBAC(below).

To verify the cluster role got created as expected, just run the below.

Now, lets enable RBAC in you api server, you do so by adding the below.

Also, replace the below line.

Note: Make sure to create the below Role and RoleBinding, this will allow the kube-proxy running on the worker nodes to generate the proper iptable rules for clusterIP / service IP access.
Create the ClusterRoleBinding.

Create the cluster Role.

To make sure the proper iptable rules got generated run the below and look for your service ip’s.

Make sure your API server(s) are restarted properly. In many instances you will need a full master node restart, otherwise you might see many errors and the node will not work as expected.

Now is a good idea to updated your coredns configuration to work with RBAC.
Below is an updated coredns.yaml which includes RBAC as part of the config.

Just apply the configuration by running the below.

Next, lets move on to Node bootstrapping

Auto SSL/TLS Node bootstrapping

If you run a kubernetes cluster with SSL enabled, one of the issues you might run in-to (sooner or latter), are generating Node SSL certificates to work with the masters as well as certificate rotation, more is explained below.

When we initially configure our kubernetes cluster, we used/generated our own CA certificate. we also generated a certificate containing a list of Alt valid Names and Alt valid IP’s .

For example in our case we used kmaster1-kmaster3 as well as knode1-knode3. Now what happens in day2, if we would like to add Worker Node4(knode4), we would normally have to re-generated all certificates to achieve that.
Also, we would like to streamline / automate the creation of new additional worker nodes with SSL fully working.

To address this issues, kubernetes created a set of additional roles and process. the full process is outlined below.

Note: The process below assumes you are using a kubernetes version of 1.10+, since there are many enhancements/changes in the recent versions. However, I will try to highlight some of them key point changes.

Creating the bootstrap account(s)

Lets begin by creating a bootstrap service account (if it doesn’t exist).

Now lets assign the node bootstrap role to this user, you do so by running the below.

Configuring / generating Node token(s)

Let me try to explain the process of joining a new Node to the cluster without having a pre- generated certificate.

  • New Node comes online.
  • Connects to one of the Master API servers with SSL.
  • Authenticates by using a token.
  • Creates a certificate requests with his Node name to the API server.
  • Master API server validates the Token and certificate request
  • Master API servers (auto) approves certificate request and signs (by the Master controller-manager)
  • Master API signs the Node certificate request (by the Master controller-manager)
  • Master API hands back the signed certificate to the Node
  • The new Node creates a new Node client certificate file
  • The Node creates a permanent kubeconfig.yaml file, and comes online

So now that you have a better understating of the process, lets try to implement the changes required to makes this happen.

Note: In order for a new Node to communicate to the Master API server to request a certificate he will still need the Master CA certificate.

First, lets generated a token that we will be using as the initial (temporary) authentication, you do so by running the below.

Next, take the output and lets generated our csv token file which we will be using latter for initial authentication.

An example file would look like the below – save that in token-bootstrap-auth.csv.

Next, we have to modify our kube-apiserver.yaml
Make sure to add Node first in the –authorization-mode line.

Next add the below two lines to your config.yaml.

Create a Node bootstrap file

On the new Worker Node, we need to create a bootstrap-kubeconfig.yaml, this file will be used at the Node initial bootstrap process i.e. to request a certificate.
Note: The token line below correspond and shuld contain to the same tokan generated on the master in token-bootstrap-auth.csv.

We are almost done.
The last thing we need to modify is the kubelet.service file.

Make sure to remove the /etc/kubernetes/ssl/kubeconfig.yaml (if you have one), by running the below, otherwise the bootstarp wont kick-in.

Now we are ready for prime time.
Reload the kubelet service and start the process.

To verify all is working as expected, head over to the master and run the below.

You can also check for the new node, something like the below.

If you are running a cluster with an older kubernetes version, you might need to add the below entry’s which are now obsolete in version 1.11.

By default kubernetes certificates are generated with a 1 year expiration, to update / change that for a longer/shorter period just set the below.

Kubernetes pre- 1.9 might require additional steps outlined below, not tested but feel free to try.

Generating a user certificate for cluster access

It is a good idea to generate a certificate for an admin (or regular) user to use while administrating the cluster. below are outlined the steps to do so.

First lets prepare our user certifcate reqest.
Note: The example below uses a user account of usera.

Reqest the user usera certfricate to be signed by the cluster

Next, Approve the usera certfricate reqest, you do so by running the below.

Last, sign the certificate reqest, you do so by running the below.

To verify the newly created certfciate, just run the below.

To assign privleges to this user (usera), run the below. (by default he will have no privlages assigned).
Note: The below will assign full cluster admin rights to usera, this might not be the desired results you like. you shuld rely only assign to the user the role he rely needs (use kubectl get clusterroles for a list of roles).

To set and use the newly created certficate for usera, just run the below.
Note: You can always set KUBECONFIG=your_new_kubeconfig_config_file (in our case ~/.kube/config), and kubectl will use that.

Helpful commends to check your Kubernetes roles and bindings.

On-line helpful resource
Effective RBAC – Jordan Liggitt, Red Hat

Other On-line resource defiantly helpful – maybe a bit outdated with the most recent kubernetes versions.

In Part 8 I will show you how to install / Configuring Helm, Prometheus, Alertmanager, Grafana and Elasticsearch.

You might also like – Other related articles to Docker and Kubernetes / micro-service.

Like what you’re reading? please provide feedback, any feedback is appreciated.

Leave a Reply

Notify of