VCluster Introduction
Does a Kubernetes analog to Docker-in-Docker (DinD) exist?
Well it turns out that there is such a technology. However, it is not called KinK (for better or worse depending on your view). I am talking about vCluster. vCluster allows you to run a virtual Kubernetes cluster within your already existing host cluster. In fact, you can run a vCluster within another vCluster. What makes vCluster an amazing tool is that in order to create a virtual cluster you do not need elevated privileges within the cluster. In fact if you are able to deploy a simple web app in a namespace you are more than likely able to create a vCluster.
Use Cases
vCluster allows us to run a brand new cluster within our host cluster in a fashion that requires very minimal additional resource usage. Through using a virtual cluster we can more easily accomplish tasks that would sometimes require the creation and management of another host k8s cluster.
- Multi tenancy
- Virtual admin access within a controlled environment
- Test out new k8s versions
Although multi tenancy could be accomplished to some extent through the use of namespaces, the other two example use cases can not. However, if you use namespace isolation you can’t make a CRD available to one tenant but not another. You could restrict access to the CRD through k8s RBAC, but in some sense both tenants will “know” of the CRD since it is a cluster level resource. But with a vCluster, the CRDs defined in one cluster aren’t known to/shared with another vCluster.
Architecture
A vCluster consists of two main components, the control plane and syncer. The control plane is responsible for vCluster’s API server, controller manager, and the connection used to talk to its datastore. By default the used datastore is a sqlite database, but this can be swapped out with minimal impact on the architecture. While the syncer works with the host cluster’s scheduler to place pods on nodes and sharing this information with the vCluster. It is syncs other low level objects that involve networking, such as services, with the host cluster.
The control plane lives within a namespace of the host cluster, while the syncer lives within
the host cluster’s kube-system namespace. A high level visual representation of the interaction
between the virtual and host cluster is depicted below, in an image copied from the vCluster
documentation.
Nodes and scheduling
Since a vCluster offloads the task of scheduling pods to the host cluster, it also makes sense
that it does not contain any of its own nodes. Instead the nodes that are returned by running
kubectl get nodes from within the vCluster are fake nodes. One for each spec.nodeName that
is encountered. In some cases, for instance running daemon sets, it is beneficial to copy the
real node’s data with the fake node used by vCluster. This is accomplished through modifying the
.sync.nodes.enabled, .sync.nodes.syncAllNodes, and .sync.nodes.nodeSelector helm chart values.
Additionally, the scheduling of pods is delegated to the host cluster’s scheduler by default. However, this limits in the types of pod distributions we are able to enforce. This is due to the fact that some node data, labels for instance, are may not always be kept in sync with the node data of the vCluster. This is the case when using the default node syncing behavior. To overcome this you can do one of the following:
- Have the vCluster use its own scheduler instead of the host cluster’s scheduler.
- Provide helm values that allow the syncer to use an updated RBAC role to allow for more “powerful” use of the host scheduler.
- Enable node data syncing. This way the vCluster scheduler will adhere to rules caused by the node topology.
More information about vCluster node data syncing and pod scheduling can be found in the documentation (pod scheduling, node data).
Installation
In order to install vCluster you will first need to have helm installed. Once installed, the next step is to install the vCluster CLI. This will mostly be used to allow us to easily switch between the host cluster and any vCluster that we may create. Alternatively, we could change which KUBECONFIG file that is being used. But I find it easier to just use the CLI to perform this task.
The process used to install the CLI will vary depending on what OS you are running. Instructions
for Linux, Mac, and Windows can be found here. Once installed, we will need to add the loft-sh helm repository.
helm repo add loft-sh https://charts.loft.sh
helm repo update
Once the loft-sh repo is added you can use helm search repo loft-sh to see which Kubernetes
variants can be used out of the box to create a virtual cluster. By default k3s will be used, but
you could also use k0s or even k8s. For this post I will use the default k3s variant and install
it within the k3d cluster that I have created (see previous post in this series).
To create a virtual cluster you can either use the vCluster CLI or use kubectl. I personally prefer to use the kubectl approach since I can save the cluster configuration. Which can later be reused or updated as needed.
kubectl create namespace my-vcluster-ns
helm template my-vcluster loft-sh/vcluster -n my-vcluster-ns > my_vcluster.yaml
kubectl apply -f my_vcluster.yaml
Or if you want, with the vCluster CLI (which will create the vcluster-my_vcluster namespace
automatically).
vcluster create my_vcluster
After running one of the above commands you should check that the cluster is available either via
vcluster list and see that the cluster is running or kubectl -n my-vcluster-ns get sts and
see that all of the StatefulSet pods are running.
Expose Servers
With applications being added to the vCluster we would like the ability to interact with those applications from outside of our cluster. Be that outside of the vCluster, with host cluster apps sending it requests, or even from outside of the host cluster. Since a vCluster will sync its Pod and Services resources with the host cluster, host cluster to vCluster communication is possible by using the cluster’s DNS. Or if we want via port-forwarding or an Ingress within the host cluster.
However, it would be preferable to instead specify the Ingress resource within the vCluster
and not the host-cluster. This way we don’t have to look up the Service’s modified name within
the host cluster before creating the Ingress routing rule. Unfortunately, this capability isn’t
available out of the box by default since Ingress resources are stored within the vCluster and
not synced with the host cluster. But enabling the sharing of Ingress resources is possible with a
simple modification. We add the following to a values.yaml file
sync:
ingresses:
enabled: true
Then run vcluster create my-vcluster --update -f values.yaml. Or if you are creating a cluster
for the first time helm template my-vcluster loft-sh/vcluster -n my-vcluster-ns -f values.yaml.