microk8s - small kubernetes - part 3
Intro
I recently sat down and thought to myself “I wonder how the current market of mini or micro kubernetes distributions is going”. I’ve been using minikube for a bit and thought to myself it was about time to start using a few other distributions.
So I have been experimenting with different micro or mini distributions.
This post is about microk8s.
microk8s
microk8s is an open source project that you can find here: [https://microk8s.io/] (https://microk8s.io/).
microk8s is sponsored and developed primarily by Canonical.
Install
The installation is very easy and similarly low touch as the other distributuions I’ve looked at so far.
I even got very self indulgent and did another video, you can see it here
Single Node Cluster
Single node cluster installation is really easy if you use the defaults.
Run the command:
snap install microk8s --classic
It’s a SNAP install so will work out of the box on an ubuntu distribution.
Run configure hook of "microk8s" snap if present |
microk8s (1.23/stable) v1.23.4 from Canonical✓ installed
Run a check to see if microk8s is running.
root@ubuntu:~# microk8s status
microk8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
ambassador # Ambassador API Gateway and Ingress
cilium # SDN, fast with full network policy
dashboard # The Kubernetes dashboard
dashboard-ingress # Ingress definition for Kubernetes dashboard
dns # CoreDNS
fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
gpu # Automatic enablement of Nvidia CUDA
helm # Helm 2 - the package manager for Kubernetes
helm3 # Helm 3 - Kubernetes package manager
host-access # Allow Pods connecting to Host services smoothly
inaccel # Simplifying FPGA management in Kubernetes
ingress # Ingress controller for external access
istio # Core Istio service mesh services
jaeger # Kubernetes Jaeger operator with its simple config
kata # Kata Containers is a secure runtime with lightweight VMS
keda # Kubernetes-based Event Driven Autoscaling
knative # The Knative framework on Kubernetes.
kubeflow # Kubeflow for easy ML deployments
linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
metallb # Loadbalancer for your Kubernetes cluster
metrics-server # K8s Metrics Server for API access to service metrics
multus # Multus CNI enables attaching multiple network interfaces to pods
openebs # OpenEBS is the open-source storage solution for Kubernetes
openfaas # OpenFaaS serverless framework
portainer # Portainer UI for your Kubernetes cluster
prometheus # Prometheus operator for monitoring and logging
rbac # Role-Based Access Control for authorisation
registry # Private image registry exposed on localhost:32000
storage # Storage class; allocates storage from host directory
traefik # traefik Ingress controller for external access
The status command allows you to install various add-ons, such a Ingress Load Balancers and Istio. This is a fantastically easy way to try out various types of software.
I’m goint to enable some of the addons before installing a second node.
root@ubuntu:~# microk8s enable dashboard dns registry
Enabling Kubernetes Dashboard
Enabling Metrics-Server
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-admin created
Metrics-Server is enabled
Applying manifest
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
If RBAC is not enabled access the dashboard using the default token retrieved with:
token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token
In an RBAC enabled setup (microk8s enable RBAC) you need to create a user with restricted
permissions as shown in:
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
Restarting kubelet
DNS is enabled
The registry will be created with the default size of 20Gi.
You can use the "size" argument while enabling the registry, eg microk8s.enable registry:size=30Gi
Enabling default storage class
deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Applying registry manifest
namespace/container-registry created
persistentvolumeclaim/registry-claim created
deployment.apps/registry created
service/registry created
configmap/local-registry-hosting configured
The registry is enabled
The command above will enable the dashboard, coredns and a registry. Technically, I don’t need a registry, but it’s nice to know it’s there.
When I come to the ingress portions below, I will enable various default ingress controllers.
I can validate what’s running inside my cluster like this:
First I get a kubeconfig so that I can use native kubectl commands.
root@ubuntu:~# microk8s config > kubeconfig
root@ubuntu:~# more kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZlZ0F3SUJBZ0lVTkR0R1dUUjBqc21
OSWhUbTBRZTdmRDBYcVNrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0Z6RVZNQk1HQTFVRUF3d01NVEF1TVRVeUxqRTRNeTR4TUI0WERUSXlNRE15TU
RBeU16VTBNRm9YRFRNeQpNRE14TnpBeU16VTBNRm93RnpFVk1CTUdBMVVFQXd3TU1UQXVNVFV5TGpFNE15NHhNSUlCSWpBTkJna3Foa2lHCjl3M
EJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF1MXBOVDFhb21lSU9FbE5UcThFNndabVA2MFQ2b1JnUTlXeUsKaGlvVVJyaHQ0dlk3emIrRXFHRjRQ
MU9ta25LRTRjS3lYbk9mTTgxWS9qL21LSjh2VGt0Qy9XMWs4RE5keVdhZApUcXVZL2lNR002QnZBWkE3blJiUHovZW13ckErakoyUFJxaDFkbmF
vbFBIL01QY3N3R0dQRmErOHlLYW9kNFZVCjc2QXhBaXUvMGt0SWMvbnF6ZllKOXBaQTJVemRjK0NOMjlwcW1WV2svTDJ5OTFwU0pWckxDQ3MvST
k0U2tjY1gKNTJrTUl3M291cHlxVThwclVFSlE2U2xqU0JCQmFZYUVFMytQRWx4OEx3TDg4Rk4rdDY5Wnk1N1FyeEp4cWtmbgo1bDRSWkF6WktsT
FVicDVMY1lBYVptTjJzWlVYL0llaVB2OTFJSXpFMFFVS0NRM0Mvd0lEQVFBQm8xTXdVVEFkCkJnTlZIUTRFRmdRVWtzcktTVEJkQlVsM0N0Z3Vv
MnB4TUEvejlQZ3dId1lEVlIwakJCZ3dGb0FVa3NyS1NUQmQKQlVsM0N0Z3VvMnB4TUEvejlQZ3dEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2t
xaGtpRzl3MEJBUXNGQUFPQwpBUUVBV1JCSFJFYzdOUVcyb0FERk5xeUtla2VSUlAzcEtsNGd1L3k3czVTRFhGSVZZM0pKZnVuc2tyMEpQeHBuCn
h2QnpXd3lQL3lZbml4VzQ1NVFHK0ZkWTRHWm5aOWMzV25oSWtJRTJYbnJ6dlQ2L0l3anpWK2N3M1BDSGs1RnMKWmQvL2sxU3B5L2R3dDZ1VTBDV
jlLdytRM1J3ZjhLRmVHTlBKNGM0b0JtOTBpRmQ4cFdMUWNhT0NsbTlzV1dDZApTd3BvRmJZRXk0bjM5V1BMcWJZbU9ZRlZUQ2dUUEpFVFk1aFA2
eDMvNFpjN1BCL2RGQUNQWEUzMDRmYVJjKzZpCmlRMjZzcDNkM1JRSjlzTnNFOGpKQjdHOURSUWM4VXJKekc5Y0cxUW9FRFBXNkNrcnJJK1NLMEt
GWDF0VFhXS3AKVGw4UmhxUG5SbjlUb1ZhMlFPa0FBbCtKMmc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://192.168.50.110:16443
name: microk8s-cluster
contexts:
- context:
cluster: microk8s-cluster
user: admin
name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
user:
token: cjJtaUZwaU1rMUVtQW9lVjZhTEZFd1dlcFV1bFR3Q0MySzNUNERqQlg3QT0K
Now I can simply export my KUBECONFIG environment variable and kubectl should work as normal.
I can see that I have nodes and pods.
root@ubuntu:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu Ready <none> 9m53s v1.23.4-2+98fc2022f3ad3e
root@ubuntu:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-7cm7m 1/1 Running 0 10m
kube-system calico-kube-controllers-758bff5fc6-lw582 1/1 Running 0 10m
kube-system metrics-server-679c5f986d-28kln 1/1 Running 0 5m44s
kube-system coredns-64c6478b6c-7rp6g 1/1 Running 0 4m12s
kube-system kubernetes-dashboard-585bdb5648-mj78b 1/1 Running 0 4m12s
kube-system dashboard-metrics-scraper-69d9497b54-mfbrb 1/1 Running 0 4m12s
kube-system hostpath-provisioner-7764447d7c-qdxr7 1/1 Running 0 4m12s
container-registry registry-5f697bb7df-pk2kr 1/1 Running 0 4m12s
The pods include the dashboard, coredns and a registry - all of the addons I installed earlier.
Multi Node Cluster
Adding additional nodes is really easy. It’s a similar process to other micro kubernetes distributions in that you need to have a key to add the second node.
On the first node you installed run the following command:
root@ubuntu:~# microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.50.110:25000/3728d3a71c3a81c8da4ab4783cfee2ff/3524ecb2590c
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.50.110:25000/3728d3a71c3a81c8da4ab4783cfee2ff/3524ecb2590c --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.50.110:25000/3728d3a71c3a81c8da4ab4783cfee2ff/3524ecb2590c
This command outputs the exact command that you need to use on the worker node to join to the first node you installed. This is quite nice and handy and means that you don’t actually need to mess around with files and so on.
On the new worker node simply run the appropriate command. In my case the second node will be a worker only.
root@worker:~# snap install microk8s --classic
microk8s (1.23/stable) v1.23.4 from Canonical✓ installed
root@worker:~# microk8s join 192.168.50.110:25000/3728d3a71c3a81c8da4ab4783cfee2ff/3524ecb2590c --worker
Contacting cluster at 192.168.50.110
The node has joined the cluster and will appear in the nodes list in a few seconds.
Currently this worker node is configured with the following kubernetes API server endpoints:
- 192.168.50.110 and port 16443, this is the cluster node contacted during the join operation.
If the above endpoints are incorrect, incomplete or if the API servers are behind a loadbalancer please update
/var/snap/microk8s/current/args/traefik/provider.yaml
Once this completes you should have an api server and a single worker node.
We can validate this by checking the nodes from the API server
root@ubuntu:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
worker Ready <none> 60s v1.23.4-2+98fc2022f3ad3e
ubuntu Ready <none> 20m v1.23.4-2+98fc2022f3ad3e
Deployment
Again, I’ll use my star wars API as an example deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: svk-swapi-api
labels:
app: svk-swapi
spec:
replicas: 2
selector:
matchLabels:
app: svk-swapi
template:
metadata:
labels:
app: svk-swapi
spec:
containers:
- image: public.ecr.aws/y6q2t0j9/demos:swapi-api
imagePullPolicy: IfNotPresent
name: svk-swapi
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: swapi-svc
labels:
app: svk-swapi
spec:
type: ClusterIP
selector:
app: svk-swapi
ports:
- name: port
port: 3000
targetPort: 3000
I deploy this and check the deployment and service:
root@ubuntu:~# kubectl apply -f swapi-1.yaml
deployment.apps/svk-swapi-api created
service/swapi-svc created
root@ubuntu:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
svk-swapi-api-74dc79dcc4-5k9q9 1/1 Running 0 103s
svk-swapi-api-74dc79dcc4-r6tc6 1/1 Running 0 103s
I can see that there is a service running as well and that this corresponds to the service in my manifest.
root@ubuntu:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 34m
swapi-svc ClusterIP 10.152.183.131 <none> 3000/TCP 2m18s
I can test the service and should get the familiar first result from my API
root@ubuntu:~# curl 10.152.183.131:3000/people/1
{
"edited": "2014-12-20T21:17:56.891Z",
"name": "Luke Skywalker",
"created": "2014-12-09T13:50:51.644Z",
"gender": "male",
"skin_color": "fair",
"hair_color": "blond",
"height": "172",
"eye_color": "blue",
"mass": "77",
"homeworld": 1,
"birth_year": "19BBY",
"image": "luke_skywalker.jpg",
"id": 1,
"vehicles": [
14,
30
],
"starships": [
12,
22
],
"films": [
1,
2,
3,
6
]
}
Storage
There is a default storage class provisioned by default in microk8s. This is the hostpath storage class.
root@ubuntu:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
microk8s-hostpath (default) microk8s.io/hostpath Delete Immediate false 36m
OpenEBS
Using microk8s' inbuilt add-ons, it is possible to enable other storage provisioners. One that is included as an add-on is openEBS.
You will need to ensure that iscsid is installed on all nodes within the cluster before enabling openebs.
sudo systemctl enable iscsid
Now you can run the command to enable openebs
microk8s enable openebs
This will give you the following output:
root@ubuntu:~# microk8s enable openebs
Addon dns is already enabled.
Enabling Helm 3
Fetching helm version v3.5.0.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.7M 100 11.7M 0 0 3272k 0 0:00:03 0:00:03 --:--:-- 3271k
Helm 3 is enabled
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/3021/credentials/client.config
"openebs" has been added to your repositories
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/3021/credentials/client.config
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "openebs" chart repository
Update Complete. ⎈Happy Helming!⎈
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/3021/credentials/client.config
NAME: openebs
LAST DEPLOYED: Sun Mar 20 03:13:51 2022
NAMESPACE: openebs
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Successfully installed OpenEBS.
Check the status by running: kubectl get pods -n openebs
The default values will install NDM and enable OpenEBS hostpath and device
storage engines along with their default StorageClasses. Use `kubectl get sc`
to see the list of installed OpenEBS StorageClasses.
**Note**: If you are upgrading from the older helm chart that was using cStor
and Jiva (non-csi) volumes, you will have to run the following command to include
the older provisioners:
helm upgrade openebs openebs/openebs \
--namespace openebs \
--set legacy.enabled=true \
--reuse-values
For other engines, you will need to perform a few more additional steps to
enable the engine, configure the engines (e.g. creating pools) and create
StorageClasses.
For example, cStor can be enabled using commands like:
helm upgrade openebs openebs/openebs \
--namespace openebs \
--set cstor.enabled=true \
--reuse-values
For more information,
- view the online documentation at https://openebs.io/ or
- connect with an active community on Kubernetes slack #openebs channel.
OpenEBS is installed
-----------------------
When using OpenEBS with a single node MicroK8s, it is recommended to use the openebs-hostpath StorageClass
An example of creating a PersistentVolumeClaim utilizing the openebs-hostpath StorageClass
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: local-hostpath-pvc
spec:
storageClassName: openebs-hostpath
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5G
-----------------------
If you are planning to use OpenEBS with multi nodes, you can use the openebs-jiva-csi-default StorageClass.
An example of creating a PersistentVolumeClaim utilizing the openebs-jiva-csi-default StorageClass
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: jiva-volume-claim
spec:
storageClassName: openebs-jiva-csi-default
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5G
I really like the way that this distribution gives helpful configuration hints when you install add-ons.
If I check storage classes now, I can see the following:
root@ubuntu:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
microk8s-hostpath (default) microk8s.io/hostpath Delete Immediate false 34m
openebs-device openebs.io/local Delete WaitForFirstConsumer false 2m50s
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 2m50s
openebs-jiva-csi-default jiva.csi.openebs.io Delete Immediate true 2m50s
CNI
The default CNI in microk8s is calico. This is tried and tested, and just wokrs. Using the add-on architecture, there are other network providers that can be installed and swapped out easily.
Included are: multus as a CNI, and cillium as a fuill SDN - depending on what your needs are.
Again, each of these can be turned on or off by the microk8s enable or microk8s disable command.
Ingress
This is where things get fun. There are two ingress controllers provided by microk8s out of the box. NGINX and Traefik.
NGINX Ingress
To install the NGINX ingress (I still bleed green a little) you can use the following steps:
root@ubuntu:~# microk8s enable ingress
Enabling Ingress
ingressclass.networking.k8s.io/public created
namespace/ingress created
serviceaccount/nginx-ingress-microk8s-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
configmap/nginx-load-balancer-microk8s-conf created
configmap/nginx-ingress-tcp-microk8s-conf created
configmap/nginx-ingress-udp-microk8s-conf created
daemonset.apps/nginx-ingress-microk8s-controller created
Ingress is enabled
This gives you the default ingress controller of NGINX.
First I can validate that the installation worked in the following way:
root@ubuntu:~# kubectl get pods -n ingress
NAME READY STATUS RESTARTS AGE
nginx-ingress-microk8s-controller-sbjtn 1/1 Running 0 76s
nginx-ingress-microk8s-controller-9m6zw 1/1 Running 0 76s
root@ubuntu:~#
A namespace named “ingress” is created.
If I redeploy my manifest, and add in an ingress portion up the top, you can see that it defines a standard NGINX style ingress with no fancy annotations.
The only difference to my previous deployment is the addition of ingress, and the change of the port on the container and ingress to port 80.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: swapi-ingress
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: swapi-svc
port:
number: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: svk-swapi-api
labels:
app: svk-swapi
spec:
replicas: 2
selector:
matchLabels:
app: svk-swapi
template:
metadata:
labels:
app: svk-swapi
spec:
containers:
- image: public.ecr.aws/y6q2t0j9/demos:swapi-api
imagePullPolicy: IfNotPresent
name: svk-swapi
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: swapi-svc
labels:
app: svk-swapi
spec:
selector:
app: svk-swapi
ports:
- name: port
port: 80
targetPort: 3000
Let’s apply this:
root@ubuntu:~# kubectl apply -f swapi-3.yaml
ingress.networking.k8s.io/swapi-ingress created
deployment.apps/svk-swapi-api unchanged
service/swapi-svc configured
root@ubuntu:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 54m
swapi-svc ClusterIP 10.152.183.131 <none> 80/TCP 22m
root@ubuntu:~# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
swapi-ingress public * 80 11s
We can see that the Ingress is created on port 80 as is the service.
I can connect to the ingress on the hosts public IP address of my worker node like this:
root@ubuntu:~# curl worker:80/people/1
{
"edited": "2014-12-20T21:17:56.891Z",
"name": "Luke Skywalker",
"created": "2014-12-09T13:50:51.644Z",
"gender": "male",
"skin_color": "fair",
"hair_color": "blond",
"height": "172",
"eye_color": "blue",
"mass": "77",
"homeworld": 1,
"birth_year": "19BBY",
"image": "luke_skywalker.jpg",
"id": 1,
"vehicles": [
14,
30
],
"starships": [
12,
22
],
"films": [
1,
2,
3,
6
]
}
NGINX Ingress works out of the box!
I can just as easily disable NGINX Ingress like this:
root@ubuntu:~# microk8s disable ingress
Disabling Ingress
ingressclass.networking.k8s.io "public" deleted
namespace "ingress" deleted
serviceaccount "nginx-ingress-microk8s-serviceaccount" deleted
clusterrole.rbac.authorization.k8s.io "nginx-ingress-microk8s-clusterrole" deleted
role.rbac.authorization.k8s.io "nginx-ingress-microk8s-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-microk8s" deleted
rolebinding.rbac.authorization.k8s.io "nginx-ingress-microk8s" deleted
configmap "nginx-load-balancer-microk8s-conf" deleted
configmap "nginx-ingress-tcp-microk8s-conf" deleted
configmap "nginx-ingress-udp-microk8s-conf" deleted
daemonset.apps "nginx-ingress-microk8s-controller" deleted
Ingress is disabled
Traefik Ingress
microk8s also comes with Traefik for Ingress. This is super cool to have two ingress types that I can switch on and off and just use.
I enable it like this:
root@ubuntu:~# microk8s enable traefik
Enabling traefik ingress controller on port 8080
serviceaccount/traefik-ingress-controller created
daemonset.apps/traefik-ingress-controller created
service/traefik-ingress-service created
clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created
service/traefik-web-ui created
ingress.networking.k8s.io/traefik-web-ui created
traefik ingress controller has been installed on port 8080
Again, I can re-deploy my star wars API with an additional annotation to denote that it is a traefik ingress.
Note that the ingress definition has an additional annotation that tells the ingress that it is a class of traefik. Other than the annotation it is the same as the NGINX ingress above.
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: swapi-ingress
annotations:
kubernetes.io/ingress.class: "traefik"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: swapi-svc
port:
number: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: svk-swapi-api
labels:
app: svk-swapi
spec:
replicas: 2
selector:
matchLabels:
app: svk-swapi
template:
metadata:
labels:
app: svk-swapi
spec:
containers:
- image: public.ecr.aws/y6q2t0j9/demos:swapi-api
imagePullPolicy: IfNotPresent
name: svk-swapi
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: swapi-svc
labels:
app: svk-swapi
spec:
selector:
app: svk-swapi
ports:
- name: port
port: 80
targetPort: 3000
Let’s deploy this ingress and see what happens.
root@ubuntu:~# kubectl apply -f swapi-2.yaml
ingress.networking.k8s.io/swapi-ingress created
deployment.apps/svk-swapi-api unchanged
service/swapi-svc configured
root@ubuntu:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 63m
swapi-svc ClusterIP 10.152.183.91 <none> 80/TCP 5m4s
root@ubuntu:~# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
swapi-ingress <none> * 127.0.0.1 80 13s
Similarly, I can test the ingress again, and should see the same person as in all of my other examples.
root@ubuntu:~# curl worker:8080/people/1
{
"edited": "2014-12-20T21:17:56.891Z",
"name": "Luke Skywalker",
"created": "2014-12-09T13:50:51.644Z",
"gender": "male",
"skin_color": "fair",
"hair_color": "blond",
"height": "172",
"eye_color": "blue",
"mass": "77",
"homeworld": 1,
"birth_year": "19BBY",
"image": "luke_skywalker.jpg",
"id": 1,
"vehicles": [
14,
30
],
"starships": [
12,
22
],
"films": [
1,
2,
3,
6
]
}
The observant among you may note that traefik runs on a different port, port 8080, and that I had to use port 8080 in my curl command to get to my service.
If I describe the traefik pods, I can see that they have been started using port 8080.
root@ubuntu:~# kubectl describe pod traefik-ingress-controller-fgql2 -n traefik
Name: traefik-ingress-controller-fgql2
Namespace: traefik
Priority: 0
Node: ubuntu/192.168.50.110
Start Time: Sun, 20 Mar 2022 03:36:18 +0000
Labels: controller-revision-hash=597444df48
k8s-app=traefik-ingress-lb
name=traefik-ingress-lb
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 192.168.50.110
IPs:
IP: 192.168.50.110
Controlled By: DaemonSet/traefik-ingress-controller
Containers:
traefik-ingress-lb:
Container ID: containerd://193816ebac3af45f3bd568d551221f954a9008b2df400c291d39d29a76454e54
Image: traefik:2.5
Image ID: docker.io/library/traefik@sha256:7d5a6ae66572b27afe1059ce893b59acb28de70f4ce298385e153d0a398ce7f8
Port: 8080/TCP
Host Port: 8080/TCP
Args:
--providers.kubernetesingress=true
--providers.kubernetesingress.ingressendpoint.ip=127.0.0.1
--log=true
--log.level=INFO
--accesslog=true
--accesslog.filepath=/dev/stdout
--accesslog.format=json
--entrypoints.web.address=:8080
--entrypoints.websecure.address=:8443
State: Running
Started: Sun, 20 Mar 2022 03:36:33 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8w4bl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-8w4bl:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: :NoSchedule op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
There are two arguments that have been passed to the containers at startup that we can see:
--entrypoints.web.address=:8080
--entrypoints.websecure.address=:8443
This is quite smart because it means that you can run both traefik and NGINX as ingress controllers at the same time if you want. Doing this by default is smart and makes life easy.
Conclusion
microk8s is a good solid distribution that works well. I really like the fact that it has sensible defaults and is easy to install.
The installation again is different to k0s and k3s but is easy enough.
One minor gripe is that I had to install ubuntu to get everything to work seamlessly. My usual distribution is Fedora, and on Fedora 35 I hit a problem with a few pods not starting correctly. I didn’t bother to troubleshoot it and moved to ubuntu to see the difference. If you’re an ubuntu user it probably wont bother you at all.
I would say that the add-on architecture is solid and a neat way of making the distribution user friendly. Using snap is also something lowers the barrier of entry - although again, additional steps are needed on Fedora 35.
The add-on architecture and using HELM to install addons is super slick and very nice. The model that microk8s uses removes the need to manually handle helm templates and install helm yourself. I cannot overstate how good this is.
The addition of sensible defaults is also quite nice. Multiple Ingress controllers that can both be run side by side by default without further intervention is very nice.
All in all, this is a good distribution for getting up and running quickly, and a super good distribution if you want to switch components out easily to test different scenarios.
…and I didn’t even touch on the fact that Istio is bundled as an add-on as well :)