Faris Mušić

Introduction Kubernetes is not just a container orchestration platform. It is a platform that can be extended with custom APIs. In this blog, the following topics will be covered: what the Kubernetes API is and how it can be extended using a Custom Resource Definition (CRD for short), what a Kubernetes operator is and how it can be implemented, a practical example of extending the Kubernetes API using a CRD, a practical example of implementing an operator that manages objects of the new CRD. As part of the hands-on example, a Website CRD and an operator will be implemented. Based on Website objects, the operator will automatically create a Deployment, Service, and Ingress. Kubernetes API Kubernetes is an API-driven platform, meaning all resources in the cluster are managed through the Kubernetes API server. Creating Kubernetes resources (e.g., Deployment, Service, Ingress, etc.) is done by sending a request to the Kubernetes API server, which: validates the object stores it in etcd and makes it available to other Kubernetes components It is important to understand that the API server does not execute changes directly. It only stores the desired state, while background controllers reconcile and try to align the system's current state with it. This model makes Kubernetes a declarative system so the user defines what they want, and Kubernetes tries to achieve it. How CRDs Extend the Kubernetes API Although Kubernetes comes with a large number of predefined resources, there is often a need to create custom types of objects, like: Database KafkaCluster BackupPolicy Website A CRD allows you to create new types of resources without modifying the Kubernetes source code. After creating a CRD, the new resource becomes part of the Kubernetes API and can be used through: kubectl YAML manifests API calls just like standard Kubernetes resources. CRDs therefore, turn Kubernetes into an extensible platform capable of managing resources from any domain. It is important to understand the difference between a CRD, which defines only the schema of a new Kubernetes resource, and a Custom Resource (CR for short), which represents an instance (an object) of that CRD. Kubernetes Operator and reconciliation loop A Kubernetes operator is a controller that, by observing a CRD and its instances, implements defined logic and automatically brings the system into the desired state. Its main responsibility is the reconciliation loop, which continuously compares the desired state defined in the custom resource with the actual state of the system. If there is a difference between the desired and actual state, the operator performs the necessary actions. In the hands-on example, it will be shown how the operator creates the required objects based on a Website resource, and it is going to be: Deployment Service Ingress In addition to being event-driven, meaning it reacts to events related to the resources it watches, an operator is also state-based, meaning that every time it reacts, it tries to bring the system into the desired state. Lifecycle from CRD to Managed Resource The lifecycle of an operator can be described through the following steps: CRD registration The Kubernetes API is extended with a new resource type. Creating a custom resource The user defines the desired state through a YAML manifest. Validation and storage The API server validates the resource based on the CRD schema and stores it in etcd. Change detection The operator receives an event through the watch mechanism. Reconciliation loop The operator compares the desired state with the current state of the system. Creating or updating resources Required resources are created or updated, and they can be inside or outside the cluster. Status update The operator updates the observed state in the CR status section. This process runs continuously throughout the entire lifecycle of the resource. OwnerReferences and Finalizers Operators not only manage resource creation, but also their lifecycle. OwnerReferences OwnerReferences define the relationship between the parent CR (e.g., Website) and child (Deployment, Service, Ingress …) resources. If an operator creates a Deployment, Service, or Ingress and links them to a Website resource, the Kubernetes garbage collector will automatically delete the child resources when the parent is deleted (cascading deletion). However, if a child resource is deleted first, the operator will recreate it during the reconciliation loop, as it always aims to restore the systemto the desired state defined in the Website resource. How it looks in the child resource definition: ownerReferences: - apiVersion: website.demo.atlantbh.com/v1 blockOwnerDeletion: true controller: true kind: Website name: abh-demo-website uid: 9be6fcb1-cf19-40ce-bfed-8f7c13c23057 Finalizers Finalizers allow additional cleanup logic before a resource is permanently deleted. When a resource contains a finalizer: Kubernetes sets the deletionTimestamp the resource remains in the cluster the operator performs cleanup logic Typical examples include: deleting cloud resources DNS cleanup backing up data before deletion Only after the cleanup is complete does the operator remove the finalizer, and Kubernetes permanently deletes the resource. A finalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished.nalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished. Implementation Using Kubebuilder For implementing a Kubernetes operator, one framework to consider  is Kubebuilder, which is built on top of the controller-runtime library. Kubebuilder provides: generation of CRD and RBAC manifest files controller skeleton controller-runtime integration standardized project structure for operators This keeps the focus on business logic rather than boilerplate code.  The complete source code of the hands-on example of the blog is available in the GitHub repository. Hands-on Example: Website CRD A simple example demonstrates the implementation of a Kubernetes operator and CRD. The complete CRD is available at the following location.Based on the previously defined CRD, the user defines a custom resource through a YAML file: apiVersion: website.demo.atlantbh.com/v1kind: Websitemetadata: name: abh-demo-websitespec: image: nginx:latest replicas: 3 port: 80 env: - name: ENV value: "production" resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" What Does the Operator Do? After this custom resource is created, the operator runs a reconcile loop: creates a Deployment (image, replicas, env, resources) creates a Service (exposes the port) creates an Ingress (exposing the application) Then, through the reconciliation loop, it ensures: the number of pods matches replicas the resources are available routing is functioning correctly CRD Validation CRDs use OpenAPI v3 schema validation. In this example: image is a required field - repository replicas must be between 0 and 100 - repository port must be between 1 and 65535 - repository In this way, invalid resources are rejected by the Kubernetes API before they ever reach the operator. Conclusion Custom Resource Definitions enable extending the Kubernetes API and creating custom resource types within a cluster.  When combined with the operator pattern, Kubernetes becomes a platform for automating complex systems, not just container orchestration. Kubebuilder significantly simplifies operator development, allowing developers to focus on business logic rather than infrastructure boilerplate code. This example demonstrates how a simple Website resource can become a declarative interface for automatically managing a web application.

Introduction Kubernetes scheduling is the process of placing pods on Kubernetes nodes, considering available resources and any defined rules (if they exist) for assigning pods to nodes. Kubernetes offers various techniques for scheduling pods onto nodes. The goal of these techniques is to maximize the efficiency of Kubernetes cluster usage, optimize the utilization of available resources, and ensure that workloads are highly available. This blog will cover the following techniques of Kubernetes scheduling: Taints and tolerations Node selector Affinity and anti affinity Taints and tolerations Taints and tolerations work together to prevent pods from being scheduled on inappropriate nodes. The difference between taints and tolerations is that taints are applied to nodes, while tolerations are applied to pods. Taints are used to repel inappropriate pods from a node, whereas tolerations define which taints a pod will tolerate, meaning on which nodes the pod can be scheduled despite the presence of a taint. How to taint a node: kubectl taint nodes <node> key=value:NoSchedule|PreferNoSchedule|NoExecute How to untaint a node: kubectl taint nodes <node> key=value:NoSchedule-|PreferNoSchedule-|NoExecute- It is important to understand the meaning of NoSchedule, PreferNoSchedule and NoExecute. NoSchedule will not allow pods to be scheduled on the node if the pod does not tolerate that kind of taint. PreferNoSchedule represents a soft version of NoSchedule where the scheduler will try to avoid scheduling a pod onto a node if it does not tolerate that taint but it is not guaranteed. NoExecute jumps into action in the context of pods already running on the node. If the node gets NoExecution taint, pods that do not tolerate that kind of taint will be evicted from the node immediately. If pods tolerate that taint, they will keep running always on that node, except if the tolerationSeconds attribute is specified. In that case, the pod will be evicted from the node after a defined number of seconds exceeds. How to add toleration to the pod: tolerations: - key: "key" operator: "Equal|Exists" value: "value" effect: "NoSchedule|PreferNoSchedule|NoExecute" tolerationSeconds: <number of seconds> (if effect NoExecute is specified) It is important to emphasize that Kubernetes adds some taints automatically depending on the node state. There are some of those taints: node.kubernetes.io/not-ready: Node is not ready. node.kubernetes.io/unreachable: Node is unreachable from the node controller. node.kubernetes.io/memory-pressure: Node has memory pressure. node.kubernetes.io/disk-pressure: Node has disk pressure. node.kubernetes.io/pid-pressure: Node has PID pressure. node.kubernetes.io/network-unavailable: Node's network is unavailable. node.kubernetes.io/unschedulable: Node is unschedulable. Examples There are some examples where it is useful to use taints and tolerations, like when we have dedicated nodes for core applications, gpu applications, and other kinds of applications. It is useful to use taints and tolerations if we have nodes with special hardware, and it is required to run some pods on nodes with that kind of hardware. The image below illustrates how the scheduler determines if a pod should be scheduled on the node. There is a node with taint that should repel all pods except ones which tolerate the core application taint. On the other side, there are two pods representing the core application and the application that we developed. Pod A does not tolerate the node’s taint, meaning it is not an appropriate application for the node, so pod A is not going to be scheduled onto the node. Pod B is a core application, meaning the application which node is designed for, so pod B is going to be scheduled successfully onto the node. In the image below, it is illustrated how NoExecute affects pods which are already running on the node. Pod A tolerates taint on the node A and pod is scheduled successfully on the node (step 1). Step 2 shows tainting node with NoExecute effect, and in that case pod A will be evicted from the node because it does not tolerate new taint. In case that there is a tolerationSeconds attribute defined in the tolerations block, pod A will be running for the seconds specified and after that will be evicted.   When is the NoExecute taint important? Imagine that there is a DaemonSet in the cluster with replicas across all of the nodes and we want to have those replicas running during the whole node lifecycle, even if the node encounters an issue (eg, unreachable, not-ready). In this case, the DaemonSet replicas need tolerations for such taints to ensure they are not evicted from the node when these conditions occur. Administrators of Kubernetes clusters do not have to worry about this because the node controller automatically adds such taints onto nodes and the DaemonSet controller automatically adds NoSchedule and NoExecute tolerations to the DaemonSet. Node Selector Node selector is one of the easiest ways to define where we want to schedule pods. This approach is taking node labels into consideration. Node selector is needed to be defined in pod configuration, where we say on which node the pod should be scheduled based on node labels. How to label Kubernetes node: kubectl label nodes <node-name> <label-key>=<label-value> (eg. kubectl label nodes core-node-1 region=europe) How to set node selector: nodeSelector: region: europe If there are multiple node selectors, a node must have all of the specified labels for the pod to be successfully scheduled on it. The image below shows the labels of nodes and pods. Pod A will not be scheduled because the node does not match all the labels defined in its node selector. In contrast, Pod B has all the required labels, so it will be scheduled on the node. Affinity and anti affinity In contrast to taints which are used to repel pods from the inappropriate nodes, affinity is used to attract pods to nodes. This approach offers us the possibility to define detailed rules for scheduling pods onto nodes, specifying how strict the rules should be, and enabling scheduling based on pod labels, in addition to node labels. Node affinity There are two types of node affinity: requiredDuringSchedulingIgnoredDuringExecution - represents hard rule. If a rule is not satisfied, the scheduler will not schedule a pod onto a node. preferredDuringSchedulingIgnoredDuringExecution - represents a soft rule. The scheduler will try to schedule a pod onto a node considering rules, but if the rule is not satisfied, the scheduler will schedule a pod onto the inappropriate node. In the following example, there is a deployment with specified node affinity rules. Rule that must be met is that the application must be scheduled onto a gpu node (node with instance=gpu label). The second rule, that should not be strictly met, is that the application should be scheduled onto a node placed in Europe (node with region=europe label). apiVersion: apps/v1 kind: Deployment metadata: name: gpu-application namespace: demo spec: replicas: 2 selector: matchLabels: app: gpu-application template: metadata: labels: app: gpu-application spec: containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ["tail", "-f", "/dev/null"] affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: instance operator: In values: - gpu preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: region operator: In values: - europe Operator attribute can take one of the following values: In, NotIn, Exists, DoesNotExists, Gt, and Lt. PreferredDuringSchedulingIgnoredDuringExecution supports the weight parameter, giving more importance to the rule while evaluating rules. Rules with the highest score will be prioritized when the scheduler makes a scheduling decision for the Pod. Inter-pod affinity and anti affinity Imagine that we want to schedule a pod to the node where it is already running a specific pod, or in contrast, that we do not want to schedule pods on the same nodes. This is possible by using inter-pod affinity and anti affinity, which takes into consideration pod labels already running on some nodes. There are two types of inter-pod affinity and anti affinity: requiredDuringSchedulingIgnoredDuringExecution preferredDuringSchedulingIgnoredDuringExecution Where this approach could be useful? It might be that we want to schedule pods, which communicate frequently, on the same node. Or we want to spread pods across all nodes in the cluster running in different data centers, regions, and so on. This kind of scheduling will be shown in a basic example. apiVersion: apps/v1 kind: Deployment metadata: name: user-api namespace: demo spec: replicas: 2 selector: matchLabels: app: api template: metadata: labels: app: api db: mysql spec: tolerations: - key: "application" operator: "Equal" value: "true" effect: "NoSchedule" nodeSelector: instance: application containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ["tail", "-f", "/dev/null"] affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: db operator: In values: - mysql topologyKey: availabilityZone The example above shows us how to force a user-api application to be scheduled onto a node in the same region where the database application is already running (database pod with the db=mysql label). An important parameter here is topologyKey, which defines at what level the affinity or anti-affinity should be applied in the infrastructure. Simply put, it tells Kubernetes how to group or separate pods based on a specific characteristic (topology). The value of topologyKey is actually the labels on the nodes. Let’s look at the following example: apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: demo spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: tolerations: - key: "core" operator: "Equal" value: "true" effect: "NoSchedule" nodeSelector: instance: core containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ["tail", "-f", "/dev/null"] affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx topologyKey: region The example above shows how to force the nginx application (in case there are multiple replicas) to be scheduled onto nodes in different regions to ensure that if one region fails, nginx will still be running in another region. This allows us to have as many Nginx replicas as we have regions. It is important to note that for the pod anti-affinity rule requiredDuringSchedulingIgnoredDuringExecution, the admission controller LimitPodHardAntiAffinityTopology limits the topologyKey to kubernetes.io/hostname. You can modify or disable the admission controller if you want to allow custom topologies. What if we want to have 6 Nginx replicas, but only 3 regions are available? In that case, we can use the soft version preferredDuringSchedulingIgnoredDuringExecution so that the scheduler can schedule more Nginx pods in one region. An example solution is shown below. apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: demo spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: tolerations: - key: "core" operator: "Equal" value: "true" effect: "NoSchedule" nodeSelector: instance: core containers: - name: busybox image: busybox imagePullPolicy: IfNotPresent command: ["tail", "-f", "/dev/null"] affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - nginx topologyKey: region Operator parameter could have one of the following values: In, NotIn, Exists, and DoesNotExist. Node autoscaling In Kubernetes, all scheduling approaches provide a powerful way to control pod scheduling on specific node types. By tainting nodes, you ensure that only workloads with matching tolerations can be scheduled on them. This is particularly useful for specialized workloads such as GPU-based machine learning, high-memory applications, or spot instance workloads. Imagine that there are some pods that need to be scheduled but there are no available nodes for them. So in that case, we need to provision additional appropriate nodes for those pods. But what if we can make it automatically with a node autoscaler? When combining all mentioned scheduling approaches with a node autoscaler, Kubernetes can dynamically provision the correct node types based on pending pod requests. If a pod remains in a Pending state due to missing node capacity, the autoscaler detects the need and scales up an appropriate node pool. Also, what if we have more nodes than we need? Actually, we pay more for resources we do not really use. Node autoscaler could help us here with downscaling of unneeded resources. With this pattern, you optimize resource usage, prevent unnecessary costs, and ensure that critical workloads always get the required infrastructure. This approach makes Kubernetes smarter and more efficient in managing workloads across heterogeneous node types. Conclusion Kubernetes scheduling provides powerful mechanisms to control where pods run, ensuring optimal resource usage, performance, and availability. Taints and tolerations allow nodes to repel certain pods unless they explicitly tolerate them, enforcing constraints at a broad level. Node selectors offer a simple way to schedule pods based on key-value labels, while node affinity and anti-affinity provide more flexible rules, allowing preferred or required placement based on labels. Inter-pod affinity and anti-affinity extend this logic to influence pod co-location or separation based on application needs. These features are crucial for achieving efficient, fault-tolerant deployments, offering fine-grained control over pod placement while ensuring high availability and workload isolation. "Kubernetes scheduling" Tech Bite was brought to you by Faris Mušić, DevOps Engineer at Atlantbh. (more…)