How HPA, VPA and Cluster Autoscaler Work Together in Kubernetes and Where They Clash

Introduction Kubernetes is not just a container orchestration platform. It is a platform that can be extended with custom APIs. In this blog, the following topics will be covered: what the Kubernetes API is and how it can be extended using a Custom Resource Definition (CRD for short), what a Kubernetes operator is and how it can be implemented, a practical example of extending the Kubernetes API using a CRD, a practical example of implementing an operator that manages objects of the new CRD. As part of the hands-on example, a Website CRD and an operator will be implemented. Based on Website objects, the operator will automatically create a Deployment, Service, and Ingress. Kubernetes API Kubernetes is an API-driven platform, meaning all resources in the cluster are managed through the Kubernetes API server. Creating Kubernetes resources (e.g., Deployment, Service, Ingress, etc.) is done by sending a request to the Kubernetes API server, which: validates the object stores it in etcd and makes it available to other Kubernetes components It is important to understand that the API server does not execute changes directly. It only stores the desired state, while background controllers reconcile and try to align the system's current state with it. This model makes Kubernetes a declarative system so the user defines what they want, and Kubernetes tries to achieve it. How CRDs Extend the Kubernetes API Although Kubernetes comes with a large number of predefined resources, there is often a need to create custom types of objects, like: Database KafkaCluster BackupPolicy Website A CRD allows you to create new types of resources without modifying the Kubernetes source code. After creating a CRD, the new resource becomes part of the Kubernetes API and can be used through: kubectl YAML manifests API calls just like standard Kubernetes resources. CRDs therefore, turn Kubernetes into an extensible platform capable of managing resources from any domain. It is important to understand the difference between a CRD, which defines only the schema of a new Kubernetes resource, and a Custom Resource (CR for short), which represents an instance (an object) of that CRD. Kubernetes Operator and reconciliation loop A Kubernetes operator is a controller that, by observing a CRD and its instances, implements defined logic and automatically brings the system into the desired state. Its main responsibility is the reconciliation loop, which continuously compares the desired state defined in the custom resource with the actual state of the system. If there is a difference between the desired and actual state, the operator performs the necessary actions. In the hands-on example, it will be shown how the operator creates the required objects based on a Website resource, and it is going to be: Deployment Service Ingress In addition to being event-driven, meaning it reacts to events related to the resources it watches, an operator is also state-based, meaning that every time it reacts, it tries to bring the system into the desired state. Lifecycle from CRD to Managed Resource The lifecycle of an operator can be described through the following steps: CRD registration The Kubernetes API is extended with a new resource type. Creating a custom resource The user defines the desired state through a YAML manifest. Validation and storage The API server validates the resource based on the CRD schema and stores it in etcd. Change detection The operator receives an event through the watch mechanism. Reconciliation loop The operator compares the desired state with the current state of the system. Creating or updating resources Required resources are created or updated, and they can be inside or outside the cluster. Status update The operator updates the observed state in the CR status section. This process runs continuously throughout the entire lifecycle of the resource. OwnerReferences and Finalizers Operators not only manage resource creation, but also their lifecycle. OwnerReferences OwnerReferences define the relationship between the parent CR (e.g., Website) and child (Deployment, Service, Ingress …) resources. If an operator creates a Deployment, Service, or Ingress and links them to a Website resource, the Kubernetes garbage collector will automatically delete the child resources when the parent is deleted (cascading deletion). However, if a child resource is deleted first, the operator will recreate it during the reconciliation loop, as it always aims to restore the systemto the desired state defined in the Website resource. How it looks in the child resource definition: ownerReferences: - apiVersion: website.demo.atlantbh.com/v1 blockOwnerDeletion: true controller: true kind: Website name: abh-demo-website uid: 9be6fcb1-cf19-40ce-bfed-8f7c13c23057 Finalizers Finalizers allow additional cleanup logic before a resource is permanently deleted. When a resource contains a finalizer: Kubernetes sets the deletionTimestamp the resource remains in the cluster the operator performs cleanup logic Typical examples include: deleting cloud resources DNS cleanup backing up data before deletion Only after the cleanup is complete does the operator remove the finalizer, and Kubernetes permanently deletes the resource. A finalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished.nalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished. Implementation Using Kubebuilder For implementing a Kubernetes operator, one framework to consider  is Kubebuilder, which is built on top of the controller-runtime library. Kubebuilder provides: generation of CRD and RBAC manifest files controller skeleton controller-runtime integration standardized project structure for operators This keeps the focus on business logic rather than boilerplate code.  The complete source code of the hands-on example of the blog is available in the GitHub repository. Hands-on Example: Website CRD A simple example demonstrates the implementation of a Kubernetes operator and CRD. The complete CRD is available at the following location.Based on the previously defined CRD, the user defines a custom resource through a YAML file: apiVersion: website.demo.atlantbh.com/v1kind: Websitemetadata: name: abh-demo-websitespec: image: nginx:latest replicas: 3 port: 80 env: - name: ENV value: "production" resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" What Does the Operator Do? After this custom resource is created, the operator runs a reconcile loop: creates a Deployment (image, replicas, env, resources) creates a Service (exposes the port) creates an Ingress (exposing the application) Then, through the reconciliation loop, it ensures: the number of pods matches replicas the resources are available routing is functioning correctly CRD Validation CRDs use OpenAPI v3 schema validation. In this example: image is a required field - repository replicas must be between 0 and 100 - repository port must be between 1 and 65535 - repository In this way, invalid resources are rejected by the Kubernetes API before they ever reach the operator. Conclusion Custom Resource Definitions enable extending the Kubernetes API and creating custom resource types within a cluster.  When combined with the operator pattern, Kubernetes becomes a platform for automating complex systems, not just container orchestration. Kubebuilder significantly simplifies operator development, allowing developers to focus on business logic rather than infrastructure boilerplate code. This example demonstrates how a simple Website resource can become a declarative interface for automatically managing a web application.

Introduction One of the most widely used tools in the application management field is Kubernetes. However, it's essential to monitor activity within your Kubernetes cluster. The Go Informers package fulfills that need. It functions as a helpful assistance to help you stay informed about the activities happening in Kubernetes clusters. Let's get started and see how it works! Understanding Kubernetes Monitoring Keeping an eye on your Kubernetes cluster includes monitoring its performance. This includes checking if your applications are running smoothly, ensuring they have enough resources and how many resources they are using, and being notified if something goes wrong. Introducing the Go Informers Package The Go Informers package is like a special toolbox for working with Kubernetes. It helps you get information from the Kubernetes API server quickly and easily. With this package, you can stay updated about what's happening in their clusters without writing a lot of complicated code. The package keeps track of changes in your cluster in real time. So you always know what's happening without having to check constantly. But it's not just about seeing what's happening now. Go Informers can also tell you when important events occur in your cluster, like when a new application is deployed or something gets deleted. How Informers Work Event-driven architecture is the foundation upon which informers function. When a Kubernetes resource undergoes a modification, such as creation, update, or deletion, the API server emits an event. Informers ensure that monitoring applications are up to date with the most recent cluster state by constantly monitoring for these events and responding appropriately. Caching Mechanism One of the key features of informers is their caching mechanism, which improves performance and reduces the load on the Kubernetes API server. The resource objects that informers have retrieved from the API server are kept in a local cache. To ensure that the client always has access to the most recent data without overloading the server with too many requests, this cache is periodically synchronized with the server to fetch any updates or changes. Indexing for Efficient Retrieval Informers employ indexing mechanisms to facilitate efficient data retrieval. They simplify the lookup process by indexing resource objects based on specific fields or labels, providing rapid and focused access to relevant data. This indexing strategy enhances the responsiveness of monitoring applications, particularly when working with large-scale Kubernetes clusters that have many resources. Anatomy of Kubernetes Informers The diagram illustrates the workflow of Kubernetes informers: Initialization: Monitoring applications initialize informers by specifying the types of resources they want to monitor (e.g., pods, services). Watch Mechanism: Informers set up watch mechanisms with the Kubernetes API server to get alerts about resource events. Event Processing: Informers process events based on predefined event handlers upon receiving events from the API server. These handlers define how the monitoring application reacts to different types of events (e.g., creation, update, deletion). Caching Strategy: Informers maintain a local cache of resource objects retrieved from the API server.  To guarantee data consistency, this cache and the server are periodically synchronized. Indexing for Efficiency: Informers use indexing techniques to optimize the efficiency of data retrieval.  By indexing resource objects based on specific criteria, informers facilitate fast and efficient access to relevant data. Client Interaction: Monitoring applications interact with informers to retrieve real-time information about the Kubernetes cluster state. This interaction allows applications to monitor and respond to changes dynamically. Access configuration There are two ways how we can access Kubernetes API, if we are accessing from inside the cluster we use InClusterConfig to access KubernetesAPI: corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" // Get the in-cluster configuration config, err := rest.InClusterConfig() // Create the Kubernetes clientset clientset, err := kubernetes.NewForConfig(config) If a monitoring app is going to be deployed in a targeted cluster then we can use  cluster config file: // Load the Kubernetes client configuration from the kubeconfig file config, err := clientcmd.BuildConfigFromFlags("", kubeconfig) // Create a Kubernetes clientset clientset, err := kubernetes.NewForConfig(config) Practical Usage Let's say you want to keep track of whenever a new pod (a basic unit of deployment in Kubernetes) is created or deleted in your cluster. Now with the clientset configured we can instantiate podInformer and monitor pods resources and lifecycle. // Stop signal for the informer stopper := make(chan struct{}) defer close(stopper) // Set resync interval to 10 minutes ResyncInterval = 10 * time.Minute // Create shared informers for resources in all known API group versions with a reSync period and namespace factory := informers.NewSharedInformerFactoryWithOptions(clientSet, ResyncInterval) podInformer := factory.Core().V1().Pods().Informer() defer k8sruntime.HandleCrash() // Start informer go factory.Start(stopper) Now we can create an event handler: podInformer.AddEventHandler(cache.ResourceHandlerException{ UpdateFunc: func(oldPodObject, newPodObject interface{}) { // Old version of the Pod Object     oldPod := oldPodObj.(*corev1.Pod)     // New version of the Pod Object     newPod := newPodObj.(*corev1.Pod) // If resource version is different we know there was a change with the pod oldPodData.ResourceVersion != newPodData.ResourceVersion { // Handle update } else { fmt.Println(“Nothing to update for pod %s.”, newPodData.Name) } },   AddFunc: func(podObj interface{}) { pod, ok := podObj.(*corev1.Pod) if ok { fmt.Printf("Added Pod: %s. Current status: %s", pod.GetName(), pod.Status.Phase) } }, DeleteFunc: func(podObj interface{}) { pod, ok := podObj.(*corev1.Pod) if ok { fmt.Printf("Deleted Pod: %s", pod.GetName()) }     },  } In this example, we're using Go Informers to track new pods created in the cluster, pods deleted in the cluster, and any changes made in any pod.  However, it's important to understand that AddFunc reacts to each pod creation in the cluster, so whenever a pod enters the pending state, we get notified. At the same time, UpdateFunc handles the event when the pod's status switches to running. Therefore, we may track the status change in UpdateFunc to mimic on-event handling. When a pod is deleted from the cluster, DeleteFunc will be called concurrently. However, there may be a delay between the pod stopping and being deleted, so we can also monitor the status change in UpdateFunc. When the pod status stops running, we can determine that the pod has entered the terminate status. UpdateFunc: func(oldPodObject, newPodObject interface{}) { // Old version of the Pod Object oldPod := oldPodObject.(*corev1.Pod) // New version of the Pod Object newPod := newPodObject.(*corev1.Pod) // If resource version is different we know there was a change with the pod oldPodData.ResourceVersion != newPodData.ResourceVersion { if oldPod.Status.Phase == corev1.PodPending && newPod.Status.Phase == corev1.PodRunning { // Handle pod creation } if oldPod.Status.Phase == corev1.PodRunning && newPod.Status.Phase != corev1.PodRunning { // Handle pod deletion } } else { fmt.Println(“Nothing to update for pod %s.”, newPodData.Name) } } Accessing resources data If you want to know more about how resources are used, you can access information about requests, limits, and allocatable resources.  Understanding these metrics can help you optimize resource utilization, improve application performance, and efficiently allocate resources within your Kubernetes cluster.  Requests are the minimal resources needed for starting a pod, limits are set inside the cluster and indicate the maximum number of resources a pod can use, and allocatable is the entire amount of resources available within the node. It is important to remember that a pod consists of several containers. Therefore, to get the total resource data for the pod, we must combine the resources of these containers. We can obtain information about these resources through informers in the following way: UpdateFunc: func(oldObj, newObj interface{}) { pod := newObj.(*corev1.Pod) fmt.Printf("Pod Name: %s\n", pod.Name) fmt.Printf("Pod Namespace: %s\n", pod.Namespace) var PodCPULimit int64 var PodCPURequest int64 var PodMemoryLimit int64 var PodMemoryRequest int64 // Access pod resource information for _, container := range append(pod.Spec.InitContainers, pod.Spec.Containers...) { PodCPULimit += container.Resources.Limits.Cpu().MilliValue() PodCPURequest += container.Resources.Requests.Cpu().MilliValue() PodMemoryLimit += container.Resources.Limits.Memory().Value() PodMemoryRequest += container.Resources.Requests.Memory().Value() } // Calculate allocatable resources based on node capacity nodeName := pod.Spec.NodeName node, err := clientset.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) if err != nil { fmt.Printf("Error getting node: %v\n", err) return } NodeAllocatableMemory := NodeCapacityCPU := node.Status.Capacity.Cpu().MilliValue() NodeCapacityMemory := node.Status.Capacity.Memory().Value() fmt.Printf("Pod CPU Limit: %d\n", PodCPULimit) fmt.Printf("Pod CPU Request: %d\n", PodCPURequest) fmt.Printf("Pod Memory Limit: %d\n", PodMemoryLimit) fmt.Printf("Pod Memory Request: %d\n", PodMemoryRequest) fmt.Printf("Node Allocatable CPU: %d\n", NodeAllocatableCPU) fmt.Printf("Node Allocatable Memory: %d\n", NodeAllocatableMemory) fmt.Printf("Node Capacity CPU: %d\n", NodeCapacityCPU) fmt.Printf("Node Capacity Memory: %d\n", NodeCapacityMemory) }, Pros and cons of using Go Informer for custom monitoring service Pros: Flexibility: With Go Informer, you have complete control over the monitoring logic and can tailor it to meet your unique needs. You can customise the monitoring service to monitor any aspect of your applications or Kubernetes cluster. Integration: Go Informer enables smooth integration with additional application or infrastructure components. Adding monitoring logic directly to your current Go-based services or applications is a straightforward process. Scalability: Go Informer can grow with your Kubernetes cluster because it is a component of the Kubernetes client library. It efficiently uses Kubernetes' scalability features, like horizontal pod autoscaling, to manage demanding monitoring jobs. Real-time monitoring: Go Informer enables real-time monitoring by reacting to Kubernetes events as they occur. This can be helpful for monitoring important events or performing immediate actions based on changes in the cluster. Cons: Development overhead: Using Go Informer to provide a personalized monitoring service calls for knowledge and work in development. Writing and maintaining code is required for handling data storage, alerting, event processing, and visualization. This may require an extensive amount of time. Complexity:  Go Informer-built custom monitoring systems may become more sophisticated as time goes on and the requirements for the monitoring evolve. Overhead and maintenance expenses might rise when overseeing a sophisticated monitoring system. Limited features: Go Informer offers the basic elements needed to monitor Kubernetes clusters. However, it can lack some of the more advanced capabilities and functionalities found in specialized monitoring programs like Prometheus or Grafana. For instance, it does not have built-in support for long-term data retention, complex alerting rules, or sophisticated visualization options. Resource utilization: Running a custom monitoring service alongside your applications may consume additional resources, such as CPU, memory, and network bandwidth, which could impact the overall performance and scalability of your cluster. Conclusion Using the Go Informers package to monitor your Kubernetes cluster is like having a dependable assistant. It simplifies the cluster monitoring process, allowing immediate reactions to any issues or changes and enabling even inexperienced developers to stay informed. With Go Informers, mastering Kubernetes monitoring becomes a breeze! Also, compared to specialized monitoring tools like Prometheus or Grafana, Go Informer lacks some advanced features and may need additional effort, despite its flexibility and real-time monitoring capabilities. Thus,to select the best monitoring solution for your Kubernetes cluster, consider your unique needs, available resources, and level of experience. "Simplifying Kubernetes Monitoring with Go Informers" Tech Bite was brought to you by Kenan Bajrić, Junior Software Engineer at Atlantbh. Tech Bites are tips, tricks, snippets or explanations about various programming technologies and paradigms, which can help engineers with their everyday job.

Every year, zettabytes of data are transferred over the Internet. Managing, processing, and storing the data can be a really complex task and requires cutting-edge tools. Many tools can be used, but two technologies that are almost essential to every modern data pipeline: Kafka and Snowflake. If you are planning to stream a large amount of data, Kafka is usually the best choice. Scalability, reliability, low latency and popularity are reasons why Kafka stands out. Kafka is by far the most popular event streaming platform and a go-to for many cases when it comes to streaming data. Since you need to process and store the data somewhere, Snowflake is one of the top contenders for that.  If you are wondering why Snowflake is so popular, look at the user satisfaction - it’s over the roof. Snowflake took most of the drawbacks and oversights that other data storage solutions had and addressed it. Probably the most important feature is having the storage and compute separately scaled.  What is Kafka Connect, and why did we decide to use it? Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. It provides streaming integration between Kafka and other data sources or sinks, enabling seamless data movement and transformation. Kafka Connect uses connectors to ingest data into Kafka topics or export data from Kafka topics to external systems. Kafka Connect is made for use cases like this. Installing Kafka Connect and choosing the correct mode If you search for Kafka Connect, you will see that there is no download link. That is because Kafka Connect is not a standalone tool; it is bundled with the main Kafka installation. If you communicated with Kafka from the command line before, you probably used one of the scripts from the bin directory like kafka-topics.sh. Two more scripts that are used for Kafka Connect are in the same directory as the kafka-topics.sh script. Those two scripts are for two different modes, standalone and distributed: connect-standalone.sh - Only runs one node of Kafka Connect and saves all the configs, offsets, and statuses locally. If the node goes down, all Kafka Connect processes are down. This is okay for testing but should not be used in production as it is neither reliable nor fault-tolerant.  connect-distributed.sh - Runs a cluster of Kafka Connect nodes. Saves all configs, offsets and statuses in specific Kafka topics designated for Kafka Connect. If one node goes down, another node will take over the workload from that node. The connect-standalone.sh script can be used for testing, but connect-distributed.sh should be used for production. Kafka Connect components There are a few Kafka Connect components, but the most important for us are: Connectors (sink and source) Converters (including transforms) Dead Letter Queue (DLQ) If you are new to Kafka Connect, you may be confused about the difference between connectors and converters.  Connectors Connectors, as the name says, connect your Kafka Connect cluster to a specific source/destination. In our case, that will be Snowflake, but it can also be HDFS, S3, Elasticsearch, and many more. There are two types of connectors: Source Connectors - Take the data from a source and send it to Kafka topic(s). In the diagram below, we can see how it would look if we wanted to take files from S3 and publish them to a Kafka topic. Sink Connectors - Take the data from Kafka topic(s) and send it to a destination. In the diagram below, we can see how it would look if we wanted to take messages from a Kafka topic and store them in Snowflake. Converters Converters are the most confusing component to set up because it's easy to misunderstand and misconfigure them. Since Kafka messages are just bytes, converters translate data between the internal data format used by Kafka Connect and the format specific to the source or sink connector. When we store messages in Kafka topics they will always get serialized, even if we do not specify the serialization format. The Kafka producer will use the String serializer by default to serialize the message and save it to a topic as an array of bytes. Usually, you will use serialization formats like Avro or Protobuf for more complex pipelines. Since we are using both connectors and converters, let's refer to them together as a Kafka Connect system to make it less confusing. We have two systems: a source system and a sink system.  You may see that in some places, those two components are both referred to as connectors, but that is why many people get confused. We will expand both previous diagrams to demonstrate how both source and sink systems look. Source system As you can see in the diagram, the source connector (in our case S3 connector) establishes the connection to S3 and starts polling the desired file. After the S3 connector gets a file, it will hand out the contents of that file to the converter. The converter will take that content and serialize it with the specified serializer to the specified format. After serialization, it will publish the serialized message to a Kafka topic. Sink system In sink systems, the converter first takes the serialized message from a Kafka topic, deserializes it, and hands it over to the sink connector. After that, the sink connector (in our case the Snowflake connector) saves the formatted message into the Snowflake table. If you are still confused, imagine that the connector connects Kafka Connect to a source/destination and acts as a bridge between those two points while the converter does all the data transformations (serialization/deserialization). Understanding converters is a crucial part of setting up Kafka Connect. I highly recommend spending some time to understand connectors and converters before attempting to set up Kafka Connect for a complex pipeline. It will save you a lot of time in the long run.  Confluent published a great blog about converters here: Kafka Connect Deep Dive – Converters and Serialization Explained | Confluent.  Dead Letter Queue (DLQ) A Dead letter queue or DLQ is a component that can be easily overlooked since it is not a required component for setting up Kafka Connect. In my opinion, DLQ is a must for any production grade data pipeline. DLQ is a component that is responsible for handling messages that failed to be consumed. In terms of Kafka, DLQ is a separate topic that is only used to catch messages that had some issues and could not be consumed. The most common issue when this happens is when using the wrong converter. If you have worked with Avro, there is a good chance you saw the “Unknown magic byte!” error. That error occurs when you try to use an Avro converter to deserialize messages that are not Avro. A wrong converter is one example, but there can be more issues. To avoid losing messages, we need a DLQ that will handle those cases. By default, the Kafka Connect cluster will stop if it cannot deserialize a message, and we will need to manually fix the issue and restart it.  In production environments we cannot allow downtime nor lose messages and that's where DLQ can help us. We can configure Kafka Connect to not crash when a message cannot be consumed or deserialized but instead send it to the DLQ topic. These are the properties that we need to define if we want to do that: "errors.tolerance": "all", "errors.deadletterqueue.topic.name": "MY_DLQ_TOPIC", "errors.deadletterqueue.topic.replication.factor": 3, "errors.deadletterqueue.context.headers.enable": true Schema registry Schema registry is an additional service that helps us improve the data quality of our Kafka data. It's a centralized repository where we can store our schemas for our Kafka messages. You can think of schemas as blueprints for how the message should look like. Schema definition contains all the fields and their types that a message needs to have. When a producer wants to publish a message to a topic, it must first validate whether the message satisfies the schema definition. Only then can the message be published. The same applies for consumers. This helps us have standards, and establishes better collaboration across different teams, and helps prevent mistakes and inconsistencies. This is an example how a schema definition looks like: {   "doc": "Sample schema",   "fields": [     {       "doc": "ID of the user. Type Int.",       "name": "id",       "type": "int"     },     {       "doc": "Users full name. Type String.",       "name": "name",       "type": "string"     }   ],   "name": "sampleRecord",   "namespace": "com.dk.mynamespace",   "type": "record" } Kafka Connect is made by Confluent and has many integrations with the Confluent Platform, and also different licenses. To reduce the complexity of the blog, in part 2 we will use the Confluent Platform schema registry.  In case you cannot use the Confluent Platform, Confluent provides the schema registry under Community License and in 99% of the cases you will be fine to host your own instance of the schema registry. The only exception is if you want to offer it as a service and compete with Confluent. In those cases, feel free to use alternatives like Karapace.  Conclusion When it comes to streaming messages from Kafka to Snowflake, Kafka Connect was a clear winner for us. It provided us with high reliability, scalability and fault tolerance while also having out of the box integration with schema registry and dead letter queues. In this blog, we learned about Kafka Connect, how to use it, and which features are necessary for a production-grade big data pipeline.  For a practical guide on how to set up Kafka Connect; stay tuned for part 2 of the blog where we configure and deploy three Kafka Connect clusters (avro, protobuf and string) on Kubernetes and demonstrate the process how we can utilize Schema Registry and DLQ (Dead Letter Queue) to establish a production grade data pipeline. The architecture we will set up is shown here: