Drive Growth with High-Quality Solutions

When writing SQL queries, we write SELECT first, sprinkle in all the necessary aggregations and joins, mostly without considering what the database executes first. In reality, PostgreSQL runs SELECT almost last, and once that clicks, a bunch of “why is this slow” moments suddenly make sense. Most of those moments boil down to a couple of concepts: early filtering, early aggregation, and early projection. In Postgres, a CTE is one of the cleanest ways to achieve all of that. This blog is about PostgreSQL query performance that boils down to execution order, early filtering with CTEs, and then how all of that carries over to the code, in the case of this post, Scala. Claims in this post come with benchmarks that you can run yourself; don’t trust the post, test it yourself. The order you write vs the order it runs We write SQL naturally in the following order SELECT … FROM … WHERE … GROUP BY … HAVING … ORDER BY … LIMIT …  Postgres logically evaluates: 1. FROM / JOIN    -> Get the rows2. WHERE          -> Filter rows3. GROUP BY       -> Group rows4. HAVING         -> Filter groups5. SELECT         -> Construct output columns6. DISTINCT       -> Deduplicate7. ORDER BY       -> Sort8. LIMIT / OFFSET -> Trim This one list explains every SQL scoping rule alongside the order of execution: Why you cannot use aliases in WHERE, it being at step 2, alias from step 5 still does not exist Aggregates in WHERE, take sum() for example, it still does not exist until after GROUP BY clause, that is why we use HAVING LIMIT on huge aggregation still being slow, trim happens at the very end, all the grouping work is finalized before it Important note: this is logical order. Query planner reorders things however it likes, as long as the result is the same. That freedom is exactly what the next part is about. Early filtering, CTEs and benchmarks Remember this one rule of query performance: filter out rows as early as possible. Rows that are filtered in a scan never get joined, grouped, sorted, or shipped to your JVM. CTEs (WITH … AS) are a nice way to make that explicit.  A little bit about Postgres CTEs history: Pre v12: every CTE was an optimization fence. It got fully materialized, and outer filters could not be pushed inside. After v12: CTE referenced once, with no side effects gets inlined by default, the planner treats it like a subquery and pushes your filters in. A CTE referenced more than once is still materialized by default. You can force either behavior with NOT MATERIALIZED or MATERIALIZED Sounds interesting? Let us test it. Setup (2 minutes, ~1GB of fake data) Setup docker with the latest Postgres, latest version at the time of writing this post is 17.x: docker run –name postgres-perf -e POSTGRES_USER=root -e POSTGRES_PASSWORD=root -e POSTGRES_DB=root -p 5432:5432 -d postgres:18.4 Now you can run psql from the container itself: docker exec -it postgres-perf psql -U root -d root Run the following psql against the database to generate ~1GB of fake data: DROP TABLE IF EXISTS public.orders;DROP TABLE IF EXISTS public.customers; CREATE TABLE public.customers ( “id” BIGINT PRIMARY KEY, “name” TEXT NOT NULL, “country” TEXT NOT NULL);INSERT INTO public.customers (“id”, “name”, “country”)SELECT series, ‘customer_’ || series, (ARRAY[‘DE’, ‘FR’, ‘US’, ‘JP’, ‘BR’])[1 + (random() * 4)::int]FROM generate_series(0, 100000) AS series;CREATE TABLE public.orders ( “id” BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, “customer_id” BIGINT NOT NULL, “amount” NUMERIC(10, 2) NOT NULL, “created_at” TIMESTAMPTZ NOT NULL);INSERT INTO public.orders (“customer_id”, “amount”, “created_at”)SELECT (random() * 100000)::BIGINT, round((random() * 500)::NUMERIC, 2), now() – (random() * INTERVAL ‘730 days’)FROM generate_series(1, 10000000); ALTER TABLE public.orders ADD CONSTRAINT fk_orders_customer FOREIGN KEY (“customer_id”) REFERENCES public.customers (“id”);CREATE INDEX idx_orders_customer_created ON public.orders (“customer_id”, “created_at”); That one index that was created at the end deserves a comment, because indexing is an early filtering delivery mechanism. An index on customer_id, created_at allows Postgres to jump straight to one of the customer’s recent rows instead of scanning 10M rows. Another important thing to note, column order inside index matters, equality columns first, range columns second. Flip them, and the index gets far less useful. A WHERE clause without a supporting index still filters early logically, but physically it is a full table scan. You need both. All of the benchmark results are reported from running the queries with Postgres 17.x, on MacBook Pro M1, 16GB RAM. All plans run with parallel query disabled so the execution plans stay readable, with parallelism on, expect these queries to run faster. If you want to reproduce these exact plans, disable parallelism first: SET max_parallel_workers_per_gather = 0SET max_parallel_workers = 0;SET max_parallel_maintenance_workers = 0; Benchmark 1: Materialization fence We will run the same query twice, once letting the planner inline the CTE, once fencing it off. — inlined, default in Postgres 12+, made explicit hereEXPLAIN (ANALYZE, TIMING)WITH recent AS NOT MATERIALIZED ( SELECT o.* FROM public.orders o WHERE o.”created_at” >= now() – INTERVAL ’30 days’)SELECT r.* FROM recent rWHERE r.”customer_id” = 42;QUERY PLAN ——————————————————————Index Scan using idx_orders_customer_created on orders o –(cost=0.44..20.52 rows=4 width=30) (actual time=0.313..0.530 rows=10.00 –loops=1)|– Index Cond: ((customer_id = 42) AND (created_at >= (now() – ’30 –days’::interval)))– Index Searches: 1– Buffers: shared hit=4 read=9–Planning Time: 0.276 ms–Execution Time: 0.596 ms– fenced, old pre Postgres 12 behaviorEXPLAIN (ANALYZE, BUFFERS, TIMING)WITH recent AS MATERIALIZED ( SELECT o.* FROM public.orders o WHERE o.”created_at” >= now() – INTERVAL ’30 days’)SELECT r.* FROM recent rWHERE r.”customer_id” = 42;– QUERY PLAN—————————————————————-CTE Scan on recent r (cost=248531.40..257885.98 rows=4 width=40) (actual –time=66.902..1380.055 rows=10.00 loops=1)– Filter: (customer_id = 42)– Rows Removed by Filter: 410569– Storage: Disk Maximum Storage: 16840kB– Buffers: shared hit=16166 read=57364, temp written=2105– CTE recent– -> Seq Scan on orders o (cost=0.00..248531.40 rows=415759 width=30) –(actual time=2.165..1319.926 rows=410579.00 loops=1)– Filter: (created_at >= (now() – ’30 days’::interval))– Rows Removed by Filter: 9589421– Buffers: shared hit=16166 read=57364–Planning Time: 0.070 ms–JIT:– Functions: 4– Options: Inlining false, Optimization false, Expressions true, –Deforming true– Timing: Generation 0.148 ms (Deform 0.058 ms), Inlining 0.000 ms, –Optimization 0.145 ms, Emission 1.707 ms, Total 1.999 ms–Execution Time: 1381.469 ms Same query with the same data, ~2000x difference, purely because the fence blocked the customer ID filter from reaching the index. Benchmark 2: Materialization is a friend Fences are a tool. If a CTE is expensive and referenced multiple times, this benchmark will demonstrate the benefit that materialization gives you: — referenced twice, not materialized, computed twiceEXPLAIN (ANALYZE, TIMING)WITH totals AS NOT MATERIALIZED ( SELECT o.customer_id, sum(o.amount) AS total FROM public.orders o GROUP BY o.customer_id)SELECT count(t.*) FROM totals t JOIN totals tb ON t.total < tb.total AND t.customer_id = tb.customer_id + 1;– QUERY PLAN—————————————————————- Aggregate (actual time=89813.883..89813.899 rows=1)– -> Hash Join (actual time=44888.306..89805.494 rows=49970)– Hash Cond: ((tb.customer_id + 1) = t.customer_id)– Join Filter: (t.total < tb.total)– Rows Removed by Join Filter: 50030– -> GroupAggregate (actual time=214.027..45005.852 rows=100001)– Group Key: o.customer_id– -> Index Scan … on orders o (actual time=0.113..43270.754 rows=10000000)– -> Hash (actual time=44671.326..44671.337 rows=100001)– -> GroupAggregate (actual time=0.058..44561.166 rows=100001)– Group Key: o_1.customer_id– -> Index Scan … on orders o_1 (actual time=0.012..43081.357 rows=10000000)– Execution Time: 89816.048 ms– referenced twice, materialized, computed onceEXPLAIN (ANALYZE, TIMING)WITH totals AS MATERIALIZED ( SELECT o.customer_id, sum(o.amount) AS total FROM public.orders o GROUP BY o.customer_id)SELECT count(t.*) FROM totals t JOIN totals tb ON t.total < tb.total AND t.customer_id = tb.customer_id + 1;– QUERY PLAN—————————————————————- Aggregate (actual time=43554.152..43554.158 rows=1)– CTE totals– -> GroupAggregate (actual time=126.229..43368.654 rows=100001)– Group Key: o.customer_id– -> Index Scan … on orders o (actual time=0.121..41789.548 rows=10000000)– -> Hash Join (actual time=43506.342..43552.120 rows=49970)– Hash Cond: ((tb.customer_id + 1) = t.customer_id)– Join Filter: (t.total < tb.total)– Rows Removed by Join Filter: 50030– -> CTE Scan on totals tb (actual time=126.234..134.753 rows=100001)– -> Hash (actual time=43379.620..43379.621 rows=100001)– -> CTE Scan on totals t (actual time=0.871..43317.044 rows=100001)– Execution Time: 43557.286 ms In non materialized run you will typically see the 10M row aggregation appear twice in the plan, in materialized run it runs once and the second reference is cheap CTE scan over ~100k rows. Rules of thumb from two previous benchmarks: If a CTE is used once, leave it alone, default inlining lets filters combine If a CTE is used many times it is MATERIALIZED by default, keep explicit MATERIALIZED for a pricey single use CTE that you do not want recomputed Never guess, always profile, it is free and it does not lie. Benchmark 3: Filter first with CTE vs filter alongside aggregation Two ways to get the 30 day totals per customer: Stage work with CTE: filter first, then aggregate what is left Skip the staging process and push the condition inside aggregation itself, every row gets read and grouped — Early CTE filter, referenced only onceEXPLAIN (ANALYZE, TIMING)WITH recent AS NOT MATERIALIZED ( SELECT o.”customer_id”, o.”amount” FROM public.orders o WHERE o.”created_at” >= now() – INTERVAL ’30 days’)SELECT r.”customer_id”, sum(r.”amount”) FROM recent rGROUP BY r.”customer_id”;– QUERY PLAN—————————————————————- HashAggregate (cost=275155.97..280431.83 rows=97379 width=40) (actual — time=1798.378..1911.304 rows=98317.00 loops=1)– Group Key: o.customer_id– Planned Partitions: 4 Batches: 5 Memory Usage: 8241kB Disk Usage: — 11616kB– Buffers: shared hit=16147 read=57383, temp read=1309 written=2424– -> Seq Scan on orders o (cost=0.00..248531.40 rows=415603 width=14) — (actual time=7.015..1630.068 rows=410404.00 loops=1)– Filter: (created_at >= (now() – ’30 days’::interval))– Rows Removed by Filter: 9589596– Buffers: shared hit=16147 read=57383– Planning Time: 0.430 ms– Execution Time: 1918.483 ms– condition lives alongside aggregate, scans everythingEXPLAIN (ANALYZE, TIMING)SELECT o.”customer_id”, sum(o.”amount”) FILTER (WHERE o.”created_at” >= now() – INTERVAL ’30 days’)FROM public.orders oGROUP BY o.”customer_id”HAVING sum(o.”amount”) FILTER (WHERE o.”created_at” >= now() – INTERVAL ’30 days’) IS NOT NULL– QUERY PLAN—————————————————————- GroupAggregate (cost=0.43..724378.16 rows=98225 width=40) (actual time=45.492..44438.285 rows=98317.00 loops=1)– Group Key: customer_id– Filter: (sum(amount) FILTER (WHERE (created_at >= (now() – ’30 days’::interval))) IS NOT NULL)– Rows Removed by Filter: 1684– Buffers: shared hit=2205027 read=7833152– -> Index Scan using idx_orders_customer_created on orders o (cost=0.43..598143.18 rows=10000080 width=22) (actual time=0.192..42454.429 — rows=10000000.00 loops=1)– Index Searches: 1– Buffers: shared hit=2205027 read=7833152– Planning Time: 0.133 ms– Execution Time: 44448.656 ms Two things worth noting from the previous test: CTE in the first query is referenced only once, so Postgres 12+ inlines it. The second query is not just a made-up example. It shows up in real code whenever someone computes several conditional sums in one pass and fails to notice that no shared filter means scanning everything. Benchmark 4: Aggregate first with CTE vs join first then aggregate Early filtering has its sibling, early aggregation. If you are going to collapse 10M order rows into 100k customer totals, do it before JOIN, not after. Joining first means handling 10M rows, aggregating first means handling only 100k. — join first, then aggregate– join sits below the aggregate and processes all 10M roesEXPLAIN (ANALYZE, TIMING)SELECT c.”name”, sum(o.”amount”) AS totalFROM public.orders oJOIN public.customers c ON c.”id” = o.”customer_id”GROUP BY c.”id”, c.”name”ORDER BY total DESCLIMIT 10;– QUERY PLAN—————————————————————- Limit (cost=780135.67..780135.70 rows=10 width=54) (actual time=52038.923..52038.930 rows=10.00 loops=1)– Buffers: shared hit=2204772 read=7834420 written=1– -> Sort (cost=780135.67..780385.67 rows=100001 width=54) (actual time=51984.158..51984.159 rows=10.00 loops=1)– Sort Key: (sum(o.amount)) DESC– Sort Method: top-N heapsort Memory: 26kB– Buffers: shared hit=2204772 read=7834420 written=1– -> GroupAggregate (cost=7.21..777974.69 rows=100001 width=54) (actual time=1.726..51935.407 rows=100001.00 loops=1)– Group Key: c.id– Buffers: shared hit=2204769 read=7834420 written=1– -> Merge Join (cost=7.21..726724.27 rows=10000080 width=28) (actual time=0.074..50530.788 rows=10000000.00 loops=1)– Merge Cond: (o.customer_id = c.id)– Buffers: shared hit=2204769 read=7834420 written=1– -> Index Scan using idx_orders_customer_created on — orders o (cost=0.43..598143.18 rows=10000080 width=14) (actual time=0.029..48764.432 rows=10000000.00 loops=1)|– Index Searches: 1– Buffers: shared hit=2204767 read=7833412 written=1 — -> Index Scan using customers_pkey on customers c — (cost=0.29..3342.31 rows=100001 width=22) (actual time=0.030..37.442 rows=100001.00 loops=1)– Index Searches: 1– Buffers: shared hit=2 read=1008– Planning Time: 5.058 ms– Execution Time: 52042.793 ms– aggregate first in CTE, then join– join sits above the aggregate and only sees 100k pre-collapsed rowsEXPLAIN (ANALYZE, TIMING)WITH totals AS NOT MATERIALIZED ( SELECT o.”customer_id”, sum(o.”amount”) AS total FROM public.orders o GROUP BY o.”customer_id”)SELECT c.”name”, t.totalFROM totals tJOIN public.customers c ON c.”id” = t.”customer_id”ORDER BY t.total DESCLIMIT 10;– QUERY PLAN—————————————————————- Limit (cost=657324.33..657324.36 rows=10 width=46) (actual time=44288.081..44288.088 rows=10.00 loops=1)– Buffers: shared hit=2204819 read=7834373– -> Sort (cost=657324.33..657571.13 rows=98719 width=46) (actual time=44242.794..44242.795 rows=10.00 loops=1)– Sort Key: (sum(o.amount)) DESC– Sort Method: top-N heapsort Memory: 26kB– Buffers: shared hit=2204819 read=7834373– -> Merge Join (cost=0.73..655191.05 rows=98719 width=46) (actual time=1.644..44199.715 rows=100001.00 loops=1)– Merge Cond: (o.customer_id = c.id)– Buffers: shared hit=2204816 read=7834373– -> GroupAggregate (cost=0.43..649377.56 rows=98719 width=40) (actual time=1.569..44130.428 rows=100001.00 loops=1)– Group Key: o.customer_id– Buffers: shared hit=2204816 read=7833363– -> Index Scan using idx_orders_customer_created on — orders o (cost=0.43..598143.18 rows=10000080 width=14) (actual time=0.100..42614.593 rows=10000000.00 loops=1)|– Index Searches: 1– Buffers: shared hit=2204816 read=7833363– -> Index Scan using customers_pkey on customers c (cost=0.29..3342.31 rows=100001 width=22) (actual time=0.066..27.299 rows=100001.00 loops=1)– Index Searches: 1– Buffers: shared read=1010– Planning Time: 1.718 ms — Execution Time: 44290.105 ms In the first query, mege join sits below the aggregate and processes all 10M rows. In the second query, join sits above the aggregate and only sees 100k pre-collapsed rows. Since these are simple test tables, imagine real tables with 10s of columns, more columns, bigger second queries win, because the first query drags all those columns through JOIN and the GROUP BY for nothing. Scala with Doobie Scala on the JVM talks to Postgres over JDBC, and Doobie wraps that in something pleasant. Postgres performance rules carry over directly, with one addition: the JVM is the most expensive place to filter. Every row you pull over the network gets deserialized by the JDBC driver and becomes GC pressure before .filter throws it away. Good version, CTE, early filters, parameters, all filtering inside Postgres: import doobie._import doobie.implicits._import cats.effect.IOfinal case class BigSpender(name: String, total: BigDecimal) derives Readdef bigSpenders(minTotal: BigDecimal, days: Int): Query0[BigSpender] = sql””” WITH recent_orders AS ( SELECT o.”customer_id”, o.”amount” FROM public.orders o WHERE o.”created_at” >= now() – make_interval(days => $days) ), totals AS ( SELECT ro.”customer_id”, sum(ro.”amount”) AS total FROM recent_orders ro GROUP BY ro.”customer_id” HAVING sum(ro.”amount”) > $minTotal ) SELECT c.”name”, t.”total” FROM totals t JOIN public.customers c ON c.”id” = t.”customer_id” ORDER BY t.”total” DESC “””.query[BigSpender]def run(xa: Transactor[IO]): IO[List[BigSpender]] = bigSpenders(BigDecimal(1000), 30).to[List].transact(xa) Why is this a good shape: $days and $minTotal become JDBC PreparedStatement parameters, making it safe from SQL injection and adding plan caching for free Heavy lifting happens in Postgres, Scala receives final result and maps it into case classes CTE is single use, so Postgres 12+ inlines it and pushes filters freely, same situation as in our first benchmark. Antipattern for contrast: sql””” SELECT o.”customer_id”, o.”amount”, o.”created_at” FROM orders o “”” .query[(Long, BigDecimal, java.time.Instant)] .to(List) .map(_.filter(_._3.isAfter(cutoff))) Expect the JVM side filtering version to be noticeably slower, and to spike up heap usage quite a bit compared to Postgres heavy lifting version. Cleanup Remove Docker container docker container rm postgres-perf Clean up Docker image docker rmi postgres:18.4 Takeaways Logical order first – SQL runs logically as FROM -> WHERE -> GROUP BY -> HAVING -> SELECT -> ORDER BY -> LIMIT. SELECT executes after filtering, that is why aliases work in ORDER BY but not in WHERE. Filter early and use indexes – filter early, and give the filter an index, WHERE without index is physically a full table scan (adding the index on a large live table is a challenge on its own). Aggregate before you join – aggregate early, collapsing rows before join means joins handle significantly fewer rows. CTEs are inlined by default – in Postgres 12+, single use CTEs are inlined. MATERIALIZED is a deliberate fence: great for reuse, terrible when it blocks an index. Let Postgres do the work – use parameterized SQL queries in Scala, leave heavy lifting to Postgres, return necessary results. Never fetch and then filter Never guess, always profile, it is free and it does not lie Sources PostgreSQL: Documentation: 18: 7.8. WITH Queries (Common Table Expressions)  PostgreSQL: Documentation: 18: 11.3. Multicolumn Indexes  PostgreSQL: Documentation: 18: 14.1. Using EXPLAIN  PostgreSQL: Documentation: 18: SELECT  Parameterized Queries · doobie 

Introduction Kubernetes is not just a container orchestration platform. It is a platform that can be extended with custom APIs. In this blog, the following topics will be covered: what the Kubernetes API is and how it can be extended using a Custom Resource Definition (CRD for short), what a Kubernetes operator is and how it can be implemented, a practical example of extending the Kubernetes API using a CRD, a practical example of implementing an operator that manages objects of the new CRD. As part of the hands-on example, a Website CRD and an operator will be implemented. Based on Website objects, the operator will automatically create a Deployment, Service, and Ingress. Kubernetes API Kubernetes is an API-driven platform, meaning all resources in the cluster are managed through the Kubernetes API server. Creating Kubernetes resources (e.g., Deployment, Service, Ingress, etc.) is done by sending a request to the Kubernetes API server, which: validates the object stores it in etcd and makes it available to other Kubernetes components It is important to understand that the API server does not execute changes directly. It only stores the desired state, while background controllers reconcile and try to align the system’s current state with it. This model makes Kubernetes a declarative system so the user defines what they want, and Kubernetes tries to achieve it. How CRDs Extend the Kubernetes API Although Kubernetes comes with a large number of predefined resources, there is often a need to create custom types of objects, like: Database KafkaCluster BackupPolicy Website A CRD allows you to create new types of resources without modifying the Kubernetes source code. After creating a CRD, the new resource becomes part of the Kubernetes API and can be used through: kubectl YAML manifests API calls just like standard Kubernetes resources. CRDs therefore, turn Kubernetes into an extensible platform capable of managing resources from any domain. It is important to understand the difference between a CRD, which defines only the schema of a new Kubernetes resource, and a Custom Resource (CR for short), which represents an instance (an object) of that CRD. Kubernetes Operator and reconciliation loop A Kubernetes operator is a controller that, by observing a CRD and its instances, implements defined logic and automatically brings the system into the desired state. Its main responsibility is the reconciliation loop, which continuously compares the desired state defined in the custom resource with the actual state of the system. If there is a difference between the desired and actual state, the operator performs the necessary actions. In the hands-on example, it will be shown how the operator creates the required objects based on a Website resource, and it is going to be: Deployment Service Ingress In addition to being event-driven, meaning it reacts to events related to the resources it watches, an operator is also state-based, meaning that every time it reacts, it tries to bring the system into the desired state. Lifecycle from CRD to Managed Resource The lifecycle of an operator can be described through the following steps: CRD registration The Kubernetes API is extended with a new resource type. Creating a custom resource The user defines the desired state through a YAML manifest. Validation and storage The API server validates the resource based on the CRD schema and stores it in etcd. Change detection The operator receives an event through the watch mechanism. Reconciliation loop The operator compares the desired state with the current state of the system. Creating or updating resources Required resources are created or updated, and they can be inside or outside the cluster. Status update The operator updates the observed state in the CR status section. This process runs continuously throughout the entire lifecycle of the resource. OwnerReferences and Finalizers Operators not only manage resource creation, but also their lifecycle. OwnerReferences OwnerReferences define the relationship between the parent CR (e.g., Website) and child (Deployment, Service, Ingress …) resources. If an operator creates a Deployment, Service, or Ingress and links them to a Website resource, the Kubernetes garbage collector will automatically delete the child resources when the parent is deleted (cascading deletion). However, if a child resource is deleted first, the operator will recreate it during the reconciliation loop, as it always aims to restore the systemto the desired state defined in the Website resource. How it looks in the child resource definition: ownerReferences: – apiVersion: website.demo.atlantbh.com/v1 blockOwnerDeletion: true controller: true kind: Website name: abh-demo-website uid: 9be6fcb1-cf19-40ce-bfed-8f7c13c23057 Finalizers Finalizers allow additional cleanup logic before a resource is permanently deleted. When a resource contains a finalizer: Kubernetes sets the deletionTimestamp the resource remains in the cluster the operator performs cleanup logic Typical examples include: deleting cloud resources DNS cleanup backing up data before deletion Only after the cleanup is complete does the operator remove the finalizer, and Kubernetes permanently deletes the resource. A finalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished.nalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished. Implementation Using Kubebuilder For implementing a Kubernetes operator, one framework to consider  is Kubebuilder, which is built on top of the controller-runtime library. Kubebuilder provides: generation of CRD and RBAC manifest files controller skeleton controller-runtime integration standardized project structure for operators This keeps the focus on business logic rather than boilerplate code.  The complete source code of the hands-on example of the blog is available in the GitHub repository. Hands-on Example: Website CRD A simple example demonstrates the implementation of a Kubernetes operator and CRD. The complete CRD is available at the following location.Based on the previously defined CRD, the user defines a custom resource through a YAML file: apiVersion: website.demo.atlantbh.com/v1kind: Websitemetadata: name: abh-demo-websitespec: image: nginx:latest replicas: 3 port: 80 env: – name: ENV value: “production” resources: requests: cpu: “100m” memory: “128Mi” limits: cpu: “500m” memory: “256Mi” What Does the Operator Do? After this custom resource is created, the operator runs a reconcile loop: creates a Deployment (image, replicas, env, resources) creates a Service (exposes the port) creates an Ingress (exposing the application) Then, through the reconciliation loop, it ensures: the number of pods matches replicas the resources are available routing is functioning correctly CRD Validation CRDs use OpenAPI v3 schema validation. In this example: image is a required field – repository replicas must be between 0 and 100 – repository port must be between 1 and 65535 – repository In this way, invalid resources are rejected by the Kubernetes API before they ever reach the operator. Conclusion Custom Resource Definitions enable extending the Kubernetes API and creating custom resource types within a cluster.  When combined with the operator pattern, Kubernetes becomes a platform for automating complex systems, not just container orchestration. Kubebuilder significantly simplifies operator development, allowing developers to focus on business logic rather than infrastructure boilerplate code. This example demonstrates how a simple Website resource can become a declarative interface for automatically managing a web application.

Introduction At the time of writing this blog, it seems the most popular topics in software engineering are Artificial Intelligence (AI) and Large Language Models (LLMs). LLMs are being proactively integrated into production systems for various purposes and goals, leading to the critical question: “How can we be certain the model performs well?” Traditional software is deterministic and can be tested with clear pass/fail decisions. On the other hand, LLMs are notorious for being non-deterministic, which is embedded in their very nature. Without getting into details on the underlying architecture or how they perceive the world, LLMs are known for their hallucinations, drifting away from their targeted task, being too agreeable, etc.  These types of problems force us to adopt a different approach when it comes to grading the quality of our LLM-dependent systems. We shift our perspective to evaluation, i.e., scoring the output by a predefined scale, usually [0,1], instead of the traditional binary pass/fail. Different frameworks try to provide the best solution, and the one that stands out amongst many is DeepEval. What is DeepEval? DeepEval is an open-source Python evaluation framework for LLMs. With the main goal being to unit test LLMs, DeepEval operates similarly to Pytest, along with the additional caveats attributed to LLMs. DeepEval supports end-to-end as well as component-level evaluation.  DeepEval’s approach to evaluation relies on the following components: Test Case – a single interaction between the user and the LLM. This is a container that includes data for a specific evaluation and consists of: input – the prompt sent to the LLM actual_output – the recorded LLM response expected_output – the gold standard response for comparison (if available) Metrics – definition of the scoring rule used to evaluate the Test Case. They may be: Deterministic – use fixed algorithms and formulae (ex., exact matches, regex, ROUGE/BLEU, LaBSE…) Non-deterministic (LLM-as-a-Judge) – use an independent powerful model to “read” and assess the output. Evaluation Dataset – collection of Test Cases grouped together Individual Test Case Evaluation Ideal evaluation workflow using DeepEval Basic evaluation through an example The problem We can imagine working on an LLM-based assistant that answers questions from retrieved documents. How do we verify whether the response is faithful to the source and is semantically correct? Setup pip install deepeval sentence-transformersexport OPENAI_API_KEY=”your-api-key-here” The OpenAI key is needed for the LLM-as-a-judge metric, while deterministic metrics run without external calls.  The test case from deepeval.test_case import LLMTestCasetest_case = LLMTestCase( input=”What is Varnish?”, actual_output=”Varnish is an HTTP reverse proxy that caches responses in RAM.”, expected_output=”Varnish is a caching HTTP reverse proxy that accelerates web applications.”, retrieval_context=[“Varnish is a HTTP reverse proxy and a caching server that “,”speeds up web applications by caching HTTP responses.” ]) Two metrics will be used for this test case. One is an LLM-as-a-Judge metric, and the other is a deterministic metric that will utilize LaBSE. The sentence-transformers package is used for LaBSE.  The retrieval_context is text fetched from a knowledge base that the LLM uses as source material when generating its answer. The retrieval_context is needed when we want to test the model against specific issues. (hallucinations, etc.) Language-agnostic BERT Sentence Embedding (LaBSE) is a multilingual sentence embedding model. It is used to compute the cosine similarity between the actual and expected outputs to evaluate semantic similarity. Metric 1: Faithfulness (LLM-as-a-Judge) LLM-as-a-Judge metrics use a secondary independent LLM to evaluate and score the primary model’s output. DeepEval has different built-in options: from deepeval.metrics import ( AnswerRelevancyMetric, # Does the output answer the question? FaithfulnessMetric, # Is the output grounded in the retrieval context? ContextualRelevancyMetric,# Is the retrieved context relevant to the question? HallucinationMetric, # Does the output contain fabricated claims not in the context? GEval, # Custom LLM-judged metric using criteria you define. The criteria is defined via a separate prompt.) For our use case, we will use the Faithfulness metric to check if the actual LLM output is grounded in the retrieval context. from deepeval.metrics import FaithfulnessMetricfaithfulness = FaithfulnessMetric(threshold=0.8) Because the evaluation itself is performed by an LLM, scores may slightly vary between runs. Metric 2: LaBSE semantic similarity (deterministic) from deepeval.metrics import BaseMetricfrom deepeval.test_case import LLMTestCasefrom sentence_transformers import SentenceTransformer, utilclass LaBSESimilarityMetric(BaseMetric): “””Deterministic semantic similarity using LaBSE embeddings.””” def __init__(self, threshold: float = 0.7): self.threshold = threshold self.score = 0 self.success = False self.model = SentenceTransformer(“sentence-transformers/LaBSE”) async def a_measure(self, test_case: LLMTestCase, *args, **kwargs) -> float: return self.measure(test_case) def measure(self, test_case: LLMTestCase) -> float: embeddings = self.model.encode( [test_case.actual_output, test_case.expected_output], convert_to_tensor=True, ) self.score = util.cos_sim(embeddings[0], embeddings[1]).item() self.success = self.score >= self.threshold return self.score def is_successful(self) -> bool: return self.success @property def __name__(self): return “LaBSE Semantic Similarity” The class extends BaseMetric from DeepEval and implements: measure — encodes both actual_output and expected_output into embeddings, computes their cosine similarity, and tests it against the threshold. a_measure — async wrapper required by DeepEval’s interface is_successful — returns whether the measurement passed the threshold. For the same inputs, LaBSE always produces the same embeddings, which follow with the same cosine similarity score. As a result, this metric is fully deterministic and reproducible. Running both metrics on one test case from deepeval import assert_testdef test_varnish_response(): assert_test(test_case, [ FaithfulnessMetric(threshold=0.8), # non-deterministic LaBSESimilarityMetric(threshold=0.7), # deterministic ]) Assuming the file containing the mentioned code is named test_varnish.py, the tests can be invoked through the following command: deepeval test run test_varnish.py Running the command produces the output: The test passes only if both metrics meet their thresholds. The faithfulness check ensures the output doesn’t hallucinate beyond the context, while the LaBSE similarity check ensures it remains semantically close to the expected answer. Conclusion LLM-as-a-Judge can be used for free-form text where it is hard (or sometimes impossible) to define the correctness of an answer. (summaries, explanations, faithfulness, etc.) Deterministic metrics should be used for tasks with clearly defined correct outputs and where reproducible scores are needed. Another advantage of deterministic metrics is the avoidance of LLM-judge-related costs. In practice, the most effective evaluation suites combine both approaches. DeepEval provides a bridge between the familiar world of Pytest and the modern requirements of LLM testing. By combining LLM-judged metrics for subjective quality assessment with deterministic metrics for measurable correctness, it provides a complete evaluation toolkit that can be easily run from the CLI or a CI pipeline. References: Feng, F., Yang, Y., Cer, D., Arivazhagan, N., & Wang, W. (2020, July 3). Language-agnostic BERT sentence embedding. arXiv.org. https://arxiv.org/abs/2007.01852 DeepEval by Confident AI. (2026, April 5). Quick Introduction. https://deepeval.com/docs/getting-started LLM-as-a-Judge Metrics | Confident AI Docs. (n.d.). Confident AI Docs. https://www.confident-ai.com/docs/llm-evaluation/core-concepts/llm-as-a-judge Ip, J. (2025, October 10). LLM-as-a-Judge simply explained: The complete guide to run LLM evals at scale. Confident AI. https://www.confident-ai.com/blog/why-llm-as-a-judge-is-the-best-llm-evaluation-method DeepEval by Confident AI. (2026a, April 5). “Do it yourself” Metrics. https://deepeval.com/docs/metrics-custom Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023, June 9). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv.org. https://arxiv.org/abs/2306.05685