Simplifying Data Lake Metadata Management with Apache Iceberg REST Catalog

Ognjen Lubarda

Data Engineering

•

March 30, 2026

Introduction

If you have ever run multiple query engines (such as Spark and Trino) on the same data lake, you have probably encountered situations where those query engines do not see the same schemas in production. Apache Iceberg’s REST Catalog solves this metadata management problem by providing a centralized HTTP API for managing table metadata, ensuring that all engines have a consistent view of partitions, schemas, and snapshots.

This blog will help data engineers learn:

– How Iceberg manages metadata through its layered architecture

– The role of catalogs as the single source of truth

– Why REST Catalog is preferred for multi-engine environments

– How to set up and test the REST Catalog locally using Docker

What is Iceberg?

To help you understand how Apache Iceberg organizes large amounts of data, I will first briefly explain to you what Apache Iceberg is and how it works.

Apache Iceberg is an open-source table format that provides ACID transactions, time travel, and schema evolution for data lakes. It provides a metadata management layer that enables data processing tools (such as Spark, Presto, Hive) to efficiently access data in storage (such as Hadoop or S3). Separating processing tools from storage creates flexibility, allowing any processing tool to access any storage.

Apache Iceberg Architecture Overview

Figure 1. Apache Iceberg Architecture Overview Figure 1. Apache Iceberg Architecture Overview (source)

Figure 1 shows the architecture of the Apache Iceberg format, which consists of two layers:

Data layer – this layer contains the physical data, partitioned into multiple files. The data is usually stored in formats like Parquet, Avro, or ORC.

Metadata layer – this layer holds the metadata that describes the table and is divided into three types of files:

A snapshot captures the state of your table at a specific point in time. Every time you modify data through write commands such as INSERT, DELETE, UPDATE, and MERGE, a new metadata file is created. Schema-only changes may or may not result in the creation of a new metadata file. The metadata file contains a list of previous snapshots and creates a new one for the current state of the table. Each snapshot points to a manifest list file. These snapshots give you safety. If something goes wrong, you can always roll back to any previous state of your data.

Apache Iceberg Catalog

The third layer in Apache Iceberg’s architecture is the Iceberg Catalog. Simply put, the Iceberg Catalog is a concept that manages a pointer to the most recent version of the table’s metadata. It’s important to remember that catalogs don’t contain any actual data – they only maintain these references.

Types of Iceberg Catalogs

Based on how they store and access references to metadata, Iceberg catalogs can be divided into two types:

File-based catalogs:

These catalogs work directly with the file system. All metadata is stored right at the location, and the user or system accessing it must know the exact path to the metadata. These catalogs are easier to set up, but it’s harder to share them between teams. An example of a catalog using a file-based approach is Hadoop.

Service-based catalogs:

These catalogs rely on an external service (like Hive Metastore, REST API, Glue, Nessie, etc.) to centrally manage metadata. They are designed for scale and collaboration, and you only need to know a table name. The trade-off with these catalogs is that they are more challenging to configure initially. Examples of catalogs that use services are:

HiveCatalog
RESTCatalog
GlueCatalog
NessieCatalog

Picking the right service catalog:

If you are unsure which catalog fits your needs and you expect an evolving infrastructure, choose the REST Catalog. It is one of the most flexible catalog implementations in the Iceberg ecosystem.

Iceberg REST Catalog Architecture Overview

Figure 2. Rest Catalog Flow Figure 2. Rest Catalog Flow

Figure 2 shows a diagram of a typical REST catalog architecture in the Iceberg ecosystem:

Client
The client is a tool that uses Apache Iceberg to manage tables. This can be any system; examples include Spark, Flink, Trino, Presto, and others.
The client sends REST requests to the REST catalog to retrieve table metadata.

REST Catalog
This is the first point of contact between the client and the Iceberg metadata. It is a client-side library that implements the Iceberg REST API specification and translates Iceberg API calls into HTTP requests. Those requests are then forwarded to the REST Server for processing.

REST Server
The REST server is a backend application that implements the entire business logic of the Iceberg REST API. It performs all metadata operations, such as viewing and retrieving metadata.

Database
The database stores metadata about tables (for example, PostgreSQL, MySQL, or another database). The REST server uses this database to store and read data about tables and snapshots. The advantage of the REST server is that it can use any other catalog in the background to process requests (for example, HiveCatalog). The client still accesses metadata through REST API calls.

Why You Should Use Iceberg REST Catalog

REST Catalog gives one HTTP endpoint, which all tools use to communicate; they are all using the same API calls to get metadata. This solves some of the problems:

- Consistency: There are no more situations where something works in Spark but not in Trino due to metadata mismatches.
- Backend that is easy to swap: You can start with a simple backend solution and then migrate to a more complex one without touching client configuration. You were using PostgreSQL, but all of a sudden you have to migrate to something else? Do it without touching client configs.
- Simple scaling: It is HTTP, so to scale it, all you need is a load balancer, run a few instances, and that’s it.
- Centralised authentication: REST catalog simplifies security by introducing a single access control point. It supports OAuth2, API keys, and other modern authentication methods.
- Observability: Log every metadata request, track who is accessing what

Why You Should Not Use Iceberg REST Catalog

REST Catalog has some limitations as well that you should consider before implementing:

- Maintenance and Complexity: You have introduced an additional backend service into your system that needs to be deployed, monitored, secured, and maintained.
- Network dependency: If your REST server is not available, the whole system is unavailable.
- Network overhead: Every HTTP request adds some latency. Complex queries with a lot of tables and partitions can add significant cumulative latency.
- Scalability bottleneck: If you have not planned your infrastructure well, the REST Catalog can easily become a bottleneck.
- Implementation limits: Some implementations can introduce their own limitations, such as a lack of multi-region or custom-region bucket support, or support only a single-level namespace.

Local Setup Walkthrough

To see the REST Catalog in action, we will set up a local environment using Docker.

What you need:

MinIO: An S3-compatible storage for keeping data.
Iceberg REST Catalog: A RESTful service that manages metadata and uses MinIO as the storage.
Apache Spark: For testing and interacting with the REST catalog through the spark-shell.

Full setup with code: https://github.com/ATLANTBH/rest-catalog-setup/

The repository includes the complete Docker and Spark configuration, detailed step-by-step instructions, and additional demo queries. Just clone it and run docker-compose up.

Demo queries

Once Docker Compose is running and you have created a MinIO bucket, you are ready to run the Spark shell and start querying. Here are a few queries to try:

Verifying the REST Catalog:

spark.sql("SELECT current_catalog()").show()

Expected output: irc

Create a Test Table:

 spark.sql("CREATE NAMESPACE IF NOT EXISTS demo")

spark.sql(""" 
    CREATE TABLE IF NOT EXISTS demo.users (
       id INT,
       name STRING,
       email STRING,
       age INT,
       created_at TIMESTAMP
     )
     USING iceberg
     PARTITIONED BY (days(created_at))
   """)

Insert data and query it:

spark.sql("""
    INSERT INTO demo.users VALUES
      (1, 'Alice Johnson', '[email protected]', 30, TIMESTAMP '2024-01-15 10:00:00'),
      (2, 'Bob Smith', '[email protected]', 25, TIMESTAMP '2024-01-16 11:30:00'),
      (3, 'Charlie Brown', '[email protected]', 35, TIMESTAMP '2024-01-17 09:15:00')
  """)

spark.sql("SELECT * FROM demo.users ORDER BY id").show()

You should be able to see the data inserted in the table. Also, if you check the MinIO dashboard, you will see your namespace, table, data, and all metadata files.

Production Considerations

While our Docker setup demonstrates the core architecture, production deployments require additional consideration:

Storage: Replace MinIO with production-grade object storage such as Azure Blob Storage or AWS S3.
High Availability: Deploy multiple REST Catalog instances behind a load balancer and configure some health check endpoints.
Security: Plan your security architecture across three layers: authentication, authorization, and network security (TLS encryption, private subnets, etc.).
Observability: Implement metrics visualisation (such as Prometheus + Grafana) and an alerting system.
Performance: Enable caching at multiple layers to reduce latency.

Conclusion

As we have seen through this blog, REST Catalog is a complex but powerful solution. If you are a small team or in a single-engine environment, the REST Catalog is not for you; it would introduce more problems than benefits.

However, for heterogeneous environments, the REST Catalog is one of the cleanest solutions available. As more tools are introduced into the system, this architecture becomes increasingly practical.

Ready to Achieve More?

We’ll help you reach your goals quickly with an easy and straightforward process to kick off our collaboration. Here’s what happens next.

STEP 1

Discovery Call

Let’s chat to understand your company, project needs, and answer any questions along the way.

STEP 2

Free Consultation

Work closely with our experts to explore the right solutions for your business.

STEP 3

Collaboration Proposal

We'll recommend the best strategy for your goals, ensuring you get the most from our expertise.

STEP 4

30-Day Cancellation
Policy Contract

Spoiler: It’s Never Been Used

Enjoy peace of mind while we deliver excellence from day one—our track record speaks for itself.

Full Name

Message

Services you're interested in (Optional)

Read more about similar topics

July 29, 2026

Software Development

Evaluating LLMs Using DeepEval

Introduction At the time of writing this blog, it seems the most popular topics in software engineering are Artificial Intelligence (AI) and Large Language Models (LLMs). LLMs are being proactively integrated into production systems for various purposes and goals, leading to the critical question: “How can we be certain the model performs well?” Traditional software is deterministic and can be tested with clear pass/fail decisions. On the other hand, LLMs are notorious for being non-deterministic, which is embedded in their very nature. Without getting into details on the underlying architecture or how they perceive the world, LLMs are known for their hallucinations, drifting away from their targeted task, being too agreeable, etc. These types of problems force us to adopt a different approach when it comes to grading the quality of our LLM-dependent systems. We shift our perspective to evaluation, i.e., scoring the output by a predefined scale, usually [0,1], instead of the traditional binary pass/fail. Different frameworks try to provide the best solution, and the one that stands out amongst many is DeepEval. What is DeepEval? DeepEval is an open-source Python evaluation framework for LLMs. With the main goal being to unit test LLMs, DeepEval operates similarly to Pytest, along with the additional caveats attributed to LLMs. DeepEval supports end-to-end as well as component-level evaluation. DeepEval’s approach to evaluation relies on the following components: Test Case - a single interaction between the user and the LLM. This is a container that includes data for a specific evaluation and consists of: input - the prompt sent to the LLM actual_output - the recorded LLM response expected_output - the gold standard response for comparison (if available) Metrics - definition of the scoring rule used to evaluate the Test Case. They may be: Deterministic - use fixed algorithms and formulae (ex., exact matches, regex, ROUGE/BLEU, LaBSE...) Non-deterministic (LLM-as-a-Judge) - use an independent powerful model to “read” and assess the output. Evaluation Dataset - collection of Test Cases grouped together Individual Test Case Evaluation Ideal evaluation workflow using DeepEval Basic evaluation through an example The problem We can imagine working on an LLM-based assistant that answers questions from retrieved documents. How do we verify whether the response is faithful to the source and is semantically correct? Setup pip install deepeval sentence-transformersexport OPENAI_API_KEY="your-api-key-here" The OpenAI key is needed for the LLM-as-a-judge metric, while deterministic metrics run without external calls. The test case from deepeval.test_case import LLMTestCasetest_case = LLMTestCase( input="What is Varnish?", actual_output="Varnish is an HTTP reverse proxy that caches responses in RAM.", expected_output="Varnish is a caching HTTP reverse proxy that accelerates web applications.", retrieval_context=["Varnish is a HTTP reverse proxy and a caching server that ","speeds up web applications by caching HTTP responses." ]) Two metrics will be used for this test case. One is an LLM-as-a-Judge metric, and the other is a deterministic metric that will utilize LaBSE. The sentence-transformers package is used for LaBSE. The retrieval_context is text fetched from a knowledge base that the LLM uses as source material when generating its answer. The retrieval_context is needed when we want to test the model against specific issues. (hallucinations, etc.) Language-agnostic BERT Sentence Embedding (LaBSE) is a multilingual sentence embedding model. It is used to compute the cosine similarity between the actual and expected outputs to evaluate semantic similarity. Metric 1: Faithfulness (LLM-as-a-Judge) LLM-as-a-Judge metrics use a secondary independent LLM to evaluate and score the primary model’s output. DeepEval has different built-in options: from deepeval.metrics import ( AnswerRelevancyMetric, # Does the output answer the question? FaithfulnessMetric, # Is the output grounded in the retrieval context? ContextualRelevancyMetric,# Is the retrieved context relevant to the question? HallucinationMetric, # Does the output contain fabricated claims not in the context? GEval, # Custom LLM-judged metric using criteria you define. The criteria is defined via a separate prompt.) For our use case, we will use the Faithfulness metric to check if the actual LLM output is grounded in the retrieval context. from deepeval.metrics import FaithfulnessMetricfaithfulness = FaithfulnessMetric(threshold=0.8) Because the evaluation itself is performed by an LLM, scores may slightly vary between runs. Metric 2: LaBSE semantic similarity (deterministic) from deepeval.metrics import BaseMetricfrom deepeval.test_case import LLMTestCasefrom sentence_transformers import SentenceTransformer, utilclass LaBSESimilarityMetric(BaseMetric): """Deterministic semantic similarity using LaBSE embeddings.""" def __init__(self, threshold: float = 0.7): self.threshold = threshold self.score = 0 self.success = False self.model = SentenceTransformer("sentence-transformers/LaBSE") async def a_measure(self, test_case: LLMTestCase, *args, **kwargs) -> float: return self.measure(test_case) def measure(self, test_case: LLMTestCase) -> float: embeddings = self.model.encode( [test_case.actual_output, test_case.expected_output], convert_to_tensor=True, ) self.score = util.cos_sim(embeddings[0], embeddings[1]).item() self.success = self.score >= self.threshold return self.score def is_successful(self) -> bool: return self.success @property def __name__(self): return "LaBSE Semantic Similarity" The class extends BaseMetric from DeepEval and implements: measure — encodes both actual_output and expected_output into embeddings, computes their cosine similarity, and tests it against the threshold. a_measure — async wrapper required by DeepEval's interface is_successful — returns whether the measurement passed the threshold. For the same inputs, LaBSE always produces the same embeddings, which follow with the same cosine similarity score. As a result, this metric is fully deterministic and reproducible. Running both metrics on one test case from deepeval import assert_testdef test_varnish_response(): assert_test(test_case, [ FaithfulnessMetric(threshold=0.8), # non-deterministic LaBSESimilarityMetric(threshold=0.7), # deterministic ]) Assuming the file containing the mentioned code is named test_varnish.py, the tests can be invoked through the following command: deepeval test run test_varnish.py Running the command produces the output: The test passes only if both metrics meet their thresholds. The faithfulness check ensures the output doesn't hallucinate beyond the context, while the LaBSE similarity check ensures it remains semantically close to the expected answer. Conclusion LLM-as-a-Judge can be used for free-form text where it is hard (or sometimes impossible) to define the correctness of an answer. (summaries, explanations, faithfulness, etc.) Deterministic metrics should be used for tasks with clearly defined correct outputs and where reproducible scores are needed. Another advantage of deterministic metrics is the avoidance of LLM-judge-related costs. In practice, the most effective evaluation suites combine both approaches. DeepEval provides a bridge between the familiar world of Pytest and the modern requirements of LLM testing. By combining LLM-judged metrics for subjective quality assessment with deterministic metrics for measurable correctness, it provides a complete evaluation toolkit that can be easily run from the CLI or a CI pipeline. References: Feng, F., Yang, Y., Cer, D., Arivazhagan, N., & Wang, W. (2020, July 3). Language-agnostic BERT sentence embedding. arXiv.org. https://arxiv.org/abs/2007.01852 DeepEval by Confident AI. (2026, April 5). Quick Introduction. https://deepeval.com/docs/getting-started LLM-as-a-Judge Metrics | Confident AI Docs. (n.d.). Confident AI Docs. https://www.confident-ai.com/docs/llm-evaluation/core-concepts/llm-as-a-judge Ip, J. (2025, October 10). LLM-as-a-Judge simply explained: The complete guide to run LLM evals at scale. Confident AI. https://www.confident-ai.com/blog/why-llm-as-a-judge-is-the-best-llm-evaluation-method DeepEval by Confident AI. (2026a, April 5). “Do it yourself” Metrics. https://deepeval.com/docs/metrics-custom Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023, June 9). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv.org. https://arxiv.org/abs/2306.05685

June 8, 2026

Building Kubernetes Operator – Essential Components and Practical Guide

DevOps

Software Development

Building Kubernetes Operator – Essential Components and Practical Guide

Introduction Kubernetes is not just a container orchestration platform. It is a platform that can be extended with custom APIs. In this blog, the following topics will be covered: what the Kubernetes API is and how it can be extended using a Custom Resource Definition (CRD for short), what a Kubernetes operator is and how it can be implemented, a practical example of extending the Kubernetes API using a CRD, a practical example of implementing an operator that manages objects of the new CRD. As part of the hands-on example, a Website CRD and an operator will be implemented. Based on Website objects, the operator will automatically create a Deployment, Service, and Ingress. Kubernetes API Kubernetes is an API-driven platform, meaning all resources in the cluster are managed through the Kubernetes API server. Creating Kubernetes resources (e.g., Deployment, Service, Ingress, etc.) is done by sending a request to the Kubernetes API server, which: validates the object stores it in etcd and makes it available to other Kubernetes components It is important to understand that the API server does not execute changes directly. It only stores the desired state, while background controllers reconcile and try to align the system's current state with it. This model makes Kubernetes a declarative system so the user defines what they want, and Kubernetes tries to achieve it. How CRDs Extend the Kubernetes API Although Kubernetes comes with a large number of predefined resources, there is often a need to create custom types of objects, like: Database KafkaCluster BackupPolicy Website A CRD allows you to create new types of resources without modifying the Kubernetes source code. After creating a CRD, the new resource becomes part of the Kubernetes API and can be used through: kubectl YAML manifests API calls just like standard Kubernetes resources. CRDs therefore, turn Kubernetes into an extensible platform capable of managing resources from any domain. It is important to understand the difference between a CRD, which defines only the schema of a new Kubernetes resource, and a Custom Resource (CR for short), which represents an instance (an object) of that CRD. Kubernetes Operator and reconciliation loop A Kubernetes operator is a controller that, by observing a CRD and its instances, implements defined logic and automatically brings the system into the desired state. Its main responsibility is the reconciliation loop, which continuously compares the desired state defined in the custom resource with the actual state of the system. If there is a difference between the desired and actual state, the operator performs the necessary actions. In the hands-on example, it will be shown how the operator creates the required objects based on a Website resource, and it is going to be: Deployment Service Ingress In addition to being event-driven, meaning it reacts to events related to the resources it watches, an operator is also state-based, meaning that every time it reacts, it tries to bring the system into the desired state. Lifecycle from CRD to Managed Resource The lifecycle of an operator can be described through the following steps: CRD registration The Kubernetes API is extended with a new resource type. Creating a custom resource The user defines the desired state through a YAML manifest. Validation and storage The API server validates the resource based on the CRD schema and stores it in etcd. Change detection The operator receives an event through the watch mechanism. Reconciliation loop The operator compares the desired state with the current state of the system. Creating or updating resources Required resources are created or updated, and they can be inside or outside the cluster. Status update The operator updates the observed state in the CR status section. This process runs continuously throughout the entire lifecycle of the resource. OwnerReferences and Finalizers Operators not only manage resource creation, but also their lifecycle. OwnerReferences OwnerReferences define the relationship between the parent CR (e.g., Website) and child (Deployment, Service, Ingress …) resources. If an operator creates a Deployment, Service, or Ingress and links them to a Website resource, the Kubernetes garbage collector will automatically delete the child resources when the parent is deleted (cascading deletion). However, if a child resource is deleted first, the operator will recreate it during the reconciliation loop, as it always aims to restore the systemto the desired state defined in the Website resource. How it looks in the child resource definition: ownerReferences: - apiVersion: website.demo.atlantbh.com/v1 blockOwnerDeletion: true controller: true kind: Website name: abh-demo-website uid: 9be6fcb1-cf19-40ce-bfed-8f7c13c23057 Finalizers Finalizers allow additional cleanup logic before a resource is permanently deleted. When a resource contains a finalizer: Kubernetes sets the deletionTimestamp the resource remains in the cluster the operator performs cleanup logic Typical examples include: deleting cloud resources DNS cleanup backing up data before deletion Only after the cleanup is complete does the operator remove the finalizer, and Kubernetes permanently deletes the resource. A finalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished.nalizer can be seen as a blocker during deletion, meaning the resource cannot be deleted until all finalizers are removed, i.e., until the entire cleanup process is finished. Implementation Using Kubebuilder For implementing a Kubernetes operator, one framework to consider is Kubebuilder, which is built on top of the controller-runtime library. Kubebuilder provides: generation of CRD and RBAC manifest files controller skeleton controller-runtime integration standardized project structure for operators This keeps the focus on business logic rather than boilerplate code. The complete source code of the hands-on example of the blog is available in the GitHub repository. Hands-on Example: Website CRD A simple example demonstrates the implementation of a Kubernetes operator and CRD. The complete CRD is available at the following location.Based on the previously defined CRD, the user defines a custom resource through a YAML file: apiVersion: website.demo.atlantbh.com/v1kind: Websitemetadata: name: abh-demo-websitespec: image: nginx:latest replicas: 3 port: 80 env: - name: ENV value: "production" resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" What Does the Operator Do? After this custom resource is created, the operator runs a reconcile loop: creates a Deployment (image, replicas, env, resources) creates a Service (exposes the port) creates an Ingress (exposing the application) Then, through the reconciliation loop, it ensures: the number of pods matches replicas the resources are available routing is functioning correctly CRD Validation CRDs use OpenAPI v3 schema validation. In this example: image is a required field - repository replicas must be between 0 and 100 - repository port must be between 1 and 65535 - repository In this way, invalid resources are rejected by the Kubernetes API before they ever reach the operator. Conclusion Custom Resource Definitions enable extending the Kubernetes API and creating custom resource types within a cluster. When combined with the operator pattern, Kubernetes becomes a platform for automating complex systems, not just container orchestration. Kubebuilder significantly simplifies operator development, allowing developers to focus on business logic rather than infrastructure boilerplate code. This example demonstrates how a simple Website resource can become a declarative interface for automatically managing a web application.

June 8, 2026

Software Development

Streamlit Behind the Scenes: Session and Cache Explained

What is Streamlit? Working with tons of data can easily get overwhelming. Have you ever wanted to explore and visualize it quickly, without all the frontend setup? Streamlit has you covered. Streamlit is an open-source framework that enables engineers to create dynamic data web applications using Python without any frontend setup. There is no need to use HTML, CSS, JavaScript, or any complex web framework. Python knowledge is everything you need. To get started with Streamlit, simply follow the official installation guide. The Rerun Model - How Streamlit Actually Executes Coding using Streamlit is easy once you understand how it actually works. Streamlit applications are Python scripts that run from top to bottom and have a specific flow and architecture. Like any Python script, Streamlit apps execute statements sequentially from top to bottom. Every user interaction with the component triggers the entire script to be rerun. The need for this stems from Streamlit's simplicity: write apps the same way plain Python scripts are written. When coding with Streamlit, you don’t need to think in terms of callbacks and events, because each user interaction triggers a full rerun of the script to update the UI. Sessions - Preserving State Across Reruns A session is a single instance of the state of a running Streamlit application. Application state consists of the state of every UI component and is stored in-memory on the server as st.session_state. Every user session has its own st.session_state. For example, if we open our application in two tabs, each tab will have its own session. Since the script is rerun on each interaction, it is necessary to save the components' previous state to render them accurately on the next run. Streamlit components have an optional parameter key that, when set, points to the session_state key that holds the component's state, as shown in the example below. import streamlit as stst.radio( label="Choose a movie genre", options=[ "Comedy", "Drama", "Romance", ], key="radio_movie_genre")if st.session_state.radio_movie_genre == "Comedy":st.text("You have chosen Comedy")elif st.session_state.radio_movie_genre == "Drama": st.text("You have chosen Drama")elif st.session_state.radio_movie_genre == "Romance":st.text("You have chosen Romance") The important thing to remember is that values in session state are only available in the current active session. Caching - Avoiding Redundant Work The fact that the app is rerun frequently can cause significant time and memory overhead when loading data from the database or fetching it via an API call. On every rerun, the data would be fetched again. To prevent this, Streamlit provides a built-in caching mechanism. Cached values can be accessed by all users/tabs of the app, unlike a session, which can be accessed only by a single user/tab. @st.cache_data Since the cached values are used by multiple sessions, modifying the data can cause a race condition. To ensure the original data state remains unchanged, Streamlit returns a copy of the cached data instead of the original. To cache the return value of the function that fetches the data, append the decorator @st.cache_data above its declaration. @st.cache_datadef fetch_the_data(param1:str, param2:str) -> List: # API call or DB access return data To determine whether to call the function or read the cached value, Streamlit checks the two conditions: If the number, variable types, and values of the input parameters are the same If the code in the function is the same But how does Streamlit actually check whether the input parameter values are the same? This is where hashing comes into play. The input values are hashed and stored so that, on subsequent calls, they can be compared with new values. On the first run, the function is called; on subsequent runs, if the conditions are met, the returned value is read from the cache, and a copy is created. By setting the ttl (time-to-live) of the cache decorator, the time during which the cache exists is determined. After that time, the cached value is no longer available, and the function will be executed again on a call. #The cached data will be stored for 3600s ( 1 hour )@st.cache_data(ttl=3600) def fetch_the_data(param1:str, param2:str) -> List: # API call or DB access return data @st.cache_resource For caching global resources such as ML models or database connections, which are typically expensive to create, Streamlit uses the @st.cache_resource decorator. Unlike @st.cache_data, the @st.cache_resource decorator does not create and return a copy of the cached value; instead, it returns the same shared instance (singleton) of the resource every time. That makes it a perfect fit for caching resources that are not meant to be duplicated. This helps avoid unnecessary memory usage, especially for large objects like ML models or database connections. However, because a cached resource is shared across all users, sessions, and reruns, it raises thread-safety concerns. This means that multiple users or threads can access the resource simultaneously and read or write to it concurrently. Because of this, it is important to ensure that cached resources are thread-safe. A thread-safe resource behaves correctly when multiple threads access and use it simultaneously, without producing inconsistent results. In general, cached resources should be treated as read-only or designed to handle concurrent access. The example of usage is shown below: import streamlit as stimport psycopg2import pandas as [email protected]_resourcedef get_db_connection(): conn = psycopg2.connect( host="host", database="database", user="user", password="password" ) return conn# Function that fetches the data from dbdef fetch_data(query): conn = get_db_connection() # uses cached connection df = pd.read_sql_query(query, conn) return df The illustration below, from the official Streamlit documentation, represents, in a simple way, what can be cached using these two decorators from Streamlit. Summary / Key Takeaways Creating a modern, interactive UI to visualize and manage large datasets is both fun and easy with Streamlit. The diagram below shows how Streamlit works under the hood. From user interaction that triggers the script rerun, through session state restoration, to the execution of the script, with caching as a secret weapon for improving performance and reducing memory costs. Now that you understand how Streamlit handles reruns, state, and caching, you are all set to build more powerful Streamlit apps. These concepts are what make Streamlit seem simple, yet powerful and efficient in the background. References: Streamlit documentation.

Industries We Work With

Simplifying Data Lake Metadata Management with Apache Iceberg REST Catalog

Introduction

What is Iceberg?

Apache Iceberg Architecture Overview

Apache Iceberg Catalog

Types of Iceberg Catalogs

File-based catalogs:

Service-based catalogs:

Picking the right service catalog:

Iceberg REST Catalog Architecture Overview

Why You Should Use Iceberg REST Catalog

Why You Should Not Use Iceberg REST Catalog

Local Setup Walkthrough

Demo queries

Production Considerations

Conclusion

Leave a comment Cancel reply

Ready to Achieve More?

Discovery Call

Free Consultation

Collaboration Proposal

30-Day Cancellation
Policy Contract

Thank you for reaching out to us!

Read more about similar topics

Your Privacy Matters

Customize Preferences

Industries We Work With

Introduction

What is Iceberg?

Apache Iceberg Architecture Overview

Apache Iceberg Catalog

Types of Iceberg Catalogs

File-based catalogs:

Service-based catalogs:

Picking the right service catalog:

Iceberg REST Catalog Architecture Overview

Why You Should Use Iceberg REST Catalog

Why You Should Not Use Iceberg REST Catalog

Local Setup Walkthrough

Demo queries

Production Considerations

Conclusion

Leave a comment Cancel reply

Ready to Achieve More?

Discovery Call

Free Consultation

Collaboration Proposal

30-Day CancellationPolicy Contract

Thank you for reaching out to us!

Read more about similar topics

Your Privacy Matters

Customize Preferences

30-Day Cancellation
Policy Contract