VIDEO: Data Science Day

Introduction Snowflake's definition on their website is as follows: “A single, fully managed platform that powers the AI Data Cloud. Snowflake securely connects businesses globally across any type or scale of data to productize AI, applications, and more in the enterprise.” For a more in-depth introduction, you can read this blog by my fellow colleague. Structured and unstructured (via link fetching) data is supported in Snowflake, although there exists data that lies in neither category - hence, we introduce the semi-structured data category. We use semi-structured data when we: Do not know what read patterns to expect When we have metadata, we do not search for into but rather present it as a whole When we do not want to enforce a schema upon a user Snowflake supports two semi-structured data types: variant and array. Let’s get to know them. A VARIANT value can hold a value of any type. Ideal for storing JSON, Avro, ORC, or others since we do not need to define the hierarchical structure of the data. Due to performance implications, we should avoid using the VARIANT if our data has numbers in the strings or dates. The ARRAY data type compliments VARIANTs since it holds only these types. Snowflake’s array concept is the same as that of other programming languages, holding multiple data points in one place. This type is helpful when we have variants that have similar or identical structures. And we would process the array elements in the same way. An important thing to know when using the VARIANT or ARRAY data type is the maximum storage size of 16 MB. In the case we know the ARRAY or VARIANT will exceed the maximum size it is best to convert them into columns and connect them using a relational link. A natural question that arises is why shouldn’t we simply store our JSON as serialized strings? The main point lies in query optimization; if we are querying and/or filtering our data based on keys stored in the VARIANT type, we will have nearly identical performance as if we are querying based upon a column. Furthermore, it allows us to directly manipulate the data stored inside it. We can create a Snowflake table with an autoincrement ID, a field metadata of VARIANT data types, and an array field that describes the metadata further. CREATE OR REPLACE TABLE GENERIC ( ID NUMBER AUTOINCREMENT START 1 INCREMENT 1, METADATA VARIANT METADATA_TAGS ARRAY ) We will use Spring Boot to demonstrate how to store data in our newly created table. Suppose we receive the data through a REST client, and the object we receive can be represented as this: public record Generic( @NotBlank long id, @NotBlank String metadata, @NotBlank String metadataTags, ) { @JsonCreator public Generic( @JsonProperty("id") final long id, @JsonProperty("metadata") final String metadata, @JsonProperty("metadataTags") final String metadataTags ) { this.id = id; this.metadata = metadata; this.metadataTags = metadataTags; } } JDBI is a higher-level SQL convenience layer built on top of JDBC. It aims to abstract much of the boilerplate present in JDBC by automating connection management and automatic object mapping query results for Java Objects.  More can be seen in a Five Minute Introduction on the official site. To separate concerns, we will create a GenericEntity, which will be used by the JDBI. Furthermore, the toDomain function is handy when transforming the entity into the model.  public class GenericEntity { private final long id; private final String metadata; private final String metadataTags; @JdbiConstructor public GenericEntity( final long id, final String metadata, final String metadataTags ) { this.id = id; this.metadata = metadata; this.metadataTags = metadataTags; } public Generic toDomain() { return new Generic( id, metadata, metadataTags ); // getters omitted } } Now, we will need to create a repository that will have a JDBI repository injected into it. The JDBI repository will hold the actual Snowflake SQL statements, which will be called from the parent repository. @Component public class GenericRepository { @Autowired private JdbiGenericRepository jdbiGenericRepository; @Override public void saveGenerics(final List<Generic> generics) { final List<GenericEntity> genericEntities = generics.stream() .map(GenericEntity::new) .collect(Collectors.toList()); jdbiGenericRepository.insertGenerics(genericEntities); } } With the stream() function, we map from Generic to GenericEntity. Afterward, we call the JDBI Generic Repository to perform the actual inserts. @RegisterConstructorMapper(GenericEntity.class) public interface JdbiGenericRepository { @SqlBatch(""" INSERT INTO GENERIC ( id, METADATA, METADATA_TAGS, ) SELECT :id, parse_json(:metadata), parse_json(:metadataTags), """) void insertGenerics(@BindBean List<GenericEntity> generics); } The `parse_json` function indicates to Snowflake that the string we are passing in is of JSON format, so it knows to process like so. This concludes the basic setup of inserting semi-structured data, and the reader can follow the outline in this Tech Bite to extend the functionality.  References:  Semi-structured data types | Snowflake Documentation Considerations for Semi-structured Data Stored in VARIANT | Snowflake Documentation   "Storing semi-structured data in Snowflake" Tech Bite was brought to you by Muhamed Hamzić, Software Engineer at Atlantbh. Tech Bites are tips, tricks, snippets or explanations about various programming technologies and paradigms, which can help engineers with their everyday job.  

In order to successfully convey certain information to an audience or a user, we often come into a situation where we need to visualize the data we are presenting. Whether we are creating a presentation or designing an interface for an application, it’s important to know the basics of presenting data using charts or graphs. When choosing a chart for data visualization, it’s important to consider what kind of message you are trying to convey to your audience. Do you want to compare values of two or more data points, see how things change over time, or show the composition of certain data? With all of the different charts available to us, it can be challenging to choose the right one for the data we are presenting. In this tech bite, we will cover some of the most commonly used charts along with their variations and explain where they shine and when it’s better to avoid them. Bar Chart Probably the most popular chart to be used, and for a good reason, is the bar chart (Example 1). It is extremely efficient in presenting and comparing quantitative data - anything that can be counted, measured, or given a numerical value but also grouped or categorized. The bar chart can even be used to showcase change over time or make trend predictions, although it’s best used when the depicted changes are more significant. The bar chart consists of two axes. One axis represents categories that are being compared, while the other one represents the values of those categories. Example 1 - Bar Chart with Two Categories   The categories themselves are represented as bars with either fixed length or height depending on the chart's orientation. Bar Chart Types If the chart has a vertical orientation (Example 1), the width of the bars will be the same in all categories, but the height will vary. The opposite is true for a chart with a horizontal orientation. For a quicker understanding, bars should be sorted from the one with the greatest value down to the smallest value. In this way, we are removing additional cognitive effort required from the user to sort this information themselves. Example 2 - Horizontal Bar Chart When the category labels are too long (Example 3), it is better to switch to a horizontal chart (Example 2) rather than position the labels at an angle. Not only that the text displayed diagonally is not properly lined up with its corresponding bar, but it’s also harder to read, especially with multiple diagonal labels placed next to each other. Example 3 - Long Diagonal Labels (how not to position labels) If there is a need to showcase the value of more than one item within the category, a grouped bar chart (Example 4) can be used. The overall value of the category will be broken down into multiple bars or rectangles, which are grouped together and colored differently. The colors should stay consistent when repeating the group. Example 4 - Grouped Bar Chart In case we want to keep the overall value of the category visible but still show its constitution (what it’s made of), we can use the stacked bar chart (Example 5). This way, we have multiple rectangular shapes of the same width stacked on top of each other. These represent parts of the total value and are usually differentiated with various colors or shades. Example 5 - Stacked Bar Chart Examples of bar chart uses: Showcasing user survey results Showing profit and loss over time Making financial predictions Comparing the performance of different products in a certain time frame Line Chart A line chart (Example 6) is great for showing changes over time or trends when charting a continuous data set. It is more precise in depicting subtle shifts in values than the bar chart. Multiple different categories of data are presented as lines of different colors and shades. This allows us to see the relationship between the lines and compare different categories. Example 6 - Line Chart The line chart also has two axes. The Y axis (vertical) represents the quantity of the measured variable, and the X axis (horizontal) is typically used to label certain points in time (years, months, days, etc.). A spline chart (Example 7) is essentially the same as a line chart, but the lines between the data points are smoothed to create a more natural flow. This is merely a visual difference but offers a more realistic, close-to-life representation of gradual changes. Example 7 - Spline Chart UX Tip: Plotting more than 4 lines on a chart can make it look overly busy and create visual distractions, making the chart harder to read. When designing a chart for a web application/website, we can allow the users to toggle lines/data sets on and off. If, on the other hand, we are designing a static image of a chart, then we should narrow down our data sets to only the most important ones, keeping in mind that users struggle to memorize more than 7 items at a time. Another option is to separate the data sets into multiple smaller trend charts. Examples of line/spline chart use: Temperature changes during certain dates Showing growth in different departments of a company Tracking sales count for a specific product Measuring erosion levels Seasonal plant growth Pie/Donut Chart Pie charts (Example 8) are also some of the most widespread charts, primarily for their aesthetic appeal. A popular variation of the pie chart, commonly used in dashboard designs, is the donut chart (Example 9). It has a circular negative space in the center, which can be used to display additional numerical or textual information. They are generally used for comparing parts to a whole, but if the goal is to compare parts to each other, these charts are not the best choice. Pie or donut charts should only be used if the sum of parts equals a whole. Example 8 - Good and Bad Pie Chart Practices Example 9 - Donut Chart The human eye is not very good at accurately interpreting area or angle when trying to grasp quantitative data quickly. The shapes representing parts (pie slices) are made up of different angles and positioned differently. This makes the pie chart harder to read than other charts. However, there are still ways to make it work. Some rules to follow when designing pie charts: Don’t show more than 4 slices - showing 3 categories + the remaining categories grouped and named as: “Other” It’s best to use a pie chart when there is a huge contrast between parts to make the difference obvious. If the parts are too small, this will result in very thin slices, which cannot be properly observed. Always reinforce a pie chart with numerical information. Sort the slices from greatest to smallest (or vice versa) to remove additional cognitive effort from the user/audience. Don’t use 3D renders of the pie chart, as this will create foreshortening making the back part of the chart look smaller than the front. Don’t compare multiple pie charts. Interpreting one pie chart alone is challenging, but comparing them is even more difficult. When not to use a pie/donut chart? If a goal is to compare parts to each other, a better alternative to using a pie chart would be a stacked horizontal bar chart. The ideal way to get a clear understanding of differences in quantity is when the shapes are lined up, containing 90-degree angles, and have one dimension remaining consistent while the other is used to depict the change.  Tips for Choosing Colors In order to keep these sections noticeable, it’s important to establish sufficient brightness and color contrast between shapes. Due to the prevalence of people with color-impaired vision, subtle color variations can go unnoticed and make the chart unreadable. In the image below (Example 10), we can see how our original colors appear to people with vision imparities. A much more reliable way to achieve and retain contrast is by establishing a sufficient contrast in value (brightness/darkness of the color).  Example 10 - Color Accessibility Test (created in Figma, using the “Color Blind” plugin) Some colors, like green and red, can appear to have different values, but once we desaturate them (convert them to grayscale), we can see that their value is the same (Example 11). This is why we cannot fully rely on color to provide contrast but only use it to reinforce contrast already established by other means. Example 11 - Value Contrast Test Conclusion Bar charts are usually the most suitable in situations when comparing and quickly communicating quantitative data, while line or spline charts are usually best to display growth, trends, or changes happening through periods of time. Pie or donut charts, while being very aesthetically pleasing, should be used sparingly: when comparing parts to a whole and when those parts have significantly different sizes, making the point we are communicating very obvious. There are many more charts available to be used in various scenarios. Some are much more complex and serve well in specific situations, but the ones we covered here are the most easily interpreted and understood if used in a correct manner. "UX/UI Design: Most Commonly Used Charts" Tech Bite was brought to you by Jasmina Spahić, Junior UI/UX Designer at Atlantbh. Tech Bites are tips, tricks, snippets or explanations about various programming technologies and paradigms, which can help engineers with their everyday job.

Lately, we have seen a significant number of discussions regarding the potential of AI and strategies to incorporate it into the product development process. Specifically, how can we enhance, innovate, and scale in ways we were not able to do before? In this interview, we asked Samra Tanovic, our VP of Analytics and Services, to share her experience and insights on this topic. Q: Could you tell us about your journey in incorporating AI into products?  A: Several years ago, when the need for AI functionalities arose within one of our projects, we were faced with a series of questions: How do we enable our teams and build the expertise needed for such tasks? How long will it take? How do we spot opportunities for the use of AI faster? What are the costs? The key takeaway from this experience is that it’s essential to start with an explicit, well-defined goal that aligns with business and user needs and then to consider how AI could assist in achieving this goal. It’s important to recognize that not all problems require an AI solution. There’s a risk of wasting significant time and resources trying to utilize AI just for the sake of using it. Instead, we learned to carefully examine existing challenges and ask: Can AI solve this problem more efficiently? Is there potential for scalability? Does the benefit of using AI outweigh the cost? Our journey also taught us that data science projects, unlike standard software engineering, involve a series of experiments that may or may not lead to discoveries or entirely new lines of research. The inherent uncertainties in this process make it difficult to define and control, which is why a unique management approach is required. Another key aspect we realized is the importance of the data that AI uses and produces. While the size of the training data does impact the model’s performance, data quality often plays a considerably more influential role. Investing in clean, relevant, and diverse data preprocessing turned out to be essential for accurate AI outcomes, making it an investment worth prioritizing. Finally, in order to build expertise and prepare our company for the rapidly evolving AI landscape, we set up a team dedicated to the research, development, and implementation of cutting-edge AI solutions within existing and new projects. This step allowed us to continue innovating and incorporating AI where it made sense, enhancing both our products and our ability to adapt to future technological advancements. Q: Could you provide examples where your teams used AI to enhance problem-solving efficiency? A: The main criterion is to consider processes that are excessively manual, do not scale well with product growth, or areas where customer experiences could be improved through personalization or automation. ML is mostly required in cases when the targeted behavior cannot be accurately expressed in software logic without relying on external data sources. A few years back, we used AI to analyze text and extract common business attributes, such as business category, address and operating hours from an unstructured text, a process that was previously time-consuming for humans. The efficiency of the validation of the business attributes, which was our main use case, was increased by 80%. An interesting and simple example involved removing a common UI element, a dropdown menu, and using classification to determine the correct option automatically. Previously, selecting the wrong option led to downstream problems and complicated user experience. However, by leveraging ML to accurately determine the appropriate dropdown option based on the user’s input, we managed to reduce mistakes by 30%.  Moreover, an intriguing area to consider is threshold replacements. Traditional business rules with set thresholds are often used to trigger specific actions. However, these thresholds can be too rigid and may not adapt well to changing circumstances. AI models, in contrast, can learn from data patterns and make more flexible, dynamic decisions.  Q: Lastly, drawing from your experience, what are some obstacles that businesses might encounter while implementing AI, and what strategies can they employ to overcome them? A: One of the biggest challenges is the lack of trust in AI-powered solutions. Many businesses are hesitant to rely on AI for critical decisions because they don't fully understand how algorithms work or the resulting accuracy is not meeting their expectations.  AI systems are designed to learn but can make errors or provide inaccurate results based on various factors, such as the quality of the training data, complexity of the task, and limitations of the algorithms used.  In scenarios where explicit and deterministic results are critical, businesses should carefully assess the suitability of AI models. Hybrid approaches that pair AI with deterministic methods should be considered, along with models with high interpretability, to maintain control and build trust. It is also useful to be transparent about limitations and potential biases.  Speaking of biases, if not designed and trained with fairness and ethical considerations in mind, AI systems have the potential to perpetuate biases and discrimination. Ensuring compliance with regulations, regular audits, testing for bias and seeking legal advice can help. Q: Thank you. It was a pleasure discussing this topic. A: Thank you. I'm excited to see how AI continues to shape and enhance the field of product development. With over 6 years in the AI field, Atlantbh can help you utilize the potential of AI to optimize existing business operations and products. Find out how we can help, contact us.