Amina Poplata

In the global, increasingly growing IT industry, we hear the term ‘DevOps’ almost daily. But how often do you encounter ‘DataOps’ related topics? We all use some of its practices, but it still seems relatively unfamiliar to many. In a world where data is the core of almost everything we do, it’s becoming just about crucial to know and use DataOps. This blog will introduce you to the ‘DataOps’ concept and some of its best practices helping you to gain base data management knowledge. So, what is DataOps? As DevOps focuses on improving software development management, DataOps makes data analysis workflows more efficient. DataOps, short for “data operations”, is an agile-oriented methodology that gathers best practices, processes, and technologies employed in data management. You can get some great tips on applying the Agile methodology in our earlier blog with Agile and Scrum tips. DataOps follows the data from its source through the final product, optimizing the complete process. According to the definition of DataOps in TechTarget short blog:  “Inspired by the DevOps movement, the DataOps strategy strives to speed the production of applications running on big data processing frameworks. DataOps also seeks to liberate silos across IT operations, data management and software development teams, encouraging line-of-business stakeholders to work with data engineers, data scientists and analysts. The goal is to ensure the organization's data can be used in the most flexible, effective manner possible to achieve positive and reliable business outcomes.” The DataOps Lifecycle The previous photo represents the so-called DataOps lifecycle created based on the model from the Snowflake data platform blog named The Rise of DataOps: Governance and Agility with TrueDataOps. In practice, it consists of the following key steps:  Plan - Every process needs a good plan. It’s important to define how business requirements can be addressed using data analytics. You should know what steps should be taken and which technologies should be used. Most often, it’s also needed to specify the budget and define performance requirements. Develop - Adequate code and data pipelines should be developed to effectively work with data, including data analysis, transformation, and ingestion. Based on the requirements and team preferences, code is mainly written in Python, R, SQL etc.  Build - Developed models should be built into a functional whole in this phase. Test - Data should be tested to verify if it matches the defined business logic and desired output. Data should also go through base analysis checks.  Release - Data is released into the test environment for further validation. Deploy - Once validated, data is deployed into the production environment. Operate - Data product is delivered, and stakeholders are asked for feedback. Any deviations from the desired output should be fixed.  Monitor - This step includes observing the complete process continuously. Code running against data should be observed, and if any problems show up, they should be handled as soon as possible.  The DataOps lifecycle is a process that begins with raw data and ends with real insights. Now, let’s look into the best practices to apply throughout this cycle.  Top DataOps practices to make your data management extra efficient In this paragraph, we’ll focus on some of the best practices in DataOps which will make your data management more accessible and efficient.  Utilize the agile approach Advice is to start small and build up on it. Instead of building everything at once, we should begin from specific components and data subsets and then scale the DataOps process step by step.  Agree with stakeholders on what good data quality looks like It is essential to clearly define measures for good data quality to use the most of your available time and potential. You should be clear on who will use the data product, in which way they will be using it and its final purpose. These key performance indicators (KPIs) should be revisited periodically to ensure all requirements changes are updated accordingly. Automation is the time-saving key Everything that can be automated - it should! This, of course, refers to iterative tasks that your team uses frequently. If you encounter a one-time task for which automation may not be simple, there is no point in investing your time in automation when you could be resolving that task much faster manually. However, investing time in the automation of repeating tasks can be a game changer for your team. Besides saving time, you will also reduce the impact of human errors.  Define your MVP and then evolve Data products don’t need to be perfect from the start to satisfy basic requirements. The suggestion is to define a minimum viable product (MVP), test, and release it. Then, if possible, monitor it in production to improve it, working together with stakeholders and using constructive feedback. You will also have real insight into your data product so you can detect patterns and possible issues as they arise, ultimately improving your data product over time.  Empower business stakeholder with self-serving tools Avoid being a bottleneck for stakeholders seeking basic information. Enable self-service access to the right tools, allowing them to retrieve the data they need independently. This promotes efficiency within your team and empowers stakeholders to access data without unnecessary assistance. Know the priorities Recognize that not all data is equally important. Begin by focusing on the most important data assets - the ones that will be the main deal-breakers for the stakeholders.  Handle data sensitivity It is important to have in mind which type of data we are dealing with. Sensitive data always requires extra security measures, and teams need to be aware of data confidentiality and special permissions it requires. Ensure efficient data distribution Investing time in optimizing data delivery for all users can save time and energy for everyone involved. Document everything! I can not stress enough how important this is! Clear and comprehensive documentation is a revolutionary element in data-related processes (in any process, really!). It makes knowledge sharing easier and also allows you to address and fix issues more promptly.  Continuous improvement Using DataOps in your processes is an ongoing effort that requires continuous monitoring and taking action. This involves feedback from stakeholders, addressing new requests, fixing detected issues, and optimizing the process in general. Simplify your (data) life - use DataOps! Implementing DataOps practices in your teams is a significant step forward. Today, there are also many tools available, depending on your needs, which make DataOps implementation in your processes much easier. For example, there’s DataKitchen DataOps platform, which streamlines and optimizes data analytics and data engineering processes in organizations with various solutions. There are also many other useful tools, such as: dbt (an open-source tool that transforms and manages data in the cloud)  Apache Airflow (an open-source platform for instrumenting complex data workflows) Atlan Data Governance (an active data governance platform offering scalable ways to secure data while ensuring data accessibility) In an interesting article explaining 3 Examples of AI at Work in DataOps you can read about examples of DataOps real-life implementations. And not just any examples, it’s about some of the most popular organizations in the world (Uber, Netflix, and Airbnb). These companies have utilized the power of DataOps to boost their productivity and overall success significantly. The benefits of embracing DataOps are uncountable, even though we’ve only “scratched the surface” of the DataOps world in this blog. The advantages can clearly be seen, and DataOps has all the potential to lift your organization to a higher level. The real question is, are you ready to dive into the DataOps world and transform how you manage your data? If you found this useful, check out other Atlantbh blogs!

What if I told you none of the people from the picture below exists? You are wondering how? Well, the most simple answer is: synthetic data. This type of data can be anything: text, media, or tabular data. You can generate a random human face at any time using this generator. These fake portraits look so realistic that it’s almost frightening, right? They can be used for marketing purposes with no legal confirmation needed! However, the generated portraits (as any other type of synthetic data) are not the final goal themselves. They are most often just the means to train other, more complex models. For example, synthetic images are of great importance for computer vision algorithms. Let’s look deeper into the definition! Synthetic data is data which is mainly algorithmically generated by trained AI to approximate original data in order to be used for the same purpose as the original data. In the example of fake portraits mentioned previously, the system learned the properties of people's photos from real life using AI in order to generate realistic images of human faces. Generally, the algorithm first learns patterns, correlations, and statistical properties of the real-world sample data. As a result, trained AI can generate synthetic data statistically identical to real-world data, which is a true gold mine.  The Rise of Synthetic Data in AI Having good-quality data is the core of every successful project. It is the most important and challenging part of building a successful AI. Collecting real-world data is expensive, complicated, and requires a lot of time, patience, and energy. In 2006 British mathematician Clive Humby said, "Data is the new oil," which has spread worldwide and been proven correct. Data holds tremendous value - today, entire industries are powered by data. But not everyone is lucky enough to have at least sufficient data resources to complete their projects. As a result, many are resorting to creating their own data, which is affordable, effective, and efficient - this is where synthetic data comes in. In a Gartner report on synthetic data (Maverick Research: Forget About Your Real Data – Synthetic Data Is the Future of AI", Leinar Ramos, Jitendra Subramanyam, 24 June 2021), authors predicted that by 2030, most of the data used in AI would be artificially generated by rules, statistical models, simulations or other techniques.   The importance of synthetic data in AI is demonstrated by the fact that the most common application of this type of data is in AI/ML model training. Machine learning models are increasingly reliant on synthetic data for training purposes, as synthetic data often outperforms real-world data and is critical in developing high-quality AI models. Another typical application of synthetic data is in testing data. Synthetic data is simpler to generate for testing purposes, such as production or operational data than rule-based test data. Synthetic data is greatly useful in stress-testing AI models with rare real-world events or patterns and can aid in eliminating biases that are often present in real-world data. It is also critical for data-driven testing and software development. AI as the Central Engine of Synthetic Data Generation The most effective method for creating synthetic datasets is using AI. To produce dependable synthetic data, the crucial requirement is to have a sample of original (real-life based) data, which we can feed into our synthetic data generator to enable it to learn the data's statistical properties, such as correlations between data, distributions, and patterns. The greater the original data sample, the better the quality of the resulting synthetic data. Fundamentally, there are two principal methods for acquiring synthetic data: Using generative models Adopting conventional techniques, such as utilizing specialized tools and software in conjunction with procuring data from third-party sources  Both approaches can be employed to produce diverse varieties of synthetic data. Generative Adversarial Networks (GANs) consist of two constituent sub-models: a generator and a discriminator. The generator's role is to fabricate artificial data, whereas the discriminator's objective is to verify whether the generated data is authentic or fake. Since both sub-models oppose each other, the term "adversarial" is used to describe this approach.  As for conventional techniques, although you can always develop your own AI to generate synthetic data if you're solely interested in the final product or lack the resources or expertise to build your own AI, there are alternative options available. For instance, the Python programming language has several implemented options for generating synthetic data, as described in this Towards Data Science blog. Additionally, multiple synthetic data generation tools are available, some of which are free to use but with limited resources. One such example is MOSTLY AI, an AI-driven synthetic data generator that provides up to 100k rows of synthetic data per day for free (following registration). The picture below displays some additional examples of synthetic data generation tools, and more information about them is available in this Turing article. Synthetic Data Applications Having established the definition, significance, primary purpose, and acquisition methods of synthetic data, we will now go through a brief survey of some real-world instances where synthetic data forms the essential ingredient. One example of a synthetic data application is Amazon’s famous product Alexa, which I’m sure we have all heard of. Amazon trained Alexa on synthetic data to recognize requests in multiple languages. When Amazon implements a new language into Alexa’s system, the data sources for the machine learning model are extremely poor. That’s where the synthetic data kicks in.  Another interesting example of usage is in Google’s self-driving car project called Waymo. Waymo uses synthetic data to train self-driving vehicles. They’ve created an environment where a vehicle is trained on labeled synthetic data along with real data to drive safely while recognizing objects and following traffic rules. Created environment imitates both good and bad road situations for self-driving cars to learn.  Some industries that can benefit from synthetic data most are financial services, healthcare, manufacturing, security, and social media. These industries have already started leveraging synthetic data in significant ways, but there is still a vast untapped potential waiting to be discovered. Ongoing research and exploration into synthetic data applications are crucial to unlocking its full potential.  Let’s look into a bit of a different example of synthetic data application. Andrej Karpathy, a computer scientist who served as the director of artificial intelligence and Autopilot Vision at Tesla, developed an impressive language model named makemore. makemore takes one text file as input, where each line is assumed to be one training object and generates output objects similar to the provided input. He made the code open-source and very easy to adjust to your own needs. You can also play with this tool on your own, without any hussle, since it’s super-user-friendly, and there are even detailed instructions on how to use it published on Karpathy’s official YouTube channel. makemore is categorized as one of the generative models discussed in the previous section, and it’s somewhat a lite version of ChatGPT, which has already become popular worldwide.  In this blog, makemore was used to generate new Bosnian names. This seemed like an adequate example since most new parents and parents-to-be struggle with a creative name choice for their baby. In order to use makemore successfully, the first step was cloning the Git repository into desired destination folder using the following command in the terminal: git clone https://github.com/karpathy/makemore.git Now, I was able to view and edit the main code file ‘makemore.py’ to adjust it to my goal. I used Visual Studio Code editor to edit and test the code, which I strongly recommend since it’s easy to use, but also very powerful. However, you can use any Python-friendly editor which suits you. Code did not require a lot of editing (it can be used without any editing at all, which will be explained later), and with clearly provided instructions on how to use makemore and helpful comments throughout the entire code, it took me only a couple of minutes after I got an initial grip of the code structure. Since I wanted to get new Bosnian names as a result, I needed to provide Bosnian names as input for training as well. I collected a small sample of around 300 Bosnian names for girls and boys and saved it in a text file named ‘imena.txt’. Regarding this, I made a change in the code as well. The initial code uses English names for training from a default file named ‘names.txt’. I provided a path for the new file ‘imena.txt’ instead of the original default file name ‘names.txt’ in the arguments definition section: parser.add_argument('--input-file', '-i', type=str, default='/Users/admin/Desktop/Blog-SyntheticData/Code/makemore/imena.txt', help="input file with things one per line") I also edited the default output working directory path:  parser.add_argument('--work-dir', '-o', type=str, default='/Users/admin/Desktop/Blog-SyntheticData/Code/makemore/output', help="output working directory") and changed the default maximum number of optimizations as it suited my needs (10,000) after testing it a few times (it was initially set up to infinite): parser.add_argument('--max-steps', type=int, default=10000, help="max number of optimization steps to run for, or -1 for infinite.") One additional change I made in the code was adding functionality to save new sample names in an output text file where I could review them more clearly. I added this code snippet in print_samples function definition by adding a new for loop, in the end, to write those new name samples in a text file named ‘output_ba.txt’, each new name in a new line:  for new_name in new_samples: with open('/Users/admin/Desktop/Blog-SyntheticData/Code/makemore-master/output_ba.txt', 'a') as f: if(new_name!=''): f.write(new_name) f.write('\n') After all changes are applied and saved, the file can finally be run using VS Code (or any other dev environment you’d like to use). Another easier way to run and use makemore is through terminal commands without editing the initial code. You just need to open the terminal in the destination folder where your files are saved. The main script can be run using the following command: admin@192 Code % python3 makemore.py —input-file imena.txt —working-dir output —max-steps 10000 When using the previous command, there is no need to edit default input and output (working directory) paths nor the maximum number of optimizations through code - you just define them directly in this command. You can set up any other input argument through this terminal command, you just need to check the desired input variable name defined in the code. The only thing which will be missing without editing the code and just running the file using the terminal is the output text file with new names, but new names are constantly being written in the terminal as the code executes, so that you won’t miss much. By editing the code, I just wanted to show how it is extremely user-friendly and easy to understand and edit. Note that terminal commands can be slightly different if you have installed ‘python’ instead of ‘python3’, which was used in the examples above.  After the main file was successfully run, the model started to generate new meaningful names after only a few iterations. And I must admit, most of the new names sound really awesome. Several examples of training and newly generated names are shown in the table below.  To be honest, some names can sound a bit extreme, like džeon, miliš, hekslanjan, or arh, but like the old Latin saying states: De gustibus non est disputandum (In matter of taste, there can be no disputes!). However, output names improve significantly by iterations. The more iterations and the larger input sample we assign, the better quality result we can expect. For example, you can use this model to generate names for your new business or pet. The possibilities of its use are endless.  Pros and Cons  Synthetic data has significant benefits, including being cost-effective, efficient, easy to generate, (ideally) unbiased, and able to replicate rare or restricted real data. Since this type of data is generated under controlled conditions, we can also significantly improve data integrity which we’ve discussed in one of our previous blogs, “How to Ensure Data Integrity?”. In our other blog, “Data centric AI / Big data vs. Good data,” we mentioned the importance of good and big data for successful AI models - synthetic data can be both! However, there are some downsides, such as the possibility of missing outliers that may contain crucial information and the fact that the quality of generated synthetic data greatly depends on the input data and generation model. Additionally, synthetic data may be challenging to accept from users and cannot completely replicate real-world events.  Overall, synthetic data is an excellent tool for the AI world, but its benefits and drawbacks must be carefully considered before use. 

The well-known passenger ship Titanic, for which there were rumors that it was “unsinkable,” had a severe accident in 1912. after hitting an iceberg on its first trip. It is estimated that there were 2,224 passengers on the ship, and more than 1,500 people died, which made this one of the biggest maritime disasters in modern history.  Data analysis behind a disaster While some element of luck was included in surviving this shipwreck, after analyzing the data, it was concluded that certain groups of people were more likely to survive than others. This blog answers the question “Which characteristics of Titanic passengers increased their chances of survival?” using available personal data about passengers.  For this purpose, we’ll use a dataset from Kaggle, which we will train to make passenger survival predictions. “No data is clean, but most is useful“ - Dean Abbott Raw data is often hard to understand, and its preprocessing is usually the most important step in every analysis. Good quality preprocessing of the Titanic data sample enables us to make a successful model for predicting passengers survival. All of this is easily done using the Python programming language and its libraries (e.g. pandas, numpy, sklearn). In this blog, we used it, along with Google Colab, to do an analysis of the Titanic dataset.  The downloaded data sample contained 891 records in total, and its basic attributes information was accessed using Python command  data.info() The following table represents the results.  As seen at first sight, there are three attributes that don’t have full completeness. A useful code snippet to detect missing values is the following: total = data.isnull().sum().sort_values(ascending=False) percent = data.isnull().sum()/data.isnull().count()*100 percent_rounded = (round(percent, 1)).sort_values(ascending=False) missing_data = pd.concat([total, percent_rounded], axis=1, keys=['Total', '%']) missing_data.head(12) One of the steps when building a prediction model is detecting attributes with missing values and further preprocessing if necessary. Filling missing values isn’t always simple and depends on the dataset itself, the number, and the type of missing values. We need to be extra careful not to have too much impact on final model predictions by adding some random values in the dataset. In our case, the attribute ‘cabin’ with the most missing values was detected as non-relevant for further analysis and was dropped from the dataset. A few missing values in the ‘embarked’ attribute were filled with the most frequent value - “Southampton”. Finally, missing values in the ‘age’ attribute were filled with random ages in the standard deviation range around the attribute's mean value. The next preprocessing step is related to the ‘name’ attribute. By itself, the name is not relevant for survival predictive modeling. However, names in this dataset contain titles that were successfully extracted and mostly mapped to standard titles (‘Mr’, ‘Mrs’, ‘Miss’, ‘Master’). This resulted in title information, which in contrast to the name, could be helpful in the final model. Code snippet related to mentioned preprocessing is the following: # Default titles for mapping (most frequent titles and other “Rare” titles) titles = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Rare": 5} # Extracting existing titles in names data['Title'] = data.Name.str.extract(' ([A-Za-z]+)\.', expand=False) # Map existing titles to default titles data['Title'] = data['Title'].replace(['Lady', 'Countess','Capt', 'Col','Don', 'Dr','Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare') data['Title'] = data['Title'].replace('Mlle', 'Miss') data['Title'] = data['Title'].replace('Ms', 'Miss') data['Title'] = data['Title'].replace('Mme', 'Mrs') “You can have data without information, but you cannot have information without data.” - Daniel Keys Moran  It’s up to us to read from the data and extract useful information. Before making a prediction model, data were additionally analyzed to make connections between attributes. The results of the data connections research were quite interesting and helpful. I will show a few examples in this blog. In the following picture, passengers’ survival is presented as dependence on their age and sex. Code snippet used to plot these graphs is the following:  survived = 'survived' not_survived = 'not survived' fig, axes = plt.subplots(nrows=1, ncols=2,figsize=(10, 4)) women = data[data['Sex']=='female'] men = data[data['Sex']=='male'] ax = sns.distplot(women[women['Survived']==1].Age.dropna(), bins=18, label = survived, ax = axes[0], kde =False) ax = sns.distplot(women[women['Survived']==0].Age.dropna(), bins=40, label = not_survived, ax = axes[0], kde =False) ax.set_xlabel('Age') ax.legend() ax.set_title('Women') ax = sns.distplot(men[men['Survived']==1].Age.dropna(), bins=18, label = survived, ax = axes[1], kde = False) ax = sns.distplot(men[men['Survived']==0].Age.dropna(), bins=40, label = not_survived, ax = axes[1], kde = False) ax.set_xlabel('Age') ax.legend() _ = ax.set_title('Men') Observing the picture above, the following conclusions can be made:  Women were more likely to survive than men Little boys were more likely to survive than adult men Infants had a higher chance of survival As a result of mentioned observations, passenger age groups were used later in the predictive modeling. Another relation, the number of survivors in dependence on the passenger class, is shown in the picture below. The element of age was added to the graph as a vertical axis, and survivors count as the horizontal one. As seen from the graphs, the lowest survival chance was for the passengers with third class tickets in age from around 18 to 40 years, and the highest survival chance was for middle-aged passengers with first-class tickets. This resulted in adding a new attribute: age group multiplied by passenger class, where passengers with the lower value of this attribute seem to have had more chances to survive. After final data processing based on mentioned and multiple other detected data correlations, it was possible to build a model to predict the survival outcome of Titanic passengers. “The best way to predict the future is to study the past, or prognosticate.” - Robert Kiyosaki The final step in this analysis was to build a machine learning model for prediction. First, the dataset was divided into train and test data using a 70:30 percentage ratio. After that, two exemplary models were built by using two different algorithms on train data: Gaussian Naive Bayes and Random Forest. The following code snippet does the mentioned work for Random Forest model: # Split dataset to train and test data from sklearn.model_selection import train_test_split X = data.drop("Survived", axis=1) Y = data["Survived"] X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=11, stratify=Y) # Random Forest model random_forest = RandomForestClassifier(n_estimators=100) random_forest.fit(X_train, y_train) Y_prediction = random_forest.predict(X_test) random_forest.score(X_train, y_train) # Model accuracy acc_random_forest = round(random_forest.score(X_train, y_train) * 100, 2) print(acc_random_forest) Using Naive Bayes, we got a model with 78% accuracy, while Random Forest algorithm resulted in a 92% accurate model. This means that when the Random Forest model was tested, it successfully predicted survival outcomes for 92% of passengers from test data. The following picture shows the top six most important attributes for survival prediction using Random Forest. All of these attributes were expected to have high importance based on the analysis made throughout this blog. This short example showed how data analysis and machine learning are truly amazing technical areas that can be used in multiple ways to extract useful information and build models to predict outcomes and help the future. Read more Atlantbh blogs about Data Science & Analytics!