Mehmed Kadric

What are Random Strings? Data analysis often presents us with the challenge of dealing with noisy inputs. This is particularly evident when working with large datasets of user inputs. Detecting and filtering out random string inputs can prove invaluable in various scenarios, such as data validation, quality control or the development of machine learning (ML) based natural language processing (NLP) systems (check for data quality improvement steps in our previous blog post). Random string inputs can significantly impact the accuracy and effectiveness of data analysis and ML models. By identifying and filtering out these noise elements, we can enhance the reliability and quality of our results. Image 1 - Random Strings What should/can be considered a ‘random string’? The term "random string" may seem straightforward at first glance, but its interpretation can be subjective and context-dependent. What one person perceives as a random string, another may see as a meaningful pattern or representation. This relativity is particularly evident when considering visual or linguistic aspects. Let's delve into a few examples to highlight the importance of relativity when defining random strings. Take, for instance, the string '\__o__/'. At first glance, it might appear random, but one could argue that it resembles a stick figure of a man with raised hands. In this case, the interpretation of randomness becomes subjective, as different individuals may perceive different patterns or associations. Similarly, let's consider the word "Szczesny." In the English language, it may appear as a random combination of letters, devoid of any meaning. However, in Polish, it happens to be the surname of a famous goalkeeper (Image 2). This example demonstrates how the interpretation of randomness can be influenced by language and cultural context. What may seem random in one language could hold significance or meaning in another. Image 2 [Source: Yahoo/Reuters] - Wojciech Szczesny, a Polish professional footballer It's crucial to acknowledge that randomness is not an inherent property of a string itself but rather a perception based on patterns or associations that individuals may perceive. In the context of data analysis and ML, understanding this relativity is vital when developing algorithms or models that aim to detect and filter out random strings. When designing systems to identify random strings, it becomes essential to consider the specific domain, context, and purpose of the analysis. What may be considered random noise in one scenario could be a valuable signal in another. For instance, when analyzing social media data, a sequence of random characters might indicate spam or bot-generated content. On the other hand, in cryptography, random strings play a crucial role in generating secure encryption keys. Moreover, the concept of randomness itself can vary depending on the intended application. Some algorithms generate strings that may appear random to human observers but possess specific statistical properties that make them suitable for cryptographic purposes. In contrast, other applications require strings that exhibit specific patterns or structures to serve their intended function. How to detect a random string of characters in a specific locale? If the goal is to detect random strings which consist of a specific set of characters (e.g., English alphabet characters), one possible way of detecting random strings is using bigrams.  To be able to do that, you need to create a reference dictionary of bigrams for the considered locale. For the English language, we can use bigrams and their frequencies from Peter Norvig's analysis. For example, in English language text, the bigram “th” is very common, while the bigram “zx” is not. Using the available list of bigrams and their frequencies, it is possible to normalize it (e.g., min-max normalization) and create a reference dictionary. An important note here is that you can easily create your own list of bigrams (extracted from word corpus) for any locale if you are able to find open-source books, newspapers or magazines. Reference dictionary is a dictionary of all 26² = 676 combinations of English bigrams with associated frequency numbers (from 0 to 100) where higher values represent more frequent bigrams (like “th”) and lower values represent less frequent bigrams (like “zx”). Users are able to adjust the threshold value (default value is 0.1) in order to define a boundary between less common and common bigrams. For example, if the threshold is 0, all bigrams will be considered as common bigrams. Otherwise, if the threshold is 100, all bigrams will be considered as less common bigrams. You can find a reference dictionary of bigrams here. To consider whether a word is a random string or not, here is a list of steps that are used to help detect random strings: Length and Character Validation: The first checkpoint of the algorithm verifies that the word is longer than three characters (words up to three characters may be regular abbreviations) and consists solely of English alphabetic characters. This ensures that we are working with meaningful words rather than arbitrary combinations of symbols Repeating Characters: If the entire word is made up of the same character, such as “hhhhhhh”, it is immediately labeled as random. This is because the presence of repeating characters indicates a lack of complexity and structure, suggesting a random generation Lowercasing the Word: To eliminate the case sensitivity, the algorithm converts the word to lowercase. This prevents the distinction between uppercase and lowercase characters from affecting the detection process Bigrams Analysis: A bigram refers to a sequence of two consecutive characters in a word. The algorithm generates a list of all possible bigrams within the word. For example, the word "random" would produce the following bigrams: (ra, an, nd, do, om) Common and Uncommon Bigrams: Each bigram is compared against a reference dictionary of common English bigrams. If a bigram has a frequency above a given threshold, it is considered common; otherwise, it is classified as uncommon. The choice of threshold depends on the desired sensitivity of the algorithm Counting Common and Uncommon Bigrams: The algorithm calculates the number of common and uncommon bigrams in the word. If the count of common bigrams exceeds that of uncommon bigrams, the word is classified as non-random. Conversely, if the count of uncommon bigrams surpasses that of common bigrams, the word is deemed random Here are Python and Java codes for the proposed algorithm: Python: def is_random_string(word, threshold=0.1): # Allow only words longer than 3 characters which contain only English alphabetic characters if len(word) < 4 or not word.isalpha(): return False # Turn word into lowercase word = word.lower() # Repeating characters if len(set(word)) == 1: return True # Get list of bigrams from the word bigrams = [word[i:i + 2] for i in range(len(word) - 1)] # Get number of common and uncommon bigrams num_common_bigrams = sum(1 for bigram in bigrams if en_bigrams_dict.get(bigram, 0) > threshold) num_uncommon_bigrams = len(bigrams) - num_common_bigrams # Higher number wins if num_common_bigrams > num_uncommon_bigrams: return False else: return True Java: package checker; import java.util.HashMap; import java.util.Map; public class RandomStringChecker { private static final double DEFAULT_THRESHOLD = 0.1; private static final Map<String, Double> enBigramsDict = new HashMap<>(); static { // Populate the dictionary of English bigrams here enBigramsDict.put("ab", 6.461565670356169); enBigramsDict.put("bc", 0.0531330714265234); enBigramsDict.put("cd", 0.06273467822837461); // ... } public static boolean isRandomString(String word) { return isRandomString(word, DEFAULT_THRESHOLD); } public static boolean isRandomString(String word, double threshold) { // Allow only words longer than 3 characters which contain only English alphabetic characters if (word.length() < 4 || !word.matches("[a-zA-Z]+")) { return false; } // Repeating characters if (word.chars().distinct().count() == 1) { return true; } // Turn word into lowercase word = word.toLowerCase(); // Get list of bigrams from the word String[] bigrams = new String[word.length() - 1]; for (int i = 0; i < word.length() - 1; i++) { bigrams[i] = word.substring(i, i + 2); } // Get number of common and uncommon bigrams int numCommonBigrams = 0; for (String bigram : bigrams) { if (enBigramsDict.containsKey(bigram) && enBigramsDict.get(bigram) > threshold) { numCommonBigrams++; } } int numUncommonBigrams = bigrams.length - numCommonBigrams; // Higher number wins return numCommonBigrams <= numUncommonBigrams; } } This algorithm has been released as a Python package and can easily be installed using the command: pip install random-string-detector Examples Python method calls: print(is_random_string('abcd')) # True print(is_random_string('Thgrbh')) # False print(is_random_string('Thgrbh', 5)) # True - threshold adjusted Java method call: public static void main(String[] args) { System.out.println(isRandomString("abcd")); // true } Conclusion The interpretation of randomness depends on the specific domain, context, and purpose of the analysis. What may be considered random noise in one scenario could be valuable information or a critical signal in another. It is essential to align the definition of random strings with the objectives and requirements of the particular application to ensure the reliability and quality of results. While the proposed algorithm is not foolproof and cannot definitively determine randomness, it serves as a useful tool for various applications that require distinguishing between random and non-random strings. A potential algorithm improvement would be to have the bigram dictionary in a separate file that we could load into memory.

We can agree that AI is not a one-size-fits-all solution. However, for many companies, AI can provide significant benefits and help drive growth and success. Some potential benefits include the following: Improved efficiency: AI can automate tasks and processes, saving time and resources for other activities. Increased accuracy: AI can make more accurate predictions and decisions than humans, reducing the risk of errors. Enhanced customer experience: AI can personalize customer interactions, providing a more tailored and seamless experience. Cost savings: AI can help businesses reduce costs by automating tasks, improving efficiency, and reducing the need for human labor. Competitive advantage: Companies that use AI can gain a competitive advantage over those that don't by being able to analyze and act on data faster and more effectively. The last decade has brought significant growth in the field of Data Science in general. However, the total value of AI is still locked in many sectors, such as health care, manufacturing, and government technology. In a study published by Accenture, 80% of all Proof of Concepts (PoCs) do not make it into production. What exactly is going wrong, and why is AI not as successful in the real world as it is in academic studies? Model-centric vs. Data-centric approach A strong model-centric AI (traditionally presented in academia) considers data only as a static parameter. If a model's performance is not as expected, AI/ML engineers will try to tune the model's hyperparameters or even change the model while data stays untouched. Subject Matter Experts do not play a vital role in the whole process of developing AI-based systems in a model-centric approach. Instead, AI/ML engineers usually make business-related decisions during that process. As per Andrew Ng, an AI system is a combination of code and data. While a significant amount of time and resources have been dedicated to developing code and algorithms, it is now necessary to prioritize improving data quality and relevance to achieve desired outcomes.  By shifting the focus to data, we are basically moving from model-centric to data-centric AI. In a data-centric approach, the role of Subject Matter Experts is significant, and AI systems are much closer to every business. Adopting a data-centric approach in the real world or during work on a practical project is more fruitful. That means the model stays fixed and focuses on feeding the model with good or high-quality data. A decent algorithm with good data may even outperform a great algorithm with not-so-good data. Data should not be the only priority in the development process. It is also important to consider the structure and functionality of the model and code used. The choice of model can significantly impact the accuracy, as demonstrated in the analysis of the Titanic disaster, where the prediction accuracy varied from 78% to 94%, depending on the algorithm. Big Data vs. Good Data: What's More Important for AI? Good Data and Big Data are two terms often used in the field of Data Science and AI. Good data refers to data that is accurate, relevant, and well-organized. Big data, on the other hand, refers to large datasets that are too large and complex to be processed and analyzed using traditional data processing tools and techniques. In order for AI algorithms to make accurate predictions and decisions, they need to be trained on large amounts of data. However, simply having a lot of data is not enough. The data must also be high-quality and well-organized for the AI model to learn effectively. In many cases, companies only have access to small datasets, which can lead to poor results if the focus is solely on the model. Andrew Ng emphasizes the benefits of a data-centric approach to machine learning and suggests that there should be a shift towards this approach within the community. He uses the example of a steel defect detection problem, in which the data-centric approach improved the model's accuracy by 16% compared to a model-centric approach [Source: neptune.ai]. Data Quality Issues Many famous datasets have data quality issues. For instance, The COCO and ImageNet datasets are widely used in the field of computer vision for tasks such as object detection, image classification, and image segmentation. While these datasets are widely used and generally considered high quality, some issues have been discovered with them that may impact their usefulness for specific tasks. Some of the problems detected in the COCO dataset include mislabelled objects (first image) and labeling inconsistencies (second image) [Source: neuralception.com]. Ensuring Data Quality in Machine Learning Development While the specific methods for improving data quality may vary depending on a specific problem, some general steps can be applied in many situations: Verify data sources. Ensure that data sources are reliable and accurate. Make sure your model gets enough data to be able to generalize. Validate and clean data. Check labeling inconsistencies. Perform validation checks and look-ups on labeled data if applicable. Check labels distribution and manually check entries with low-frequency labels. Try to make a tool that will give you suspicious labels so you can check them manually. Standardize data to a common format or schema. Involve Subject Matter Experts. Document agreements after discussing inconsistencies with labelers. Analyze data and perform feature engineering. Train a model with labeled data and apply the model to training data. Check the diffs. Perform error analysis. The development of machine learning-based systems is a highly iterative process. Keeping track of your notes, findings, and conventions in a documentation system can help maintain data quality. By following mentioned steps and being vigilant about potential issues, you can help ensure that your data is of high quality and ready for use in your machine-learning projects. Wrapping Up It is important to consider the quality and quantity of data and the structure and functionality of the model and code used. By prioritizing data quality and involving subject matter experts, companies can improve the accuracy and effectiveness of their AI systems, drive growth and success, and gain a competitive advantage.