Dino Zivojevic

When an international client asked us to automate language detection by efficiently filtering batches of short text using Machine Learning, we ended up improving the quality of data and speeding up the process. This was achieved through a combination of existing ML models in a voting classifier manner employed to perform language detection of company names for a specific country, with geolocation information efficiently used as an additional input. We improved accuracy from 75% to 88% with a single model for common local languages. With more languages added, the model still achieved 85% accuracy. Automated language detection made it easier to clean data and spot mistakes. This also simplified other tasks, like translation, making the entire process faster and more efficient. Introduction Most of the data and knowledge today is stored in the form of plain text using natural language. Natural Language Processing (NLP) is a subfield of Machine Learning (ML) that enables interaction between humans and machines by adapting natural language to data forms that are interpretable by computers. In the multilingual environment, one of the crucial steps for several NLP tasks, such as machine translation and text classification, is determining the natural language in which a text is written. A subfield of NLP responsible for such a task is formally called Language Identification (abbr. LangID) or Language Detection. In the modern era, with the increasing use of social media, there is a significant amount of user-generated text content containing a small number of characters. Short texts are generally more challenging for language detection models. Even if an ML model achieves good performance on long text that contains multiple sentences, it does not mean it can properly generalize to shorter sequences, mostly due to the lack of context. On the other hand, in production, especially in real-time use cases, inference time is an important constraint besides the accuracy of the given model. The efficiency constraints sometimes prevent the use of complex ML models. In this blog, we use an ensemble of existing ML models in a voting classifier manner to perform language detection of company names for a specific country, effectively using geolocation information as an additional input.   Language Identification Language identification is the process of automatically determining the language of a given text. This problem can be solved with a primitive approach by using complete vocabularies of all considered languages and performing word matching. However, this is pretty inefficient and not scalable with a vast number of languages. Also, this approach is not robust in terms of inflections, compound, and misspelled words. Some traditional approaches are also based on character and word N-gram frequency analysis, assuming that specific character or word sequences are language-specific and identifying the language based on N-gram distributions. A more advanced approach is based on ML models, like the Bayesian Classifier of Support Vector Machines (SVM), which can be leveraged to perform language classification tasks on preprocessed and vectorized text. Finally, the most advanced and complex models are based on deep learning techniques, including Recurrent and Convolutional Neural Networks (RNNs and CNNs) and Transformers. Even though models based on RNNs (e.g., LSTMs and GRUs), especially Transformers, are designed to capture sequential dependencies in text, they have more complex architecture and aim to capture context from longer sequences. Therefore, they don’t seem suitable for use cases that deal with short strings and further introduce efficiency loss. Considering that, one of the most suitable pipelines is to preprocess input text (which implies data cleaning and vectorization) and then apply an ML model (e.g. a probabilistic one like Multinomial Naive Bayes Classifier). Since short strings for specific use cases can contain only one word, capturing the internal structure of words in the process of text vectorization by using character N-grams (like the FastText vectorization approach does) is desirable. The common pipeline of such classifiers is depicted in Figure 1. Figure 1: Example of the common pipeline for language identification Probabilistic models such as the Multinomial Naive Bayes Classifier are suitable for use cases that imply text processing since they perform better when applied to discrete data like word counts in documents. Besides Multinomial Naive Bayes Classifiers, linear classifiers like SVM are also applicable, if input features are properly derived from the input text. The approaches based on linear or probabilistic classifiers are more efficient than models based on neural network architectures and demand less training data, making them more suitable for this particular use case when we deal with very short strings. There are several open-source ML models for language identification available online for programming languages like Python. In the following table (Table 1) we provide an overview of such models that we used in our use case: Table 1: Overview of open-source language detection models All these models use the pipeline that encompasses the preprocessing step, and after that, probabilistic or linear ML model based on N-gram distributions. Individual outputs of the separate classifiers do not necessarily achieve great performance, especially with very short strings used as inputs and many considered languages as outputs. Therefore, with additional information for specific use cases, they can be implemented together as a voting classifier to achieve greater accuracy. Also, some additional heuristic steps can be leveraged to narrow down the set of potential outputs.   Problem Definition and Approach An interesting use case of language detection is the application of these models together to detect the language of company names for specific countries. The complexity of this problem is two-fold: (1) Company names are generally very short strings with a couple of words (sometimes even just one); (2) Company names could simply be acronyms that don’t necessarily belong to the specific language. The problem can be defined as follows: For a specific geographic region (e.g., a country) and a list of company names, the language in which a particular name is provided should be identified. Generally, language identification models face an accuracy decrease with the (1) Decrease of input text lengths and (2) Increase of potential languages considered as output candidates. The general idea is to combine different models for language detection as a voting classifier and to aggregate their individual outputs. Additional information that can be used in the implementation is geolocation, which can be leveraged to narrow down the set of considered languages by prioritizing official or most common languages for a specific country. Also, the additional heuristic step can be implemented, as in the Lingua model, which breaks down the detection pipeline into two steps: A heuristic approach uses a rule-based engine that determines the input text's alphabet and searches for unique characters in one or more languages. If exactly one language can be detected this way, there is no need for a probabilistic model. In any case, this step can be used to filter out languages that do not satisfy alphabetic constraints; Probabilistic detection based on the ML model can be executed in the second step if the heuristics approach fails to detect the language unambiguously. The complete pipeline of such an approach is presented in Figure 2. Figure 2: Language detection pipeline As previously explained, input strings that represent the company's name are first tested for specific characters that could determine output language without running an ML model. This first part of the pipeline could potentially reduce execution time. If language cannot be unambiguously determined this way, then a voting classifier encompassing 4 individual models is leveraged. Individual outputs from separate classifiers are further aggregated using a rule-based function that prioritizes local (official) language if consensus between classifiers is not reached.   Results and examples This approach was implemented and tested on a dataset representing the Register of Companies for Denmark subset. The dataset was manually labeled for ground-truth values so performance metrics could be further calculated. The distribution of languages in the test dataset is depicted in Figure 3. Figure 3: Language distribution in the test dataset The complete list of languages present in the dataset is following: Danish (75%) English (22%) German (1%) French (< 1%) Norwegian (< 1%) Spanish (< 1%) Italian (< 1%) Hebrew (< 1%) Polish (< 1%) Danish is the dominant language, followed by English (these two languages combined represent 97% of the dataset). The first step is to test every model individually without any additional information about geolocation to establish a baseline accuracy. In Figure 4, individual accuracies for respective models are presented: Figure 4: Individual accuracies without any geographic information Lingua achieves the best performance on the test dataset with 61% accuracy. Since these are the baseline metrics without any constraints, we can further use additional information about geolocation to increase accuracy for the specific country. Generally, the accuracy of the specific model can be increased by decreasing the number of considered languages for output candidates. Since our dataset is based on company names in Denmark (which is a priori input), we can use this information to limit the set of outputs. Therefore, we tested several versions of the Lingua model: Lingua v0 - Baseline version of the Lingua model that considers all available languages (the same model that was presented in Figure 4); Lingua v1 - Version of the Lingua model that considers only languages that are present in the dataset (Figure 3) as output candidates; Lingua v2 - This Lingua version excludes the Norwegian language from consideration since most of the false predictions are between Danish and Norwegian due to their similarity; Lingua v3 - This final version takes advantage of the fact that Danish and English languages cover 97% of the dataset (Figure 3) and considers only two of them as output candidates. The logic behind this approach is to use information about geolocation to limit the set of output languages to the official ones. Accuracies of different versions of the Lingua model are depicted in Figure 5. Figure 5: Accuracies of the different Lingua models Results presented in Figure 5 show that the more we use additional information about geolocation, the higher the accuracy we achieve. The best performance, an accuracy of 88%, is achieved when we limit our consideration to only official or most common languages (Danish and English in this case). This approach has one disadvantage since it is not able to detect other languages. Further, we combined different versions of the Lingua model (presented in Figure 5) with other ML models (LandID, LangDetect, and FastText) in the form of a voting classifier and measured their accuracies. These results are presented in Figure 6. Figure 6: Voting classifiers based on different Lingua versions Classifiers v1, v2, and v3 are based on respective versions of Lingua implementations. A voting classifier is implemented to additionally confirm non-official language detected by Lingua (the primary model) by other ML models. By comparing Figures 5 and 6, we can notice accuracy increases for versions v1 and v2, but we did not improve the maximum accuracy of 88% that the single Lingua v3 model achieved. On the other hand, the Lingua v3 model alone, as well as combined with other classifiers, cannot detect any languages other than Danish and English. For example, Classifier v2 was able to detect languages for the following examples that are presented in Table 2, where we can see all individual outputs as well as aggregated ones. Table 2: Minority languages detected by Classifier v2 model (de - German, fr - French, es - Spanish, he - Hebrew, en - English) Only one record (name in Hebrew) was identified by the specific characters detection approach, which is the first step in our pipeline, and there was no need to run an ML model for detection in that specific case. Other examples of correctly detected languages for Danish and English languages are presented in Table 3. Table 3: Correctly detected languages (Examples for Danish and English) by Classifier v2 model (en - English, da - Danish, fr - French, no - Norwegian) Since we are dealing with generally short strings, there are some cases when it is extremely difficult to determine the language of the company name, even for humans, like in the following examples (Table 4). Table 4: Examples of input languages that are difficult to detect (da - Danish, en - English) Table 4 shows how company names can simply be abbreviations or a mix of multiple languages (e.g., Copenhagen Business School Handelshøjskolen). Finally, during the analysis, we detected some records where ground-truth labels (manually assigned) were incorrect, and our detection approach identified the correct languages (Table 5). Considering this, the real accuracy values could be even higher than the previously presented. Table 5: Examples of incorrectly labeled languages (ground-truth) in the dataset Conclusion The language identification process is a subfield of NLP that encompasses a group of algorithms to automatically detect natural language from the input text, which is a crucial step for several NLP applications such as machine translation and text classification. This task becomes more difficult as the length of the input string decreases. Even though advanced approaches based on complex neural network architectures exist, considering the accuracy-efficiency trade-off, sometimes it is more suitable to use simpler models that combine preprocessing steps and probabilistic or linear classifiers. In this blog, we presented a custom approach based on an ensemble of ML models for language detection of company names for a specific country. As additional input, we used information about the company's geolocation. We showed how we can achieve higher accuracy by limiting a subset of potential outputs (mainly focusing on the most common languages for the specific country). With this approach, by focusing on official languages, we achieved an accuracy of 88% using a single Lingua model. However, this approach disables the detection of the other (minority) languages. Therefore, we implemented a voting classifier with one primary model (Lingua) and three additional models (LangID, LangDetect, and FastText) to boost the performance when considering a broader subset of languages as output candidates. Accuracy achieved this way was 85%. Also, as part of our detection pipeline, we used a heuristic approach based on specific alphabet detection to increase efficiency when language can be identified solely by detecting specific characters without running the ML model. Finally, during the analysis, we managed to detect some cases where ground-truth labels were false (due to the manual labeling process), where our algorithm managed to detect correct language, so the accuracies reported in this blog could be even higher.   (more…)

Introduction Most of the knowledge and information in the world is stored in the form of plain text. To process such information and enable machines to interpret, understand, and generate human language, the NLP (Natural Language Processing) field of ML encompasses a set of techniques for transforming plain text into numerical values, effectively bridging the gap between human communication and computational systems. That approach enables, among other things, applications such as language translation, sentiment analysis, and voice-activated assistants. Text vectorization is a process within NLP that is responsible for transforming textual data into meaningful numbers that algorithms can work with. In this Tech Bite, we will present the general idea behind text vectorization along with the popular algorithm for creating dense word embeddings, Word2Vec. Further, we will explain its drawbacks, considering its inability to capture morphological structure and work with previously unseen words, and present a different approach in the form of the FastText algorithm that deals with these particular problems. Text Vectorization The essence of the text vectorization process is to convert words or phrases into numerical vectors, possibly capturing their semantic meaning and contextual relationships. There are two different approaches to representing words in a vector space: Distributional representations Distributed representations Distributional representations aim to create vectors where each dimension corresponds to a word or term frequency. They don’t inherently capture word meanings beyond co-occurrence patterns. Popular methods for this approach are One Hot Encoding (OHE), Bag-of-words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF), etc. The results of these approaches are usually easily interpretable, and vectors are characterized by high dimensions and a sparse nature. Example for BoW embeddings for a dataset that contains the sentences: “I like the new movie!”, “I love the weather.” is depicted in Figure 1.1 (example from [1]).   Figure 1.1 Bag-of-Words embeddings It is obvious from Figure 1.1 that the dimension of embeddings in the case of BoW is equal to the size of the vocabulary. Every index in the embedding vectors corresponds with a specific word, and its value represents the number of appearances of that word in the input sentence. The major drawback of distributional representations is their inability to capture semantic meanings of words since they don’t encode relationships between neighboring words or contexts. Also, input representations scale with the size of vocabulary (with mostly sparse nature), which introduces efficiency issues. Distributed representations, on the other hand, are designed to capture semantic meaning in a more nuanced way by mapping words into continuous vector spaces, where each word is represented by a dense vector of real numbers. Unlike distributional representations, vector features of distributed representations are not interpretable and are black boxes (no single value in the array carries any specific meaning, the meaning is distributed throughout the vector). Considering size, this type of vector is usually low dimensional and dense. An example of distributed encoding is depicted in Figure 1.2. using a toy example using words: “King,” “Queen,” “Prince,” “Man,” and “Woman” (here we assumed that respective features represent “Royalty,” “Masculinity,” “Femininity,” and “Age” for demonstration purposes) (example from [1]). Figure 1.2 Hypothetical features to demonstrate word embeddings An interesting feature of distributed representation is that semantically similar words are mapped closer to each other in vector space. This is shown in Figure 1.3, for example, presented in Figure 1.2 (we used just two dimensions, Royalty-Masculinity, for demonstration purposes). Figure 1.3 Mapped words in a hypothetical 2D vector space We can see from Figure 1.3 how “Man,” “King,” and “Prince” have similar values for the “Masculinity” attribute and very different values for “Royalty”. The same thing is with “Queen,” and “Woman”. On the other hand, “Queen,” “Prince,” and “King” share “Royalty” property compared to “Man” and “Woman.” Finally, instances “King” and “Prince” are very close to each other since they are similar in both “Masculinity” and “Royalty.” Representation learned this way has another interesting property where arithmetic operations on word vectors seemed to retain meaning, so valid operations are for example: King - Man + Woman ~= Queen Popular methods for distributed learning approach are Word2Vec, FastText, GloVe, and BERT. It is obvious that distributed representations enable richer and more flexible modeling of language compared to traditional distributional methods. Word2Vec Word2Vec is a representative model for generating word embeddings, which are dense, continuous vector representations of words. The model was developed in 2013 by Mikolov et al. [2] and is based on a shallow neural network trained on large text corpora, and vector representations are extracted from neural network weights. Resulting vectors are such that words with similar meanings or usage patterns end up with similar vector representations, as explained in the previous section. The training process is realized in a self-supervised manner, meaning input and target words are extracted from the same corpus. For example, if we have a sentence like: “The quick brown fox jumps over the lazy dog.“, every target word is considered together with its direct neighborhood using a sliding window approach, which is demonstrated in Figure 1.4 (example from [3]).   Figure 1.4 Example of self-supervised learning using combinations of target and context words This approach allows capturing the context in which the target word appears through its relationship with neighboring words. With this idea, Word2Vec is implemented through one of two models: Continuous Bag of Words (CBoW): This approach uses surrounding context words as input to predict the target word in the output (for an example of the “quick brown fox” sequence, words “quick” and “fox” are used to predict the target word “brown”); Skip-gram: This model works in the reverse direction, predicting the context words given a target word. For instance, the model will use the word “brown” to predict the words “quick” and “fox” for the sentence “quick brown fox.” Neural network architectures for these two models are presented in Figure 1.5.   Figure 1.5 Word2Vec model architectures. The CBoW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word [2] To demonstrate how Word2Vec works, we used the Coronavirus tweets NLP database from Kaggle to form the dataset for training and testing purposes and used Word2Vec models from the free, open-source Python library Gensim [1]. The dataset is loaded and preprocessed in a way that punctuations and numbers were removed, as well as characters including and after “https.” Finally, sentences are parsed to extract separate words. dataset = pd.read_csv('Data/archive/Corona_NLP_train.csv', encoding='latin1') texts = [] for i in range(0,len(dataset)): text = re.sub('[^a-zA-Z]', ' ', dataset['OriginalTweet'][i]) text = text.lower() text = text.split() x = len(text) if text.count('https') == 0 else text.index('https') text = text[: x] text = [t for t in text if not t == 'https'] text = ' '.join(text) texts.append(text) sentences = [line.split() for line in texts] The number of sentences loaded and preprocessed is 41,157. One example of a preprocessed and further tokenized sentence is presented below: Preprocessed: “for corona prevention we should stop to buy things with the cash and should use online payment methods because corona can spread through the notes also we should prefer online shopping from our home it s time to fight against covid govindia indiafightscorona” Tokenized: # Output: ['for', 'corona', 'prevention', 'we', 'should', 'stop', 'to', 'buy', 'things', 'with', 'the', 'cash', 'and', 'should', 'use', 'online', 'payment', 'methods', 'because', 'corona', 'can', 'spread', 'through', 'the', 'notes', 'also', 'we', 'should', 'prefer', 'online', 'shopping', 'from', 'our', 'home', 'it', 's', 'time', 'to', 'fight', 'against', 'covid', 'govindia', 'indiafightscorona'] Further, we import both versions of the Word2Vec model (CBoW and Skip-gram) from gensim.model: from gensim.models import Word2Vec cbow = Word2Vec(sentences, vector_size=300, window=5, sg=0, min_count=5) skip_gram = Word2Vec(sentences, vector_size=300, window=5, sg=1, min_count=5) Used input parameters are: sentences - List of input phrases; vector_size - Size of the word embedding that model would output; window - Maximum distance between the current and predicted word within a sentence sg - Training algorithm (1 - Skip-gram, 0 - CBoW) min_count - Ignores all words with total frequency lower than this The vocabulary size for this particular example is 10,630 and can be tested using the following line of code: voc_cbow = cbow.wv.index_to_key print(f"CBoW Vocabulary length: {len(voc_cbow)}") print() # OR voc_sg = skip_gram.wv.index_to_key print(f"SkipGram Vocabulary length: {len(voc_sg)}") An example of word embedding for the word “coronavirus” can be presented using the line of code: print(cbow.wv['coronavirus']) The output is a numerical 300x1 vector. For the sake of brevity, we will generate a 5x1 embedding vector for the same word (“coronavirus”), which is presented below: cbow.wv['coronavirus'] ---> [2.3701065 3.346616 -0.0336437 -0.69798934 -2.0931847] To demonstrate how similar words are mapped closer to each other in vector space, we will test the similarity between the words “Coronavirus” and “Covid” from one side and “Coronavirus” and “Pakistan”: print(f"Similarity score: {cbow.wv.similarity('coronavirus', 'covid')}") # Output: Similarity score: 0.694911 print(f"Similarity score: {cbow.wv.similarity('coronavirus', 'pakistan')}") # Output: Similarity score: 0.345206 It is evident how the similarity score is twice as big for the combination “Coronavirus” and “Covid” compared to the “Coronavirus” and “Pakistan.” To illustrate this in more detail, we implemented a function to reduce the dimensionality of the vector space to 2D using PCA (Principal Component Analysis) and to plot a subset of words from the vocabulary [1]: import numpy as np import matplotlib.pyplot as plt from sklearn.decomposition import PCA def display_pca_scatterplot(model, words=None, sample=0, model_type=None): if words == None: if sample > 0: words = np.random.choice(list(model.wv.index_to_key), sample) else: words = [word for word in model.wv.index_to_key] word_vectors = np.array([model.wv[w] for w in words]) twodim = PCA().fit_transform(word_vectors)[:, :2] plt.figure(figsize=(6, 6)) plt.scatter(twodim[:, 0], twodim[:, 1], edgecolors='k', c='r') plt.grid() if model_type is not None: plt.title(model_type) for word, (x, y) in zip(words, twodim): plt.text(x + 0.0005, y + 0.0005, word) display_pca_scatterplot(cbow, words=['coronavirus', 'covid', 'virus', 'corona','disease', 'saudiarabia', 'doctor', 'hospital', 'pakistan', 'kenya', 'pay', 'paying', 'paid', 'wages', 'raise', 'bills', 'rent', 'charge'], sample=10, model_type='CBoW') display_pca_scatterplot(skip_gram, words=['coronavirus', 'covid', 'virus', 'corona','disease', 'saudiarabia', 'doctor', 'hospital', 'pakistan', 'kenya', 'pay', 'paying', 'paid', 'wages', 'raise', 'bills', 'rent', 'charge'], sample=10, model_type='SkipGram') plt.show() The output of this analysis is presented in Figure 1.6:   Figure 1.6 2D Vector space plot for subset of words from the vocabulary   It is evident from Figure 1.6 how similar words are mapped closer to each other in the vector space. This result is more clear in the case of Skip-gram for this particular example, where we can distinguish approximately 4 different clusters. Depending on the application, some variant of the Word2Vec performs better than the other. Generally, Skip-gram is a better choice when dealing with a small dataset and the focus is on rare words, while CBoW is a good choice when the data set is huge, and the focus is on frequent words [4]. Even though Word2Vec model outputs decent results, it has particular drawbacks for specific applications: Out of Vocabulary (OOV) words - In this approach, an embedding is created for each word, which means that Word2Vec cannot produce output for new words (the ones that were not present in the vocabulary during the training process); Morphology - Since the model is trained for whole words, it is unable to capture the internal structure of words and to make appropriate generalizations (for example, the model knows nothing about the word “eaten,” even if the word “eat” is present in the vocabulary). In other words, Word2Vec is unable to capture the morphological properties of words, which is particularly important for morphologically rich languages (like German or Czech). FastText To deal with the word embeddings on a more granular level, researchers from Facebook’s AI Research (FAIR) developed the FastText model [5], which is based on Word2Vec with subtle changes. Specifically, FastText can be taken as an extension of the Skip-gram model where subsets of characters in a single word, called character n-grams, are used for word embedding creation. For example, if we are dealing with the word “eating” and n=3, according to FastText, this word can be represented as: “<ea”, “eat”, “ati”, “tin”, “ing”, “ng>” Symbols “<” and “>” are added to mark the beginning and the end of the word. This way the algorithm captures the internal structure of the single word, taking into account the sub-word information like “eat,” even if it is not present in the vocabulary as a separate word. Finally, the whole word embedding can be represented as the sum of the character n-gram vectors.The process of a single training step for one part of the sentence presented in Figure 1.4, “fox jumps over,” is presented in Figure 1.7. Here, “jumps” is the target word used as an input along with its character n-grams, and the words “fox” and “over” are context words used as desired outputs separately.   Figure 1.7 FastText training step for m=1 (window size) and n=3 To show how FastText can be used to improve performance on syntactic word analogy, we compared it with the models from Word2Vec using “The Alchemist’s summary” as in [4] to train the models: “Santiago is a Shepherd who has a recurring dream which is supposedly prophetic. Inspired on learning this, he undertakes a journey to Egypt to discover the meaning of life and fulfill his destiny. During the course of his travels, he learns of his true purpose and meets many characters, including an “Alchemist,” that teach him valuable lessons about achieving his dreams. Santiago sets his sights on obtaining a certain kind of “treasure,” for which he travels to Egypt. The key message is, “when you want something, all the universe conspires in helping you to achieve it.” Towards the final arc, Santiago gets robbed by bandits who end up revealing that the “treasure” he was looking for is buried in the place where his journey began. The end.” In the first step, the data was cleaned by removing special characters and splitted to obtain separate words as tokens. Also, models were initialized as in the previous example: text = re.sub('[^a-zA-Z.]', ' ', text) text = [x.lower().split() for x in text.split('.')] # CBoW (sg=0) cbow = Word2Vec(text, vector_size=300, window=3, sg=0, min_count=1) # Skip-gram (sg=1) skip_gram = Word2Vec(text, vector_size=300, window=3, sg=1, min_count=1) fasttext = FastText(vector_size=300, window=3, min_count=1) fasttext.build_vocab(text) fasttext.train(text, total_examples=len(text), epochs=10) For this purpose, the vector size of 300 is used, along with a window size of 3. The obtained vocabulary size is 85, which is a pretty small set of data. Further, we tested the similarity between words with the same radicals present in the dataset, like “learns” and “learning,” for example. # Similarity between words 'learns' and 'learning' print(f"CBoW: {cbow.wv.similarity('learns', 'learning')}") print(f"SkipGram: {skip_gram.wv.similarity('learns', 'learning')}") print(f"FastText: {fasttext.wv.similarity('learns', 'learning')}") # Output: # CBoW: 0.06936778128147125 # SkipGram: 0.06970427930355072 # FastText: 0.3927953839302063 # Similarity between words 'him' and 'his' print(f"CBoW: {cbow.wv.similarity('him', 'his')}") print(f"SkipGram: {skip_gram.wv.similarity('him', 'his')}") print(f"FastText: {fasttext.wv.similarity('him', 'his')}") # Output: # CBoW: 0.060138553380966187 # SkipGram: 0.06149899959564209 # FastText: 0.18193207681179047 # Similarity between words 'dreams' and 'dreams' print(f"CBoW: {cbow.wv.similarity('dream', 'dreams')}") print(f"SkipGram: {skip_gram.wv.similarity('dream', 'dreams')}") print(f"FastText: {fasttext.wv.similarity('dream', 'dreams')}") # Output: # CBoW: -0.0955636277794838 # SkipGram: -0.09555056691169739 # FastText: 0.5688490271568298 It is evident how FastText has an order of magnitude larger similarity scores when we observe morphologically similar words that were not used in a similar context during the training. The reason for this is that Word2Vec takes these words as totally separate tokens and learns only their context concerning neighboring words. On the other hand, FastText dissects words during the training and can also deduce their morphological structure besides the context of the sequence of words. Also, if we test these models using words that were not present in the vocabulary during the training but had similar internal structure as some words that were present, for example, the word “purposes,” which is not present in the vocabulary, but the word “purpose“ is. In this case, models based on the Word2Vec approach cannot produce any output, while Fast text is able to work in this cases: # Similarity between words 'purpose' and 'purposes' print(f"FastText: {fasttext.wv.similarity('purpose', 'purposes')}") # Output: # FastText: 0.7529796957969666 This solves the OOV (Out Of Vocabulary) problem to some extent. To visually present morphological similarity in 2D space, we again used PCA to reduce vector space and plot diagrams shown in Figure 1.8 (for brevity, we compared only Skip-gram and FastText).   Figure 1.8 2D plot in the vector space for morphologically similar words   The presented examples demonstrate how FastText improves performance when considering morphologically similar words and solves the OOV problem when compared with Word2Vec. On the other hand, FastText generally has degraded performance on semantic analogy tasks and is approximately 1.5 times slower to train than regular Skip-gram due to the added overhead of n-grams [3]. Besides representation learning, the FastText library [6] provides additional functionalities like models for text classification. Conclusion Text vectorization is the process of converting textual data into meaningful numerical representation that can be further used by various ML algorithms to perform tasks such as sentiment analysis, language identification, machine translation, etc. Generally, approaches for creating such representations can be divided into two categories, ones that create distributional, and the other ones that produce distributed representations. Unlike distributional representation, distributed ones are able to capture the semantic meaning of words by considering their context through their direct neighborhood, i.e., surrounding words in the sentence. One of the most popular methods for creating such embeddings is Word2Vec, an algorithm based on a neural network that generates dense and continuous vectors that are able to capture the semantic meaning of words and consequently map them into vector space based on their context, which means that words with similar meaning are mapped close to each other in vector space. An interesting feature of this approach is that it is possible to perform arithmetic operations on vectors, retaining the meaning. On the other hand, Word2Vec is unable to capture the morphological structure of words since it considers every word as a separate token, and therefore, can’t deal with the OOV problem and is not suitable for morphologically rich languages. FastText solves these problems by enhancing the Word2Vec models with sub-word consideration, using character n-grams to build vocabulary. That way, a model can infer representations for words that were not previously seen and is a more suitable model for work with morphologically rich languages such as Arabic, German, and Russian. Also, the FastText library, besides word embedding, provides models for text classification, and it is a lightweight and open-source library. Even though FastText solved the mentioned problems considering previously unseen words and internal structure and is designed to allow quick model iteration and refinement without specialized hardware, it is still a bit slower than base models from Word2Vec and has degraded performance on semantic analogy. Literature [1] A Dummy’s Guide to Word2Vec, Link, [Accessed online, 16.9.2024.] [2] Mikolov, T. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. [3] A Visual Guide to FastText Word Embeddings, Link, [Accessed online, 16.9.2024.] [4] Text Vectorization Algorithms in NLP, Link, [Accessed online, 16.9.2024.] [5] Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5, 135-146. [6] FastText, Link, [Accessed online, 16.9.2024.] "Capturing Morphological Structure in Word Embeddings: Word2Vec vs FastText" Tech Bite was brought to you by Dino Živojević, Data Analyst at Atlantbh. (more…)