Introduction

Most of the knowledge and information in the world is stored in the form of plain text. To process such information and enable machines to interpret, understand, and generate human language, the NLP (Natural Language Processing) field of ML encompasses a set of techniques for transforming plain text into numerical values, effectively bridging the gap between human communication and computational systems. That approach enables, among other things, applications such as language translation, sentiment analysis, and voice-activated assistants.

Text vectorization is a process within NLP that is responsible for transforming textual data into meaningful numbers that algorithms can work with. In this Tech Bite, we will present the general idea behind text vectorization along with the popular algorithm for creating dense word embeddings, Word2Vec. Further, we will explain its drawbacks, considering its inability to capture morphological structure and work with previously unseen words, and present a different approach in the form of the FastText algorithm that deals with these particular problems.


Text Vectorization

The essence of the text vectorization process is to convert words or phrases into numerical vectors, possibly capturing their semantic meaning and contextual relationships. There are two different approaches to representing words in a vector space:

  • Distributional representations
  • Distributed representations

Distributional representations aim to create vectors where each dimension corresponds to a word or term frequency. They don’t inherently capture word meanings beyond co-occurrence patterns. Popular methods for this approach are One Hot Encoding (OHE), Bag-of-words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF), etc. The results of these approaches are usually easily interpretable, and vectors are characterized by high dimensions and a sparse nature.

Example for BoW embeddings for a dataset that contains the sentences: “I like the new movie!”, “I love the weather.” is depicted in Figure 1.1 (example from [1]).

 

Figure 1.1 Bag-of-Words embeddings

Figure 1.1 Bag-of-Words embeddings

It is obvious from Figure 1.1 that the dimension of embeddings in the case of BoW is equal to the size of the vocabulary. Every index in the embedding vectors corresponds with a specific word, and its value represents the number of appearances of that word in the input sentence. The major drawback of distributional representations is their inability to capture semantic meanings of words since they don’t encode relationships between neighboring words or contexts. Also, input representations scale with the size of vocabulary (with mostly sparse nature), which introduces efficiency issues.

Distributed representations, on the other hand, are designed to capture semantic meaning in a more nuanced way by mapping words into continuous vector spaces, where each word is represented by a dense vector of real numbers. Unlike distributional representations, vector features of distributed representations are not interpretable and are black boxes (no single value in the array carries any specific meaning, the meaning is distributed throughout the vector). Considering size, this type of vector is usually low dimensional and dense. An example of distributed encoding is depicted in Figure 1.2. using a toy example using words: “King,” “Queen,” “Prince,” “Man,” and “Woman” (here we assumed that respective features represent “Royalty,” “Masculinity,” “Femininity,” and “Age” for demonstration purposes) (example from [1]).

Figure 1.2 Hypothetical features to demonstrate word embeddings

Figure 1.2 Hypothetical features to demonstrate word embeddings


An interesting feature of distributed representation is that semantically similar words are mapped closer to each other in vector space. This is shown in Figure 1.3, for example, presented in Figure 1.2 (we used just two dimensions, Royalty–Masculinity, for demonstration purposes).

Figure 1.3 Mapped words in a hypothetical 2D vector space

Figure 1.3 Mapped words in a hypothetical 2D vector space


We can see from Figure 1.3 how “Man,” “King,” and “Prince” have similar values for the “Masculinity” attribute and very different values for “Royalty”. The same thing is with “Queen,” and “Woman”. On the other hand, “Queen,” “Prince,” and “King” share “Royalty” property compared to “Man” and “Woman.” Finally, instances “King” and “Prince” are very close to each other since they are similar in both “Masculinity” and “Royalty.” Representation learned this way has another interesting property where arithmetic operations on word vectors seemed to retain meaning, so valid operations are for example:

King – Man + Woman ~= Queen

Popular methods for distributed learning approach are Word2Vec, FastText, GloVe, and BERT. It is obvious that distributed representations enable richer and more flexible modeling of language compared to traditional distributional methods.


Word2Vec

Word2Vec is a representative model for generating word embeddings, which are dense, continuous vector representations of words. The model was developed in 2013 by Mikolov et al. [2] and is based on a shallow neural network trained on large text corpora, and vector representations are extracted from neural network weights. Resulting vectors are such that words with similar meanings or usage patterns end up with similar vector representations, as explained in the previous section. The training process is realized in a self-supervised manner, meaning input and target words are extracted from the same corpus. For example, if we have a sentence like: “The quick brown fox jumps over the lazy dog.“, every target word is considered together with its direct neighborhood using a sliding window approach, which is demonstrated in Figure 1.4 (example from [3]).

 

Figure 1.4 Example of self-supervised learning using combinations of target and context words

Figure 1.4 Example of self-supervised learning using combinations of target and context words


This approach allows capturing the context in which the target word appears through its relationship with neighboring words. With this idea, Word2Vec is implemented through one of two models:

  • Continuous Bag of Words (CBoW): This approach uses surrounding context words as input to predict the target word in the output (for an example of the “quick brown fox” sequence, words “quick” and “fox” are used to predict the target word “brown”);
  • Skip-gram: This model works in the reverse direction, predicting the context words given a target word. For instance, the model will use the word “brown” to predict the words “quick” and “fox” for the sentence “quick brown fox.”

Neural network architectures for these two models are presented in Figure 1.5.

 

Figure 1.5 Word2Vec model architectures. The CBoW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word [2]

Figure 1.5 Word2Vec model architectures. The CBoW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word [2]


To demonstrate how Word2Vec works, we used the Coronavirus tweets NLP database from Kaggle to form the dataset for training and testing purposes and used Word2Vec models from the free, open-source Python library Gensim [1]. The dataset is loaded and preprocessed in a way that punctuations and numbers were removed, as well as characters including and after “https.” Finally, sentences are parsed to extract separate words.

dataset = pd.read_csv('Data/archive/Corona_NLP_train.csv', encoding='latin1')

texts = []

for i in range(0,len(dataset)):

    text = re.sub('[^a-zA-Z]', ' ', dataset['OriginalTweet'][i])
    text = text.lower()
    text = text.split()
    x = len(text) if text.count('https') == 0 else text.index('https')
    text = text[: x]
    text = [t for t in text if not t == 'https']
    text = ' '.join(text)
    texts.append(text)

sentences = [line.split() for line in texts]

The number of sentences loaded and preprocessed is 41,157. One example of a preprocessed and further tokenized sentence is presented below:

Preprocessed:

“for corona prevention we should stop to buy things with the cash and should use online payment methods because corona can spread through the notes also we should prefer online shopping from our home it s time to fight against covid govindia indiafightscorona”

Tokenized:

# Output:
['for', 'corona', 'prevention', 'we', 'should', 'stop', 'to', 'buy', 'things', 'with', 'the', 'cash', 'and', 'should', 'use', 'online', 'payment', 'methods', 'because', 'corona', 'can', 'spread', 'through', 'the', 'notes', 'also', 'we', 'should', 'prefer', 'online', 'shopping', 'from', 'our', 'home', 'it', 's', 'time', 'to', 'fight', 'against', 'covid', 'govindia', 'indiafightscorona']

Further, we import both versions of the Word2Vec model (CBoW and Skip-gram) from gensim.model:

from gensim.models import Word2Vec

cbow = Word2Vec(sentences, vector_size=300, window=5, sg=0, min_count=5)
skip_gram = Word2Vec(sentences, vector_size=300, window=5, sg=1, min_count=5)


Used input parameters are:

  • sentences – List of input phrases;
  • vector_size – Size of the word embedding that model would output;
  • window – Maximum distance between the current and predicted word within a sentence
  • sg – Training algorithm (1 – Skip-gram, 0 – CBoW)
  • min_count – Ignores all words with total frequency lower than this

The vocabulary size for this particular example is 10,630 and can be tested using the following line of code:

voc_cbow = cbow.wv.index_to_key
print(f"CBoW Vocabulary length: {len(voc_cbow)}")
print()

# OR

voc_sg = skip_gram.wv.index_to_key
print(f"SkipGram Vocabulary length: {len(voc_sg)}")

An example of word embedding for the word “coronavirus” can be presented using the line of code:

print(cbow.wv['coronavirus'])

The output is a numerical 300×1 vector. For the sake of brevity, we will generate a 5×1 embedding vector for the same word (“coronavirus”), which is presented below:

cbow.wv['coronavirus'] ---> 
[2.3701065 3.346616 -0.0336437 -0.69798934 -2.0931847]

To demonstrate how similar words are mapped closer to each other in vector space, we will test the similarity between the words “Coronavirus” and “Covid” from one side and “Coronavirus” and “Pakistan”:

print(f"Similarity score: {cbow.wv.similarity('coronavirus', 'covid')}")
# Output: Similarity score: 0.694911

print(f"Similarity score: {cbow.wv.similarity('coronavirus', 'pakistan')}")
# Output: Similarity score: 0.345206

It is evident how the similarity score is twice as big for the combination “Coronavirus” and “Covid” compared to the “Coronavirus” and “Pakistan.”

To illustrate this in more detail, we implemented a function to reduce the dimensionality of the vector space to 2D using PCA (Principal Component Analysis) and to plot a subset of words from the vocabulary [1]:

import numpy as np
import matplotlib.pyplot as plt

from sklearn.decomposition import PCA

def display_pca_scatterplot(model, words=None, sample=0, model_type=None):
    if words == None:
        if sample > 0:
            words = np.random.choice(list(model.wv.index_to_key), sample)
        else:
            words = [word for word in model.wv.index_to_key]

    word_vectors = np.array([model.wv[w] for w in words])

    twodim = PCA().fit_transform(word_vectors)[:, :2]

    plt.figure(figsize=(6, 6))
    plt.scatter(twodim[:, 0], twodim[:, 1], edgecolors='k', c='r')
    plt.grid()
    if model_type is not None:
        plt.title(model_type)
    for word, (x, y) in zip(words, twodim):
        plt.text(x + 0.0005, y + 0.0005, word)


display_pca_scatterplot(cbow, words=['coronavirus', 'covid', 'virus', 'corona','disease', 'saudiarabia', 'doctor', 'hospital', 'pakistan', 'kenya',
'pay', 'paying', 'paid', 'wages', 'raise', 'bills', 'rent', 'charge'], sample=10, model_type='CBoW')
display_pca_scatterplot(skip_gram, words=['coronavirus', 'covid', 'virus', 'corona','disease', 'saudiarabia', 'doctor', 'hospital', 'pakistan', 'kenya',
'pay', 'paying', 'paid', 'wages', 'raise', 'bills', 'rent', 'charge'], sample=10, model_type='SkipGram')

plt.show()


The output of this analysis is presented in Figure 1.6:

 

Figure 1.6 2D Vector space plot for subset of words from the vocabulary

Figure 1.6 2D Vector space plot for subset of words from the vocabulary

 

It is evident from Figure 1.6 how similar words are mapped closer to each other in the vector space. This result is more clear in the case of Skip-gram for this particular example, where we can distinguish approximately 4 different clusters. Depending on the application, some variant of the Word2Vec performs better than the other. Generally, Skip-gram is a better choice when dealing with a small dataset and the focus is on rare words, while CBoW is a good choice when the data set is huge, and the focus is on frequent words [4].

Even though Word2Vec model outputs decent results, it has particular drawbacks for specific applications:

  • Out of Vocabulary (OOV) words – In this approach, an embedding is created for each word, which means that Word2Vec cannot produce output for new words (the ones that were not present in the vocabulary during the training process);
  • Morphology – Since the model is trained for whole words, it is unable to capture the internal structure of words and to make appropriate generalizations (for example, the model knows nothing about the word “eaten,” even if the word “eat” is present in the vocabulary). In other words, Word2Vec is unable to capture the morphological properties of words, which is particularly important for morphologically rich languages (like German or Czech).


FastText

To deal with the word embeddings on a more granular level, researchers from Facebook’s AI Research (FAIR) developed the FastText model [5], which is based on Word2Vec with subtle changes. Specifically, FastText can be taken as an extension of the Skip-gram model where subsets of characters in a single word, called character n-grams, are used for word embedding creation. For example, if we are dealing with the word “eating” and n=3, according to FastText, this word can be represented as:

“<ea”, “eat”, “ati”, “tin”, “ing”, “ng>”

Symbols “<” and “>” are added to mark the beginning and the end of the word. This way the algorithm captures the internal structure of the single word, taking into account the sub-word information like “eat,” even if it is not present in the vocabulary as a separate word. Finally, the whole word embedding can be represented as the sum of the character n-gram vectors.The process of a single training step for one part of the sentence presented in Figure 1.4, “fox jumps over,” is presented in Figure 1.7. Here, “jumps” is the target word used as an input along with its character n-grams, and the words “fox” and “over” are context words used as desired outputs separately.

 

Figure 1.7 FastText training step for m=1 (window size) and n=3

Figure 1.7 FastText training step for m=1 (window size) and n=3


To show how FastText can be used to improve performance on syntactic word analogy, we compared it with the models from Word2Vec using “The Alchemist’s summary” as in [4] to train the models:

“Santiago is a Shepherd who has a recurring dream which is supposedly prophetic. Inspired on learning this, he undertakes a journey to Egypt to discover the meaning of life and fulfill his destiny. During the course of his travels, he learns of his true purpose and meets many characters, including an “Alchemist,” that teach him valuable lessons about achieving his dreams. Santiago sets his sights on obtaining a certain kind of “treasure,” for which he travels to Egypt. The key message is, “when you want something, all the universe conspires in helping you to achieve it.” Towards the final arc, Santiago gets robbed by bandits who end up revealing that the “treasure” he was looking for is buried in the place where his journey began. The end.”

In the first step, the data was cleaned by removing special characters and splitted to obtain separate words as tokens. Also, models were initialized as in the previous example:

text = re.sub('[^a-zA-Z.]', ' ', text)
text = [x.lower().split() for x in text.split('.')]

# CBoW (sg=0)
cbow = Word2Vec(text, vector_size=300, window=3, sg=0, min_count=1)

# Skip-gram (sg=1)
skip_gram = Word2Vec(text, vector_size=300, window=3, sg=1, min_count=1)

fasttext = FastText(vector_size=300, window=3, min_count=1)
fasttext.build_vocab(text)
fasttext.train(text, total_examples=len(text), epochs=10)

For this purpose, the vector size of 300 is used, along with a window size of 3. The obtained vocabulary size is 85, which is a pretty small set of data. Further, we tested the similarity between words with the same radicals present in the dataset, like “learns” and “learning,” for example.

# Similarity between words 'learns' and 'learning'
print(f"CBoW: {cbow.wv.similarity('learns', 'learning')}")
print(f"SkipGram: {skip_gram.wv.similarity('learns', 'learning')}")
print(f"FastText: {fasttext.wv.similarity('learns', 'learning')}")

# Output:
# CBoW: 0.06936778128147125
# SkipGram: 0.06970427930355072
# FastText: 0.3927953839302063


# Similarity between words 'him' and 'his'
print(f"CBoW: {cbow.wv.similarity('him', 'his')}")
print(f"SkipGram: {skip_gram.wv.similarity('him', 'his')}")
print(f"FastText: {fasttext.wv.similarity('him', 'his')}")

# Output:
# CBoW: 0.060138553380966187
# SkipGram: 0.06149899959564209
# FastText: 0.18193207681179047


# Similarity between words 'dreams' and 'dreams'
print(f"CBoW: {cbow.wv.similarity('dream', 'dreams')}")
print(f"SkipGram: {skip_gram.wv.similarity('dream', 'dreams')}")
print(f"FastText: {fasttext.wv.similarity('dream', 'dreams')}")

# Output:
# CBoW: -0.0955636277794838
# SkipGram: -0.09555056691169739
# FastText: 0.5688490271568298

It is evident how FastText has an order of magnitude larger similarity scores when we observe morphologically similar words that were not used in a similar context during the training. The reason for this is that Word2Vec takes these words as totally separate tokens and learns only their context concerning neighboring words. On the other hand, FastText dissects words during the training and can also deduce their morphological structure besides the context of the sequence of words. Also, if we test these models using words that were not present in the vocabulary during the training but had similar internal structure as some words that were present, for example, the word “purposes,” which is not present in the vocabulary, but the word “purpose“ is. In this case, models based on the Word2Vec approach cannot produce any output, while Fast text is able to work in this cases:

# Similarity between words 'purpose' and 'purposes'
print(f"FastText: {fasttext.wv.similarity('purpose', 'purposes')}")

# Output:
# FastText: 0.7529796957969666

This solves the OOV (Out Of Vocabulary) problem to some extent. To visually present morphological similarity in 2D space, we again used PCA to reduce vector space and plot diagrams shown in Figure 1.8 (for brevity, we compared only Skip-gram and FastText).

 

Figure 1.8 2D plot in the vector space for morphologically similar words

Figure 1.8 2D plot in the vector space for morphologically similar words

 

The presented examples demonstrate how FastText improves performance when considering morphologically similar words and solves the OOV problem when compared with Word2Vec. On the other hand, FastText generally has degraded performance on semantic analogy tasks and is approximately 1.5 times slower to train than regular Skip-gram due to the added overhead of n-grams [3].

Besides representation learning, the FastText library [6] provides additional functionalities like models for text classification.


Conclusion

Text vectorization is the process of converting textual data into meaningful numerical representation that can be further used by various ML algorithms to perform tasks such as sentiment analysis, language identification, machine translation, etc. Generally, approaches for creating such representations can be divided into two categories, ones that create distributional, and the other ones that produce distributed representations. Unlike distributional representation, distributed ones are able to capture the semantic meaning of words by considering their context through their direct neighborhood, i.e., surrounding words in the sentence.

One of the most popular methods for creating such embeddings is Word2Vec, an algorithm based on a neural network that generates dense and continuous vectors that are able to capture the semantic meaning of words and consequently map them into vector space based on their context, which means that words with similar meaning are mapped close to each other in vector space. An interesting feature of this approach is that it is possible to perform arithmetic operations on vectors, retaining the meaning. On the other hand, Word2Vec is unable to capture the morphological structure of words since it considers every word as a separate token, and therefore, can’t deal with the OOV problem and is not suitable for morphologically rich languages.

FastText solves these problems by enhancing the Word2Vec models with sub-word consideration, using character n-grams to build vocabulary. That way, a model can infer representations for words that were not previously seen and is a more suitable model for work with morphologically rich languages such as Arabic, German, and Russian. Also, the FastText library, besides word embedding, provides models for text classification, and it is a lightweight and open-source library. Even though FastText solved the mentioned problems considering previously unseen words and internal structure and is designed to allow quick model iteration and refinement without specialized hardware, it is still a bit slower than base models from Word2Vec and has degraded performance on semantic analogy.


Literature

  • [1] A Dummy’s Guide to Word2Vec, Link, [Accessed online, 16.9.2024.]
  • [2] Mikolov, T. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  • [3] A Visual Guide to FastText Word Embeddings, Link, [Accessed online, 16.9.2024.]
  • [4] Text Vectorization Algorithms in NLP, Link, [Accessed online, 16.9.2024.]
  • [5] Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5, 135-146.
  • [6] FastText, Link, [Accessed online, 16.9.2024.]

“Capturing Morphological Structure in Word Embeddings: Word2Vec vs FastText” Tech Bite was brought to you by Dino Živojević, Data Analyst at Atlantbh.

Tech Bites are tips, tricks, snippets or explanations about various programming technologies and paradigms, which can help engineers with their everyday job.

Leave a comment

Your email address will not be published. Required fields are marked *