When an international client asked us to automate language detection by efficiently filtering batches of short text using Machine Learning, we ended up improving the quality of data and speeding up the process. This was achieved through a combination of existing ML models in a voting classifier manner employed to perform language detection of company names for a specific country, with geolocation information efficiently used as an additional input.
We improved accuracy from 75% to 88% with a single model for common local languages. With more languages added, the model still achieved 85% accuracy. Automated language detection made it easier to clean data and spot mistakes. This also simplified other tasks, like translation, making the entire process faster and more efficient.
Introduction
Most of the data and knowledge today is stored in the form of plain text using natural language. Natural Language Processing (NLP) is a subfield of Machine Learning (ML) that enables interaction between humans and machines by adapting natural language to data forms that are interpretable by computers. In the multilingual environment, one of the crucial steps for several NLP tasks, such as machine translation and text classification, is determining the natural language in which a text is written. A subfield of NLP responsible for such a task is formally called Language Identification (abbr. LangID) or Language Detection.
In the modern era, with the increasing use of social media, there is a significant amount of user-generated text content containing a small number of characters. Short texts are generally more challenging for language detection models. Even if an ML model achieves good performance on long text that contains multiple sentences, it does not mean it can properly generalize to shorter sequences, mostly due to the lack of context. On the other hand, in production, especially in real-time use cases, inference time is an important constraint besides the accuracy of the given model. The efficiency constraints sometimes prevent the use of complex ML models. In this blog, we use an ensemble of existing ML models in a voting classifier manner to perform language detection of company names for a specific country, effectively using geolocation information as an additional input.
Language Identification
Language identification is the process of automatically determining the language of a given text. This problem can be solved with a primitive approach by using complete vocabularies of all considered languages and performing word matching. However, this is pretty inefficient and not scalable with a vast number of languages. Also, this approach is not robust in terms of inflections, compound, and misspelled words. Some traditional approaches are also based on character and word N-gram frequency analysis, assuming that specific character or word sequences are language-specific and identifying the language based on N-gram distributions. A more advanced approach is based on ML models, like the Bayesian Classifier of Support Vector Machines (SVM), which can be leveraged to perform language classification tasks on preprocessed and vectorized text. Finally, the most advanced and complex models are based on deep learning techniques, including Recurrent and Convolutional Neural Networks (RNNs and CNNs) and Transformers.
Even though models based on RNNs (e.g., LSTMs and GRUs), especially Transformers, are designed to capture sequential dependencies in text, they have more complex architecture and aim to capture context from longer sequences. Therefore, they don’t seem suitable for use cases that deal with short strings and further introduce efficiency loss. Considering that, one of the most suitable pipelines is to preprocess input text (which implies data cleaning and vectorization) and then apply an ML model (e.g. a probabilistic one like Multinomial Naive Bayes Classifier). Since short strings for specific use cases can contain only one word, capturing the internal structure of words in the process of text vectorization by using character N-grams (like the FastText vectorization approach does) is desirable. The common pipeline of such classifiers is depicted in Figure 1.
Figure 1: Example of the common pipeline for language identification
Probabilistic models such as the Multinomial Naive Bayes Classifier are suitable for use cases that imply text processing since they perform better when applied to discrete data like word counts in documents. Besides Multinomial Naive Bayes Classifiers, linear classifiers like SVM are also applicable, if input features are properly derived from the input text. The approaches based on linear or probabilistic classifiers are more efficient than models based on neural network architectures and demand less training data, making them more suitable for this particular use case when we deal with very short strings.
There are several open-source ML models for language identification available online for programming languages like Python. In the following table (Table 1) we provide an overview of such models that we used in our use case:
Table 1: Overview of open-source language detection models
All these models use the pipeline that encompasses the preprocessing step, and after that, probabilistic or linear ML model based on N-gram distributions. Individual outputs of the separate classifiers do not necessarily achieve great performance, especially with very short strings used as inputs and many considered languages as outputs. Therefore, with additional information for specific use cases, they can be implemented together as a voting classifier to achieve greater accuracy. Also, some additional heuristic steps can be leveraged to narrow down the set of potential outputs.
Problem Definition and Approach
An interesting use case of language detection is the application of these models together to detect the language of company names for specific countries. The complexity of this problem is two-fold: (1) Company names are generally very short strings with a couple of words (sometimes even just one); (2) Company names could simply be acronyms that don’t necessarily belong to the specific language. The problem can be defined as follows: For a specific geographic region (e.g., a country) and a list of company names, the language in which a particular name is provided should be identified.
Generally, language identification models face an accuracy decrease with the (1) Decrease of input text lengths and (2) Increase of potential languages considered as output candidates. The general idea is to combine different models for language detection as a voting classifier and to aggregate their individual outputs. Additional information that can be used in the implementation is geolocation, which can be leveraged to narrow down the set of considered languages by prioritizing official or most common languages for a specific country. Also, the additional heuristic step can be implemented, as in the Lingua model, which breaks down the detection pipeline into two steps:
- A heuristic approach uses a rule-based engine that determines the input text’s alphabet and searches for unique characters in one or more languages. If exactly one language can be detected this way, there is no need for a probabilistic model. In any case, this step can be used to filter out languages that do not satisfy alphabetic constraints;
- Probabilistic detection based on the ML model can be executed in the second step if the heuristics approach fails to detect the language unambiguously.
The complete pipeline of such an approach is presented in Figure 2.
Figure 2: Language detection pipeline
As previously explained, input strings that represent the company’s name are first tested for specific characters that could determine output language without running an ML model. This first part of the pipeline could potentially reduce execution time. If language cannot be unambiguously determined this way, then a voting classifier encompassing 4 individual models is leveraged. Individual outputs from separate classifiers are further aggregated using a rule-based function that prioritizes local (official) language if consensus between classifiers is not reached.
Results and examples
This approach was implemented and tested on a dataset representing the Register of Companies for Denmark subset. The dataset was manually labeled for ground-truth values so performance metrics could be further calculated. The distribution of languages in the test dataset is depicted in Figure 3.
Figure 3: Language distribution in the test dataset
The complete list of languages present in the dataset is following:
- Danish (75%)
- English (22%)
- German (1%)
- French (< 1%)
- Norwegian (< 1%)
- Spanish (< 1%)
- Italian (< 1%)
- Hebrew (< 1%)
- Polish (< 1%)
Danish is the dominant language, followed by English (these two languages combined represent 97% of the dataset).
The first step is to test every model individually without any additional information about geolocation to establish a baseline accuracy. In Figure 4, individual accuracies for respective models are presented:
Figure 4: Individual accuracies without any geographic information
Lingua achieves the best performance on the test dataset with 61% accuracy. Since these are the baseline metrics without any constraints, we can further use additional information about geolocation to increase accuracy for the specific country. Generally, the accuracy of the specific model can be increased by decreasing the number of considered languages for output candidates. Since our dataset is based on company names in Denmark (which is a priori input), we can use this information to limit the set of outputs. Therefore, we tested several versions of the Lingua model:
- Lingua v0 – Baseline version of the Lingua model that considers all available languages (the same model that was presented in Figure 4);
- Lingua v1 – Version of the Lingua model that considers only languages that are present in the dataset (Figure 3) as output candidates;
- Lingua v2 – This Lingua version excludes the Norwegian language from consideration since most of the false predictions are between Danish and Norwegian due to their similarity;
- Lingua v3 – This final version takes advantage of the fact that Danish and English languages cover 97% of the dataset (Figure 3) and considers only two of them as output candidates. The logic behind this approach is to use information about geolocation to limit the set of output languages to the official ones.
Accuracies of different versions of the Lingua model are depicted in Figure 5.
Figure 5: Accuracies of the different Lingua models
Results presented in Figure 5 show that the more we use additional information about geolocation, the higher the accuracy we achieve. The best performance, an accuracy of 88%, is achieved when we limit our consideration to only official or most common languages (Danish and English in this case). This approach has one disadvantage since it is not able to detect other languages. Further, we combined different versions of the Lingua model (presented in Figure 5) with other ML models (LandID, LangDetect, and FastText) in the form of a voting classifier and measured their accuracies. These results are presented in Figure 6.
Figure 6: Voting classifiers based on different Lingua versions
Classifiers v1, v2, and v3 are based on respective versions of Lingua implementations. A voting classifier is implemented to additionally confirm non-official language detected by Lingua (the primary model) by other ML models. By comparing Figures 5 and 6, we can notice accuracy increases for versions v1 and v2, but we did not improve the maximum accuracy of 88% that the single Lingua v3 model achieved. On the other hand, the Lingua v3 model alone, as well as combined with other classifiers, cannot detect any languages other than Danish and English. For example, Classifier v2 was able to detect languages for the following examples that are presented in Table 2, where we can see all individual outputs as well as aggregated ones.
Table 2: Minority languages detected by Classifier v2 model (de – German, fr – French, es – Spanish, he – Hebrew, en – English)
Only one record (name in Hebrew) was identified by the specific characters detection approach, which is the first step in our pipeline, and there was no need to run an ML model for detection in that specific case. Other examples of correctly detected languages for Danish and English languages are presented in Table 3.
Table 3: Correctly detected languages (Examples for Danish and English) by Classifier v2 model (en – English, da – Danish, fr – French, no – Norwegian)
Since we are dealing with generally short strings, there are some cases when it is extremely difficult to determine the language of the company name, even for humans, like in the following examples (Table 4).
Table 4: Examples of input languages that are difficult to detect (da – Danish, en – English)
Table 4 shows how company names can simply be abbreviations or a mix of multiple languages (e.g., Copenhagen Business School Handelshøjskolen).
Finally, during the analysis, we detected some records where ground-truth labels (manually assigned) were incorrect, and our detection approach identified the correct languages (Table 5). Considering this, the real accuracy values could be even higher than the previously presented.
Table 5: Examples of incorrectly labeled languages (ground-truth) in the dataset
Conclusion
The language identification process is a subfield of NLP that encompasses a group of algorithms to automatically detect natural language from the input text, which is a crucial step for several NLP applications such as machine translation and text classification. This task becomes more difficult as the length of the input string decreases. Even though advanced approaches based on complex neural network architectures exist, considering the accuracy-efficiency trade-off, sometimes it is more suitable to use simpler models that combine preprocessing steps and probabilistic or linear classifiers.
In this blog, we presented a custom approach based on an ensemble of ML models for language detection of company names for a specific country. As additional input, we used information about the company’s geolocation. We showed how we can achieve higher accuracy by limiting a subset of potential outputs (mainly focusing on the most common languages for the specific country). With this approach, by focusing on official languages, we achieved an accuracy of 88% using a single Lingua model. However, this approach disables the detection of the other (minority) languages. Therefore, we implemented a voting classifier with one primary model (Lingua) and three additional models (LangID, LangDetect, and FastText) to boost the performance when considering a broader subset of languages as output candidates. Accuracy achieved this way was 85%. Also, as part of our detection pipeline, we used a heuristic approach based on specific alphabet detection to increase efficiency when language can be identified solely by detecting specific characters without running the ML model. Finally, during the analysis, we managed to detect some cases where ground-truth labels were false (due to the manual labeling process), where our algorithm managed to detect correct language, so the accuracies reported in this blog could be even higher.
