Extrinsic evaluation of various word embeddings for text classification tasks

Abstract

We aim to implement and compare the performance of different word embedding methods, including word2vec, GloVe, FastText, ELMo, BERT, GPT-2, and the quantum entanglement-inspired EmbeddingKet from word2ket on selected text classification tasks. Word embeddings are vector representations of words in an n-dimensional hyperspace, intending to capture semantic relationships as features in this embedding space. Most of the comparative analyses on word embeddings use intrinsic evaluation, assessing the structure of the embedding space itself. Here, we focus on extrinsic evaluation, assessing how metrics change with each embedding method in a downstream Natural Language Processing (NLP) task, specifically text classification. We conduct classification on two datasets: StackOverflow60k and CrisisLexT26. StackOverflow60k contains 60,000 posts from the StackOverflow website tagged in categories based on content quality. CrisisLexT26 has disaster tweets collected during 26 crisis events in 2012 and 2013, labeled for relevance and informativeness. We utilized a sequential neural network with two hidden layers to classify texts. The embeddings were evaluated regarding classification performance, efficiency, and resilience. Contextual embeddings, BERT and GPT-2, achieved the highest accuracies among the group, with increased significant computational cost and training time. In terms of resilience, a measure of how training corpus size affects performance, pre-trained embeddings, ELMo and GloVe, have a significant advantage over non-pre-trained methods like EmbeddingKet. Pre-training over a general corpus gave these methods a prior semantic structure that can be utilized when dealing with a smaller dataset. Future work can investigate applications on other downstream NLP tasks like semantic analysis, text summarization, and information retrieval.