Part of speech (POS) tagging (assigning the word class to a word) and lemmatization (converting the word to its base form) are core techniques used in natural language processing (NLP) and text mining. Therefore the performance of POS taggers and lemmatizers is important and can have a significant impact on the success of NLP and text mining tasks where these operations are used. Over the years, many POS tagging and lemmatization algorithms have been developed, which are publicly available in open source natural language processing libraries. Open source libraries provide the researchers with accessible NLP tools, but further work on evaluating these tools is still needed, since there is insufficient documentation on the comparison of different POS tagging and lemmatization tools in terms of their effectiveness.

For our research we conducted an experimental evaluation of several publicly available open source POS taggers and lemmatizers on a number of annotated evaluation corpora. Several POS taggers were evaluated and the influence of various factors, such as training corpus length, training corpus genre and the use of pretrained models has been tested. Results have shown non-linear dependence between the size of the training set and the accuracy of the POS taggers, domain-specificity of POS taggers and that the use of pretrained models is not always appropriate. In lemmatizer evaluation, we have shown that preprocessing steps, such as conversion of text to lowercase, have a significant influence on the performance of some lemmatizers. In extrinsic evaluation, where the performance of the tool is measured indirectly (in the context of a larger text mining task), we tested the POS taggers' influence on the lemmatization and gender classification tasks, while lemmatizers and stemmers were evaluated through text categorization.

The experiments were conducted in the TextFlows platform, which is a cloud-based web application for composition, execution and sharing of interactive text mining workflows. The POS tagger and lemmatizer implementations in this online platform enable transparent and reproducible results, reusability of the workflow components for new tasks and provide precomposed workflows for algorithm comparison and evaluation, enabling future algorithms to be tested in the same environment.