Linguistic Inaccuracies in Chatbots: A Comparative Analysis of Lexical and Grammatical Features

Author(s) : Gul E Raana Qureshi, Rashid Mahmood, Dr Samina Ali Asghar

Abstract:

Chatbots are now ubiquitous across fields like education, customer support, and healthcare. Despite significant advances in both artificial intelligence (AI) and natural language processing (NLP), linguistic inaccuracies: somehow termed “slip-ups”, remain present in the answers provided by Chatbots. These errors can be divided into lexical and grammatical, which impair the user experience and the reliability of the Chatbots. The current study analyses the linguistic errors committed by five Chatbots— ChatGPT, Copilot, Google Gemini, Perplexity, and Zapier Central—in eight different categories: adjectives, verbs, articles, and vocabulary range. Whether attributed to limitations in the training data of Chatbots, a lack of lexical embeddings, or a simplistic model of syntactic processing, previous studies have identified these factors as the key causes of linguistic errors. Nonetheless, transformer-based architectures including GPT and BERT have demonstrated improvements but continue to suffer from grammar consistency and contextual correctness issues, highly pronounced in multilingual and culturally rich applications. This study utilised a comparative approach analyzing responses of 200 prompts across eight linguistic categories. Errors were categorised as lexical or grammatical and analyzed according to their frequency, severity, and distribution. We used statistical methods such as ANOVA to find significant differences in chatbot performance in relation to features. The results indicate that the biggest challenges were seen with vocabulary range and adjectives, demonstrating these students' restrictions in lexical range and descriptiveness. However, question words had the lowest number of errors, which suggests strength in forming interrogative structures. Vocabulary range, verbs, and articles showed significant variation depending on the chatbot used, with statistically significant ANOVA results, and most features presented stable results with different Chatbots like orthography and question words. With the growing severity, the differences in the types of errors became more pronounced, we did take note of this difference between Google Gemini, that had a smaller ratio of major errors vs ChatGPT and Perplexity. Overall, this study highlights the importance of language-specific enhancements, particularly in vocabulary variety and grammatical alignment. For improved understanding, utilization of state-of-the-art NLP technologies, varied training data, programming live audit/error correction features may be required.