Collaborative Emotion Annotation: Assessing the Intersection of Human and AI Performance with GPT Models

Abstract

In this study, we explore emotion detection in text, a complex yet vital aspect of human communication. Our focus is on the formation of an annotated dataset, a task that often presents difficulties due to factors such as reliability, time, and consistency. We propose an alternative approach by employing artificial intelligence (AI) models as potential annotators, or as augmentations to human annotators. Specifically, we utilize ChatGPT, an AI language model developed by OpenAI. We use its latest versions, GPT3.5 and GPT4, to label a Turkish dataset having 8290 terms according to Plutchik's emotion categories, alongside three human annotators. We conduct experiments to assess the AI's annotation capabilities both independently and in conjunction with human annotators. We measure inter-rater agreement using Cohen's Kappa, Fleiss Kappa, and percent agreement metrics across varying emotion categorizations- eight, four, and binary. Particularly, when we filtered out the terms where the AI models were indecisive, it was found that including AI models in the annotation process was successful in increasing inter-annotator agreement. Our findings suggest that, the integration of AI models in the emotion annotation process holds the potential to enhance efficiency, reduce the time of lexicon development and thereby advance the field of emotion/sentiment analysis. Copyright © 2023 by SCITEPRESS - Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)