It's Difficult to Be Neutral – Human and LLM-based Sentiment Annotation of Patient Comments

Sentiment analysis is an important tool for aggregating patient voices, inorder to provide targeted improvements in healthcare services. A prerequisitefor this is the availability of in-domain data annotated for sentiment. Thisarticle documents an effort to add sentiment annotations to free-text commentsin patient surveys collected by the Norwegian Institute of Public Health(NIPH). However, annotation can be a time-consuming and resource-intensiveprocess, particularly when it requires domain expertise. We therefore alsoevaluate a possible alternative to human annotation, using large languagemodels (LLMs) as annotators. We perform an extensive evaluation of the approachfor two openly available pretrained LLMs for Norwegian, experimenting withdifferent configurations of prompts and in-context learning, comparing theirperformance to human annotators. We find that even for zero-shot runs, modelsperform well above the baseline for binary sentiment, but still cannot competewith human annotators on the full dataset.