Dr Mengdie Zhuang, Lecturer in Data Science, was a co-author of the paper ‘Impersonating the Crowd: Evaluating LLMs’ Ability to Replicate Human Judgment in Misinformation Assessment’, which received the Best Paper award at ACM ICTIR 2025.
The 15th International ACM (Association for Computing Machinery) ICTIR (Innovative Concepts and Theories in Information Retrieval) Conference was held in Padua, Italy in July 2025.
Dr Zhuang’s paper, co-written with Dr David La Barbera (University of Milano-Bicocca, Italy) and Dr Riccardo Lunardi and Dr Kevin Roitero (University of Udine, Italy), looks at whether Large Language Models (LLMs) can effectively evaluate political misinformation statements, and whether prompting them to impersonate specific demographics affects their ability to do this.
Firstly, the authors assessed the agreement between LLM-generated assessments and human judgements on political misinformation statements. Then, they prompted the LLMs with demographic data collected from real crowd workers and instructed them to assess the same political statements previously evaluated.
The paper’s findings suggest that, while some LLMs align moderately with crowd assessments, their impersonation ability remains inconsistent. Moreover, the impersonation ability of LLMs does not uniformly improve accuracy and often reinforces systematic biases, highlighting limitations in attempting to replicate human judgment.
Dr Zhuang said: “LLMs can sound like humans, but that doesn’t mean they can replicate individuals’ behavior in all tasks. Our work shows this mimicry can miss the mark and even amplify bias. I'm glad it prompts reflection on how we use AI responsibly in sensitive areas like fact-checking.”
The ACM ICTIR Conference aims to provide a forum for the presentation and discussion of research related to the foundational aspects of Information Retrieval (IR), including, for example, new or improved models of relevance, ranking, representation, information needs, and evaluation. The conference also welcomes interdisciplinary research that connects information retrieval with other research disciplines that are theoretically motivated.
Read the full paper here.
Find out more about ACM ICTIR here.