Evaluating the computational reliability of ChatGPT in calculating intercoder reliability in content analysis: Evidence from simulated data

Abstract

The increasing integration of large language models (LLMs) into the research workflow has raised important questions regarding their reliability in performing statistical analyses. While prior studies have explored the use of LLMs in text classification and qualitative coding, little is known about their accuracy in computing core statistical metrics used in content analysis. This study addresses this gap by systematically evaluating the performance of ChatGPT in calculating percentage agreement, contingency tables, and Cohen’s Kappa. Using a series of controlled simulations, we varied key parameters including sample size, number of categories, distribution balance, and levels of coding error. The outputs generated by ChatGPT 5.3 Instant were benchmarked against results obtained by the author using standard statistical procedures (ground truth). Findings indicate that ChatGPT achieves high accuracy only under simple conditions, particularly with small samples, binary variables, and balanced distributions. However, its performance declines as analytical complexity increases. In moderately complex scenarios, the model shows partial accuracy, often reproducing contingency tables correctly but introducing deviations in derived statistics. In more complex settings, especially with unbalanced distributions or multiple categories, ChatGPT produces systematically biased results, typically overestimating agreement. Additionally, in large-scale datasets, the model fails to generate outputs due to operational limitations. Overall, the results reveal a lack of consistent reliability across realistic analytical scenarios. As a bottom line, the use of ChatGPT for computing these statistical metrics is not recommended, except in very simple cases involving small samples, and only under strict supervision and validation using established statistical software.