Document detail
ID
oai:arXiv.org:2405.19323
Topic
Computer Science - Computation and... Computer Science - Artificial Inte... Computer Science - Computers and S... Computer Science - Machine Learnin...Author
Geng, Mingmeng He, Sihong Trotta, RobertoCategory
Computer Science
Year
2024
listing date
10/23/2024
Keywords
simulate surveys languageAbstract
Can large language models (LLMs) simulate social surveys?
To answer this question, we conducted millions of simulations in which LLMs were asked to answer subjective questions.
A comparison of different LLM responses with the European Social Survey (ESS) data suggests that the effect of prompts on bias and variability is fundamental, highlighting major cultural, age, and gender biases.
We further discussed statistical methods for measuring the difference between LLM answers and survey data and proposed a novel measure inspired by Jaccard similarity, as LLM-generated responses are likely to have a smaller variance.
Our experiments also reveal that it is important to analyze the robustness and variability of prompts before using LLMs to simulate social surveys, as their imitation abilities are approximate at best.
;Comment: 17 pages
Geng, Mingmeng,He, Sihong,Trotta, Roberto, 2024, Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys
High-Frequency Repetitive Magnetic Stimulation at the Sacrum Alleviates Chronic Constipation in Parkinson’s Patients
magnetic stimulation parkinson’s significant patients scale sacrum pd hf-rms chronic constipation scores
