NOTES · MODEL PANELS
Tell model who e be. Im causal judgements no too change.
October 2026
Rooms of language models for True Causes dey give each model one persona: nurse wey dey rent, homeowner wey don retire, small developer. Di idea na say different pipo dey see different causes, and panel of models wey dey play dem go see dem too. Whether model wey dem tell who e be dey answer like dat pesin na question wey pipo don know (Argyle et al., 2023); we test am on causal judgements.
For judging, e no dey happen. Across 21 models and 14 personas, di persona wey dem give model no change which causal links e accept by anytin wey we fit measure. Which model e be change am well-well.
Di study
We take 120 links from one public-record room on housing affordability, wey we choose so links wey di room certify, reject and no fit settle go dey equal. Each of di 21 models answer evri link, once as imself and four or five times as different persona: twelve housing personas (renters and owners, one landlord, one planner, one developer, one tenant organiser, one construction labourer and oda ones), and two twins, di same pesin except for one fact, renting instead of owning. Seven models play each persona. Dat one na 119 seats and 14,220 answers, evri seat dey answer evri link, so no answer depend on anoda one.
Wetin decide di answers
Di link itself dey explain about half of all di variation (52%): models mostly gree on which links make sense. Di model own reading of particular links dey explain anoda quarter (24%), no matter di persona wey e dey play. Di general tendency of model to tok yes dey add 3%.
Di persona no add anytin. Im share of di variation for which links get yes na 0.000; im share of general lean towards yes, 0.002. Test wey dey try identify di persona from di answers of one seat, by models wey dem no see am inside, no do betta than chance (6.1% against 7.1%). And di seats no move for di direction wey di features of dia persona go predict: di correlation na zero. Wen persona change answer, for between 1% and 15% of links depending on di model, e change am by random.
Dis one match wetin oda pipo don find. Persona variables dey explain small of di variance for annotation tasks (Hu & Collier, 2024), and di role wey dem name for system prompt no dey always change model answers (Zheng et al., 2024). Causal question dey read for model as question about di world, no be about di pesin wey dey ask.
Wetin personas change: who dey answer
Personas make models tok “I no fit judge” more often, 26% of di time against 19% as demself, and e no dey even. Personas wey young, wey get low income, wey dey rent, wey lean left and wey no be from US abstain more, by up to 14 points; personas wey old, wey lean right, landlord, high-income and owner abstain less, by up to 13 points. Di models play personas wey get lower status as pipo wey no too sure of wetin dem know. Pipo don know say persona prompts dey carry dat kain stereotypes enter reasoning (Gupta et al., 2024). Some models refuse well-well pass with persona: Gemini 3.5 Flash tok “I no fit judge” to 59% of links as persona and to none as imself.
Which models dey judge well
Di charts dey place each model by how often e dey tok yes (across) and how much information im answers carry (up). We dey estimate informedness without knowing di correct answers, from how di models gree with each oda (Dawid & Skene, 1979): 1 na judge wey dey consistent perfectly, 0 na answer wey no carry any signal. E dey measure agreement with wetin di panel as a whole find, no be correctness, so strong model wey no gree with weaker ones fit score low pass wetin e deserve.
Most models dey discriminate: dem dey tok yes to between one quarter and two thirds of links, and dia answers dey carry signal. Claude Sonnet 5.5 and DeepSeek V4 Flash gree pass with di panel. Smaller models like Gemma 3 4B and Command R7B dey carry less. Schematron, model wey dem build to extract data instead of to judge claims, tok “I no fit judge” to almost evritin, and wetin e answer no carry almost any signal.
Models get dia own views?
If personas no dey shape di answers, maybe di models demself get different theories of di problem, one dey blame regulation, anoda dey blame speculation. One quarter of di variation na models wey dey read di same link different ways, so we look for di directions of dat difference: model wey dey place each model and each link on few dimensions, from di answers wey di models give as demself. Dimension dey count only if e stable, wey mean say if we fit am separately on two halves of di links, e go put di models for almost di same places.
One dimension stable, and e no be theory. For one end na links wey no too make sense, like “luxury apartments wey foreign investors buy” wey dey lead to “America dey add 600,000 to 1.2 million immigrants evri year”, wey di weaker models accept. For di oda end na one mechanism wey hang togeda, residents’ votes and di interests of incumbents wey dey keep supply limited, wey all di stronger models accept. Di dimension na how well model dey judge: di strong models dey sit close togeda, and each weak model dey comot from dem for im own way.
Among di eleven stronger models, wey we fit on di 120 links of di study, notin stable. Di links too few. Wen we fit am on all di 870 links of di room, wey di same eleven models answer as demself, one difference show, and e stable: wen we fit am separately on two halves of di links, di models land for almost di same places (0.89).
For one end, models dey accept links from pipo’s interests go political decisions: homeowners’ stake for prices wey dey rise wey dey lead to government’s changes against supply, residents’ votes wey dey lead to Proposition 13. Gemini 3.5 Flash, Claude Sonnet 5.5, Nex and DeepSeek dey dia. For di oda end, models dey accept links from barriers enter di supply mechanism: regulation, di interests of incumbents, councils wey dey refuse low-cost flats and property rights, each one dey lead to “more housing units dey reduce prices”. Mistral Small, Ministral 8B, Laguna and Qwen dey dia. Roughly, one side dey explain housing through who dey gain for politics, di oda one through wetin dey block supply. Dat reading na our own, from di links for each end. Di second dimension dey put di two GPT-OSS models for one end, but im links no share any theme wey we fit name.
Di variation gree. Among di eleven stronger models, di link itself dey explain 68% of am, against 52% with all 21 models, and di readings wey belong to each model fall from 24% to 8%: about two thirds of wetin look like different readings come from di weaker models. Di rest small pass, but na di consistent difference wey dey above.
Wetin dis one mean for True Causes
Panel of models dey offer judgement, with narrow range of views. Personas no dey add perspectives. Capable models dey gree on most links, and dem dey differ di same way on di rest, along one line between political and supply explanations. Dat one na range, but narrow one, wey di way dem train di models set: no be di range of views wey pipo with different experience go bring. Research on models wey dey summarise deliberations dey point di same way: di summaries no dey represent minority positions well, across 18 models (Zhu et al., 2025), and statements wey dem write to find consensus fit win agreement through dia tone pass dia content (Tsai et al., 2024). For True Causes, model no dey tok for room. E dey write or judge single links, dem dey count evri judgement, and divide dey stay for di map as divide.
To choose panel na about quality first, den balance. Di biggest difference between models na how well dem dey judge, so panel suppose be capable models, and dem go leave di weak ones out. Among capable models, di families dey lean different ways, so panel suppose draw from plenti of dem, and no one family suppose pass di rest for number.
Dem suppose screen models bifor dem give dem seat. Model wey dey tok yes to evritin, no to evritin or “I no fit judge” to evritin dey add noise to evri pair wey e judge. Short calibration on links wey dem don know dey find dem.
Make dem no use personas for judging. Dem dey add abstentions and status stereotype, and no diversity of judgement. Writing na separate question. Room of models with personas dey write cards wey different, but different models dey write different ways, and one model dey write different way each time dem ask am, so cards wey different alone no dey show say di personas do anytin. Di test na di one wey we use here for judging: di same models go write causes with persona and without persona, and dem go count di distinct causes wey each one produce.
Rooms of pipo different. Pesin dey bring wetin e don see, no be description of somebodi wey fit don see am. Dis study no tok anytin against dat one; e dey tok say model wey dem tell say e be renter neva live like one. Wia di point na to find out wetin pipo no gree about, na only pipo fit supply am. For dat rooms, model dey more useful to ask than to answer: question about one card, one at a time and without im own view (wetin you go see if dis one true? wetin dey happen for di middle?), dey help di pesin wey write am make di cause clear without putting di model’s words inside im mouth.
Limits
One question, housing; 120 links; four or five personas for each model, each one dey answer once. Effects wey small pass wetin dis design fit see no go show. Dem write personas as role descriptions; stronger ways to bring dem out, like to ask model make e reason in character bifor e answer, fit move am further, togeda with di risk of caricature (Cheng et al., 2023). Informedness dey measure agreement with di panel, no be truth, and models wey dem use as judges of deliberation no dey gree with pipo as well as pesin go hope (Zhu et al., 2025): panel wey gree with imself fit still wrong for di same direction. Di difference between capable models show only with all di 870 links; with 120 e no show at all, so smaller differences fit still hide, and question wey get more argument than housing fit divide models more.
References
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337–351.
Cheng, M., Piccardi, T., & Yang, D. (2023). CoMPosT: Characterizing and evaluating caricature in LLM simulations. Proceedings of EMNLP 2023.
Dawid, A. P., & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C, 28(1), 20–28.
Gupta, S., Shrivastava, V., Deshpande, A., Kalyan, A., Clark, P., Sabharwal, A., & Khot, T. (2024). Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. Proceedings of ICLR 2024.
Hu, T., & Collier, N. (2024). Quantifying the persona effect in LLM simulations. Proceedings of ACL 2024.
Tsai, L. L., Pentland, A., Braley, A., Chen, N., Enríquez, J. R., & Reuel, A. (2024). Generative AI for pro-democracy platforms. An MIT Exploration of Generative AI, MIT.
Zheng, M., Pei, J., Logeswaran, L., Lee, M., & Jurgens, D. (2024). When “a helpful assistant” is not really helpful: Personas in system prompts do not improve performances of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024.
Zhu, S., Yang, S., Bakker, M. A., Pentland, A., & Pei, J. (2025). Can AI truly represent your voice in deliberations? A comprehensive study of large-scale opinion aggregation with LLMs. arXiv:2510.05154.