Better decisions start with harder questions.

Actively examine the opposing case

Lord, C. G.; Lepper, M. R.; Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243. December 1984.

Research finding

Considering the opposite reduced judgment bias in two human experiments.

Our research workflow

The method's own rule

Frozen stop criteria, source classification and verdicts are our implementation choices.

Match persona sources to the decision

Lee, Dongryeol; Łajewska, Weronika; Perelli, Leonardo; Mansour, Saab (2026). Data-Driven Personas for Survey Simulation: Insights into Simulation Alignment Across Data-Access Regimes. arXiv preprint; accepted at REALM EMNLP 2026. Submitted: 5 October; revised: 6 October 2026.

Research finding

Across five survey datasets, eight persona sources and three models, unrelated-source personas rarely beat simple demographic context; correct group assignment and relevant history helped.

Behavioral history and persona continuity

Pourtaheri, Khashayar; Zareei, Ahmad (2026). Behavioral History Outperforms Descriptions of the Person for LLM Synthetic Personas. arXiv preprint. Submitted: 2 October 2026.

Research finding

In an 845-person two-wave study, past answers improved individual prediction more than demographic, personality and cognitive descriptions, while substantial individual signal remained missing.

Evaluate outcome types and inference cost separately

Huang, Jin; Ferreras Garrucho, Diego; Xie, Yutong; Yuan, Walter M.; Mei, Qiaozhu; Lian, Chen; Hazell, Jonathon (2026). HouseholdBench: Evaluating Large Language Models as Predictors of Household Economic Behavior. arXiv preprint. Submitted: 6 October 2026.

Research finding

A 32-task benchmark compared 13 models with persistence and boosted-tree baselines. Fine-tuning plus prediction aggregation helped a small open-weight model on selected tasks; gains were not universal.

Make calibration assumptions inspectable

Gonzalez-Bonorino, Augusto; Biriukova, Kseniia; Capra, Monica (2026). Synthetic Cultural Agents from Aggregate Anchors. arXiv working paper. Submitted: 5 October 2026.

Research finding

Group-specific adapters retained declared aggregate preference signals, but transfer to new wording and agreement with human distributions were distinct and sometimes divergent.

Explicit context for synthetic personas

Argyle, L. P.; Busby, E. C.; Fulda, N.; Gubler, J. R.; Rytting, C.; Wingate, D. (2023). Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3), 337–351. Online: 21 February 2023.

Research finding

Conditioned GPT-3 samples reproduced some patterns in studied US survey populations.

Keep simulation separate from population evidence

Bisbee, J.; Clinton, J. D.; Dorff, C.; Kenkel, B.; Larson, J. M. (2024). Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4), 401–416. Online: 17 May 2024.

Research finding

Synthetic survey responses showed compressed variation and sensitivity to prompts and time.

Panel size: 10 profiles, plus 2 control repeats

Guest, G.; Bunce, A.; Johnson, L. (2006). How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods, 18(1), 59–82. February 2006. · Hennink, M.; Kaiser, B. N. (2022). Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Social Science & Medicine, 292, 114523. January 2022.

Research finding

In qualitative research with purposive samples, new themes stopped appearing within the first twelve interviews; a systematic review of 23 studies found saturation at 9–17 interviews for homogeneous populations and narrow objectives.

Profiles chosen by quota on what drives the decision

Guest, G.; Bunce, A.; Johnson, L. (2006). How Many Interviews Are Enough? An Experiment with Data Saturation and Variability. Field Methods, 18(1), 59–82. February 2006.

Research finding

Purposive samples are the most common form of non-probabilistic sampling in qualitative research: participants are chosen for what they can reveal, not drawn at random.

Each profile sees the options in its own order

Krosnick, J. A.; Alwin, D. F. (1987). An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement. Public Opinion Quarterly, 51(2), 201–219. 1987.

Research finding

In a split-ballot survey experiment, the order in which answer choices were presented changed the answers (a primacy effect).

The decision rule is fixed before the panel runs

Nosek, B. A.; Ebersole, C. R.; DeHaven, A. C.; Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. March 2018.

Research finding

Defining the questions and the analysis plan before seeing the outcomes separates tested predictions from explanations fitted after the fact.

Forced choice of the best and the weakest option

Finn, A.; Louviere, J. J. (1992). Determining the Appropriate Response to Evidence of Public Concern: The Case of Food Safety. Journal of Public Policy & Marketing, 11(2), 12–25. September 1992.

Research finding

The study that introduced best–worst choice into marketing research: respondents name the most and the least important item in a set, which yields a relative order rather than separate ratings.

Objections are asked before scores

Lord, C. G.; Lepper, M. R.; Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231–1243. December 1984.

Research finding

In two experiments, asking people to consider the opposite possibility corrected biased judgment better than asking them to be fair.

Score weights, winning margin and model mix

The method's own rule

The method sets the weights and the margin per type of decision and fixes them before each run: the criterion that matters most for that type of decision counts double, and a difference below the margin is treated as a tie.

Control product, differences instead of levels, agreement between models

The method's own rule

A product choice of the method, not validated on people: an everyday control product is asked the same way as the tested options, the panel is never shown a list of obstacles to rate, an option is placed against another only as a paired difference, and a difference on which the AI models point in different directions is not reported.

Plan allocation and sample size before testing

Kohavi, R.; Longbotham, R.; Sommerfield, D.; Henne, R. M. (2009). Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18, 140–181. Online: 30 July 2008; issue: February 2009.

Research finding

Controlled experiments use randomization, statistical power and variance considerations.

Define stopping rules before inspecting outcomes

Kohavi, R.; Deng, A.; Vermeer, L. (2022). A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments. Proceedings of ACM SIGKDD 2022, 3168–3177. Conference: 14–18 August 2022.

Research finding

Opportunistic stopping and post-hoc analysis can undermine A/B-test interpretation.

Separate a traffic pilot from proof of advertising impact

Lewis, R. A.; Rao, J. M. (2015). The Unfavorable Economics of Measuring the Returns to Advertising. The Quarterly Journal of Economics, 130(4), 1941–1973. Issue: November 2015. · Blake, T.; Nosko, C.; Tadelis, S. (2015). Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment. Econometrica, 83(1), 155–174. Issue: January 2015.

Research finding

Advertising returns can be difficult to estimate; the eBay experiment found effects differed across user groups.