A Reddit discussion simulation benchmark for LLM agents

ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions

Yaoning Yu1  ·  Ye Yu1  ·  John Sciarappo2  ·  Haojing Luo3  ·  Kai-min Chang2  ·  Yi-Chia Wang4  ·  Haohan Wang1
1University of Illinois Urbana–Champaign   2U.S. Bank   3Starc.Institute   4Carnegie Mellon University

ThreadBench is a benchmark that measures how realistically LLM agents simulate Reddit discussions. Built from 9,817 real threads across 8 domains and 116 subdomains, it compares generated threads with held-out real threads on 12 metrics of uniformity, expression, tone and interaction. The paper also introduces ThreadPDG, a Planner–Writer method evaluated against OASIS and SynthPAI.

arXiv:2606.14715 · Paper v2, October 10, 2026 · Website updated

12 metrics · 4 families · hover any wedge
THREADBENCH
ThreadBench · 12 metrics
Hover or select any wedge for the metric definition. Inner ring: 4 families. Outer ring: 12 individual metrics.
View all 12 metric definitions

Uniformity

Self-BLEU-4 · self_bleu_4
Pairwise 4-gram overlap between comments detects repeated wording.
Self-BERTScore · self_bertscore_mean_f1
Contextual similarity between comments detects paraphrases and similar expressions.
Semantic cosine · semantic_mean_cosine
Mean cosine similarity with all-mpnet-base-v2 captures convergence in meaning.

Expression

Story probability · mean_story_probability
StorySeeker estimates the prevalence of personal stories and first-person experiences.
Emotion entropy · emotion_entropy
Entropy of GoEmotions labels measures emotional diversity across comments.

Tone

Polite rate · polite_rate
Fraction of comments classified as polite by Intel/polite-guard.
Neutral rate · neutral_rate
Fraction of comments classified as neutral by Intel/polite-guard.
Impolite rate · impolite_rate
Fraction of comments classified as impolite by Intel/polite-guard.
Hard disagreement · hard_disagree_rate
Explicit disagreement in parent–reply pairs.

Interaction

Average depth · avg_depth
Mean reply depth measures how far multi-step exchanges develop.
Structural virality · structural_virality
Mean shortest-path distance between comments captures the spread of interaction across the reply tree.
Length variation · length_cv
Coefficient of variation of comment lengths measures the mixture of short reactions and longer contributions.
§ 01 — Motivation

Can LLM agents reproduce the distributional patterns of real online discussion?

LLM-based social simulation evaluation asks whether agents reproduce human behavior, social interaction, and online communities. ThreadBench focuses on how people discuss products, react to events, exchange opinions, and form thread-level patterns.

But generating conversations is not enough. Existing studies focus on individual responses, controlled role-playing, or task performance in simulated environments — not whether a set of simulated discussions reproduces the content patterns and interaction dynamics of real ones, or compares progress across systems.

Reddit discussions are a practical setting for this measurement: public, topic-grounded, and multi-party. They carry advice-seeking, personal experience, disagreement, emotional tone, repeated talking points, and complexity in structure — an observable window into broader social behavior.

ThreadBench addresses this gap. It combines 9,817 real Reddit threads across eight domains with separate seed, reference and test pools, and a framework that compares generated and real discussions through both comment-derived signals and reply-tree structure.

Illustration · same seed post, two discussion styles

Watch the difference. These illustrative panels share the same example post. The left depicts varied, nested discussion; the right depicts a fluent but more uniform style. This retained visual demonstration is not a sampled experiment or a live simulator.

r/cellphones · Posted by u/throwaway2024 Real-style illustration
Pixel 9 vs iPhone 16 for someone who hates iOS
▲ 247 ▽ 💬 84 comments Share Save
▲ 47 ▽
u/aperture_ · 47 points
Cameras are basically a wash now. Real difference is the software ecosystem and whether you actually want messages/airdrop to "just work."
ReplyShareSaveReport
▲ 12 ▽
u/ned_ · 12 points
Disagree. Pixel processing is still ahead, just nobody talks about it.
ReplyShareSave
▲ 8 ▽
u/dp_42 · 8 points
only in low light. daylight literally identical to my eye lol
ReplyShare
▲ 22 ▽
u/lo_fi__ · 22 points
switched 6 months ago. battery life is the single biggest QoL upgrade you'll feel, not the camera
ReplyShareSave
▲ −3 ▽
u/anon_ · -3 points
just get an iphone, you'll get over it
ReplyShare
r/ r/cellphones · Posted by u/SimAgent_01 Generated-style illustration
Pixel 9 vs iPhone 16 for someone who hates iOS
▲ 38 ▽ 💬 9 comments Share Save
▲ 12 ▽
u/HelpfulUser_42 · 12 points
Great question! Both the Pixel 9 and iPhone 16 are excellent devices, each with their own strengths. The right choice really depends on your specific needs and priorities.
ReplyShareSaveReport
▲ 4 ▽
u/TechFan_77 · 4 points
I agree with the previous comment. It really comes down to personal preference and which ecosystem you value most.
ReplyShareSave
▲ 9 ▽
u/PhoneEnthusiast_88 · 9 points
The Pixel 9 offers excellent camera quality and seamless Google integration, while the iPhone 16 provides a polished and consistent user experience.
ReplyShareSave
▲ 3 ▽
u/CuriousUser_23 · 3 points
That's a great point! The integration is definitely an important consideration when choosing between platforms.
ReplyShare
▲ 6 ▽
u/GadgetLover_15 · 6 points
I would recommend trying out both phones in person before making a final decision. Hope this helps!
ReplyShare

Same prompt, very different distributions. The left illustration carries blunt disagreement, lowercase fragments, a downvoted contrarian, and a 3-deep chain. The right illustration is fluent but uniformly polite, validation-led, without disagreement, and structurally flatter. ThreadBench's metrics quantify that gap thread-by-thread.

How does ThreadPDG work?

The Planner reads a complete real seed discussion and plans branches and comment roles. The Writer generates new comments from these plans; up to K = 3 drafts are composed using separate reference-pool statistics. Drafts inherit the seed’s comment slots and reply topology. Independent root-post-only generation remains future work. Method details · Paper §4.

Corpus: 9,817 threads · 8 domains · 116 reported subdomains. The per-domain counts in Table 1 sum to 117; we preserve the paper’s reported total and flag the discrepancy.

What we find

OASIS and SynthPAI remain distributionally mismatched to real Reddit threads. ThreadPDG passes 67–92% of metrics with GPT-5.4-mini, 58–100% with DeepSeek-V4.1-Flash, and 17–42% with GPT-4o-mini. The results below — and the full per-domain breakdown — make that gap concrete, family by family. Paper §5, Table 3.

§ 02 — Results

How does ThreadPDG compare with OASIS and SynthPAI?

Nine method-model combinations across eight domains; family mean distances and both-test pass counts
#Method · ModelUniformityExpressionToneInteractionBoth p ≥ 0.05 ↑
—
Real reference
150 vs. 150 · 200 repeats/domain
W₁ 0.018
|δ| 0.053
W₁ 0.052
|δ| 0.052
W₁ 0.033
|δ| 0.050
W₁ 0.100
|δ| 0.050
—
sampling reference
1
ThreadPDG
gpt-5.4-mini
W₁ 0.007 ×0.4
|δ| 0.081 ×1.5
W₁ 0.041 ×0.8
|δ| 0.043 ×0.8
W₁ 0.036 ×1.1
|δ| 0.126 ×2.5
W₁ 0.125 ×1.3
|δ| 0.060 ×1.2
78 / 96
81.2%
2
ThreadPDG
DeepSeek-V4.1-Flash
W₁ 0.007 ×0.4
|δ| 0.066 ×1.2
W₁ 0.043 ×0.8
|δ| 0.053 ×1.0
W₁ 0.036 ×1.1
|δ| 0.112 ×2.2
W₁ 0.144 ×1.4
|δ| 0.080 ×1.6
78 / 96
81.2%
3
ThreadPDG
gpt-4o-mini
W₁ 0.033 ×1.9
|δ| 0.433 ×8.2
W₁ 0.123 ×2.4
|δ| 0.215 ×4.2
W₁ 0.060 ×1.8
|δ| 0.227 ×4.5
W₁ 0.178 ×1.8
|δ| 0.148 ×3.0
27 / 96
28.1%
4
OASIS
DeepSeek-V4.1-Flash
W₁ 0.131 ×7.5
|δ| 0.805 ×15.2
W₁ 0.238 ×4.6
|δ| 0.513 ×9.9
W₁ 0.121 ×3.7
|δ| 0.403 ×8.1
W₁ 1.207 ×12.0
|δ| 0.957 ×19.3
11 / 96
11.5%
5
SynthPAI
DeepSeek-V4.1-Flash
W₁ 0.044 ×2.5
|δ| 0.687 ×13.0
W₁ 0.141 ×2.7
|δ| 0.320 ×6.2
W₁ 0.175 ×5.3
|δ| 0.625 ×12.5
W₁ 0.570 ×5.7
|δ| 0.531 ×10.7
7 / 96
7.3%
6
SynthPAI
gpt-5.4-mini
W₁ 0.048 ×2.7
|δ| 0.689 ×13.0
W₁ 0.236 ×4.5
|δ| 0.456 ×8.8
W₁ 0.143 ×4.3
|δ| 0.525 ×10.5
W₁ 0.610 ×6.1
|δ| 0.424 ×8.5
6 / 96
6.2%
7
SynthPAI
gpt-4o-mini
W₁ 0.117 ×6.7
|δ| 0.809 ×15.3
W₁ 0.187 ×3.6
|δ| 0.556 ×10.8
W₁ 0.185 ×5.6
|δ| 0.629 ×12.6
W₁ 0.538 ×5.4
|δ| 0.538 ×10.8
5 / 96
5.2%
8
OASIS
gpt-5.4-mini
W₁ 0.211 ×12.0
|δ| 0.962 ×18.2
W₁ 0.368 ×7.1
|δ| 0.755 ×14.6
W₁ 0.142 ×4.3
|δ| 0.506 ×10.1
W₁ 1.250 ×12.5
|δ| 0.970 ×19.5
3 / 96
3.1%
9
OASIS
gpt-4o-mini
W₁ 0.254 ×14.5
|δ| 0.960 ×18.2
W₁ 0.316 ×6.1
|δ| 0.738 ×14.3
W₁ 0.309 ×9.4
|δ| 0.688 ×13.8
W₁ 1.259 ×12.6
|δ| 0.971 ×19.6
1 / 96
1.0%

Family columns summarize mean Wasserstein distance W₁ and mean absolute Cliff’s delta across eight domains. Lower distances indicate closer distributions; ×N is the ratio to the repeated-real reference. W₁ means mix native metric scales and are descriptive summaries, not a scale-independent ranking.

All three methods are evaluated with the same twelve metrics across eight domains. A metric passes when pMWU ≥ 0.05 and pKS ≥ 0.05. Totals count passing metric × domain cells out of 12 × 8 = 96. Each comparison uses 150 generated threads and 150 real test threads. Passing does not prove equivalence. Paper §3.3, Table 3, Appendices C and K.

Human evaluation: ThreadPDG is not distinguished in 78.7% of its non-tied discussion pairs, versus 61.1% for OASIS and 58.7% for SynthPAI. The study uses 436 retained annotators and 240 pairs across all three systems; generated-as-real and cannot-distinguish votes are combined by pair-level majority vote.

§ 03 — Real-vs-Real Reference

What does a good score look like?

These figures show the sampling variation between two groups of 150 real threads, compared 200 times per domain across all eight domains. Any model's score on the leaderboard (or the full per-domain results) should be read against this floor. These figures are not part of the leaderboard; they characterise the measurement.

Repeated-sample MWU & KS p-value pass rates
Repeated-sample p-value pass rates across domains and metrics

Across repeated real-vs-real comparisons, MWU passes in 94.99%, KS in 96.95%, and both pass in 94.30% of metric comparisons. A pass means no difference was detected at α = 0.05; it does not prove equivalence.

Real-vs-real W1 (log scale)
Real-vs-real W1 by family and domain
Real-vs-real |Cliff's δ|
Real-vs-real Cliff's delta by family and domain
Model gap vs real-vs-real baseline
Per-model W1 and |Cliff delta| ratios over the real-vs-real baseline

For each family, the dotplot compares mean W1 and mean |δ| with the repeated-real reference, using the revised paper’s eight domains and twelve metrics. Ratios summarize discrepancy magnitude; ×1.0 is not proof of equivalence. W1 family means mix native metric scales and are descriptive summaries only. Paper Appendices C and K.

§ 04 — Quick start

How do I run the earlier MiroBench code?

These commands document the repository’s earlier MiroBench code and five-domain data release. The revised ThreadBench paper uses twelve metrics across eight domains and ThreadPDG; the expanded corpus and method are not implied to be released by these commands.

1 · Install

Core install runs scoring & comparison; the [generation] extra pulls in camel-oasis + camel-ai for the mirobench generate command.

install
$ git clone https://github.com/yyu6/ThreadBench.git
$ cd ThreadBench
$ pip install -e ".[generation]"
2 · Disagreement-scorer weights (one-time)

The disagreement scorer uses the Stance_Rel RoBERTa checkpoint from Luo et al. — the only scorer needing a manual download. The other eight scorers auto-fetch their models from HuggingFace on first use.

Download the checkpoint from the original authors' Google Drive and unzip into ./Stance_Rel/RoBERT_rel_1.5e-05/ inside your working directory. If absent, the disagreement scorer is skipped and the remaining eight still run.

3 · Generate threads from products

Set LLM_API_KEY in a .env file (template at .env.example), then run:

generate
$ mirobench generate path/to/products.json \
    --agents 30 --rounds 30 --seed-posts 3 \
    --output-dir artifacts/simulations/ \
    --discussion-backbone vanilla_oasis

Runs the six-stage pipeline (load → analyse → personas → build OASIS config → run OASIS → export discussion.json). Use --discussion-backbone geo_patched to enable MiroBench's runtime patches over the OASIS Reddit env, or --overlay best_overlay.json to apply a calibration overlay produced in step 6.

Already have discussion.json files from another simulator (e.g. OASIS's own Reddit cookbook)? Skip step 3 — MiroBench accepts any input that follows the discussion.json schema.

4 · Score generated threads
score
$ mirobench score my_threads/ --device cpu

Each thread is a directory holding one discussion.json file; see example_thread_format.json for the JSON schema. The command writes my_threads/thread_scores.csv with one row per thread and 61 columns covering all nine scorer families. A 5-row schema sample lives at example_thread_scores.csv — the same file mirobench compare accepts verbatim.

5 · Compare against real Reddit

Sample size. MWU and KS are non-parametric rank tests; below ~30 threads the p-values get noisy. Use ≥ 50 threads per domain for a meaningful similarity ratio (the revised paper evaluates 150 generated threads per method × model × domain).

compare
$ mirobench compare my_threads/thread_scores.csv \
    --core-only \
    --model-name "my-model"

The command writes mirobench_comparison.csv with one row per (metric × domain). Each row carries two primary readings — mwu_p_value and ks_p_value (the revised protocol requires both p-values ≥ 0.05; passing does not prove equivalence) — and two secondary ones — cliffs_delta and wasserstein (how far off when the p-value rejects).

The terminal also prints a headline similarity ratio per domain, e.g. Similarity (MWU p > 0.05): 5/12 (42%). This MWU-only terminal summary belongs to the earlier release. Revised ThreadBench results require both MWU and KS; see current paper results.

6 · Run the calibration loop (optional)
calibration
$ python -m calibration \
    --real-train-csv mirobench/data/credit_cards/reference_scores/thread_scores_train.csv \
    --real-val-csv   mirobench/data/credit_cards/reference_scores/thread_scores_val.csv \
    --real-test-csv  mirobench/data/credit_cards/reference_scores/thread_scores_test.csv \
    --few-shot-dir   mirobench/data/credit_cards/real_threads \
    --products-json  mirobench/data/credit_cards/products/product_descriptions.json \
    --output-dir     calibration_output/ \
    --calibration-model gpt-4o-mini \
    --iterations 12 --candidates 5 --device cpu

Outputs land in calibration_output/: before_calibration/ (Phase 0 baseline), iterations/ (Phase 1 per-iteration data), best_overlay.json (chosen prompt overlay), and after_calibration/ (Phase 2 final evaluation). Requires an OPENAI_API_KEY and an OASIS install for the candidate simulations.

Full flag reference in the README.

§ 05 — FAQ

Questions researchers ask about ThreadBench

What is ThreadBench?

ThreadBench is a benchmark for measuring realism in LLM-based social simulation of Reddit discussions, introduced in “ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions” (arXiv:2606.14715, v2, October 10, 2026). It contains 9,817 real Reddit threads across 8 domains (Camera, Celebrity, Cellphone, Game, Health Issue, Movies, News, TV Series) and 116 reported subdomains, and tests whether a set of generated threads reproduces the distributions of held-out real threads.

What does ThreadBench measure?

ThreadBench scores 12 thread-level metrics in four families: Uniformity (Self-BLEU-4, Self-BERTScore, Semantic cosine); Expression (Story probability, Emotion entropy); Tone (Polite rate, Neutral rate, Impolite rate, Hard disagreement); Interaction (Average depth, Structural virality, Length variation). Together they cover repeated wording and meaning, personal stories and emotion, politeness and disagreement, and the shape of the reply tree. Read all 12 definitions.

How does ThreadBench decide whether simulated discussions are realistic?

For each method, model and domain, ThreadBench compares 150 generated threads with 150 held-out real test threads. A metric passes only when both two-sided Mann–Whitney U and Kolmogorov–Smirnov tests give p ≥ 0.05, so each system is scored on 12 × 8 = 96 metric–domain cells. Wasserstein-1 distance and absolute Cliff’s delta describe how large the remaining gaps are. Repeated real-vs-real samples pass both tests in 94.30% of comparisons. A pass means no difference was detected, not proof of equivalence.

How do ThreadPDG, OASIS and SynthPAI compare on ThreadBench?

Under the same protocol, ThreadPDG passes 78 of 96 metric–domain cells with GPT-5.4-mini, 78 with DeepSeek-V4.1-Flash and 27 with GPT-4o-mini. The strongest baselines pass 11 of 96 (OASIS with DeepSeek-V4.1-Flash) and 7 of 96 (SynthPAI with DeepSeek-V4.1-Flash). In the human study (436 retained annotators, 240 matched pairs), ThreadPDG is not distinguished from real discussions in 78.7% of its non-tied pairs, versus 61.1% for OASIS and 58.7% for SynthPAI. See all 72 comparisons.

What is ThreadPDG?

ThreadPDG (Thread-Prompt-based Discussion Generation) is the paper’s prompt-based Planner–Writer method. The Planner reads a complete real seed thread and plans each branch and comment role; the Writer fills those plans with assigned personas; up to K = 3 drafts are combined branch by branch against reference-pool metric targets. It depends on real seed discussions, and root-post-only generation is future work. How ThreadPDG works.

When should I use ThreadBench?

Use ThreadBench to check whether an LLM agent or social simulator produces multi-party discussions whose content and reply structure match real Reddit threads, rather than individual replies that merely sound fluent. It covers Reddit across eight domains. It does not score single-persona accuracy, other platforms, or group-level phenomena such as polarization or information cascades.

How is ThreadBench different from OASIS, SynthPAI and SimBench?

ThreadBench is an evaluation benchmark: it tests whether the distribution of generated discussions matches real Reddit threads. OASIS (CAMEL-AI) is a social media simulation framework, and SynthPAI is a synthetic Reddit-comment dataset for personal attribute inference; ThreadBench evaluates their generation methods as baselines. SimBench benchmarks human-behavior simulation on structured tasks such as moral and economic choices; it is related work with a different scope and no shared score scale.

What changed from MiroBench to ThreadBench?

ThreadBench is the revised name and protocol of the earlier MiroBench project. The earlier MiroBench release, which is the Python code and data in the GitHub repository, covers 5 product domains and an archived 16-metric, MWU-only leaderboard. The revised paper uses 9,817 threads across 8 domains, 12 metrics, both MWU and KS tests, and introduces ThreadPDG. The expanded corpus and ThreadPDG implementation are not implied to be released in the repository.

Where can I find the ThreadBench paper, results and data?

The paper is on arXiv (2606.14715). Published results for all 72 comparisons and 864 metric-level p-value pairs are available as CSV and JSON, and the results explorer filters them by model, method and domain. The GitHub repository holds the earlier MiroBench code and five-domain data. Plain-text summaries for AI assistants are at llms.txt and llms-full.txt.

How do I cite ThreadBench?

Cite Yu et al. (2026), “ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions,” arXiv:2606.14715, doi:10.48550/arXiv.2606.14715. Copy the BibTeX entry below or download threadbench.bib.

§ 06 — Citation

If you find ThreadBench useful in your research, please cite our paper.

Use the revised seven-author citation for arXiv:2606.14715. Download BibTeX.

threadbench.bib
@article{yu2026threadbench,
  title = {ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions},
  author = {Yu, Yaoning and Yu, Ye and Sciarappo, John and Luo, Haojing and Chang, Kai-min and Wang, Yi-Chia and Wang, Haohan},
  journal = {arXiv preprint arXiv:2606.14715},
  year = {2026},
  eprint = {2606.14715},
  archivePrefix = {arXiv},
  primaryClass = {cs.MA},
  doi = {10.48550/arXiv.2606.14715},
  url = {https://arxiv.org/abs/2606.14715}
}
Inspired by

The earlier MiroBench implementation builds on prior open-source work: OASIS (CAMEL-AI) — the multi-agent social-interaction simulation framework that powers our generation pipeline — and MiroFish, a universal swarm-intelligence prediction engine powered by multi-agent simulation.