THE PAPER, IN NUMBERS

A closer look at
discussion realism.

Explore all 72 comparisons from ThreadBench: three methods, three models and eight domains. Each result reports how many of twelve metrics pass both statistical tests against held-out real discussions.

The model matters.

Domain-level ranges from the paper; higher is better for pass rates. ThreadPDG’s 58–100% headline covers the two stronger backbones. GPT-4o-mini reaches 17–42%.

Pass-rate ranges, by model and method
ModelMethodPassing metricsPass-rate range
gpt-5.4-miniThreadPDG8/12–11/1267–92%
gpt-5.4-miniOASIS0/12–1/120–8%
gpt-5.4-miniSynthPAI0/12–2/120–17%
DeepSeek-V4.1-FlashThreadPDG7/12–12/1258–100%
DeepSeek-V4.1-FlashOASIS1/12–2/128–17%
DeepSeek-V4.1-FlashSynthPAI0/12–2/120–17%
gpt-4o-miniThreadPDG2/12–5/1217–42%
gpt-4o-miniOASIS0/12–1/120–8%
gpt-4o-miniSynthPAI0/12–2/120–17%

Every domain.
Every comparison.

150 generated threads and 150 real test threads per comparison. Open a row to inspect its twelve metrics.

72 comparisons · 12 metrics each

Metric pass counts and detailed p-values for every paper comparison
DomainMethodModelMetrics passing both testsDetails
CameraOASISDeepSeek-V4.1-Flash
1/128%
12 metrics

OASIS · DeepSeek-V4.1-Flash · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.08.03Fail
Hard disagreementhard_disagree_rateTone.46.06Pass
CameraOASISgpt-5.4-mini
0/120%
12 metrics

OASIS · gpt-5.4-mini · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CameraOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CameraSynthPAIDeepSeek-V4.1-Flash
1/128%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.08.11Pass
Average depthavg_depthInteraction.01<.01Fail
Structural viralitystructural_viralityInteraction.39.03Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CameraSynthPAIgpt-5.4-mini
1/128%
12 metrics

SynthPAI · gpt-5.4-mini · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.55.14Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.46<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.03.07Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CameraSynthPAIgpt-4o-mini
2/1217%
12 metrics

SynthPAI · gpt-4o-mini · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.05<.01Fail
Structural viralitystructural_viralityInteraction.85.22Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone.07.17Pass
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CameraThreadPDGDeepSeek-V4.1-Flash
11/1292%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.44.71Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.92.28Pass
Semantic cosinesemantic_mean_cosineUniformity.68.44Pass
Average depthavg_depthInteraction.86.98Pass
Structural viralitystructural_viralityInteraction.81.85Pass
Length variationlength_cvInteraction.21.44Pass
Story probabilitymean_story_probabilityExpression.15.16Pass
Emotion entropyemotion_entropyExpression.50.79Pass
Polite ratepolite_rateTone.76.33Pass
Neutral rateneutral_rateTone.15.07Pass
Impolite rateimpolite_rateTone<.01.05Fail
Hard disagreementhard_disagree_rateTone.10.09Pass
CameraThreadPDGgpt-5.4-mini
10/1283%
12 metrics

ThreadPDG · gpt-5.4-mini · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.12.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity.27.19Pass
Semantic cosinesemantic_mean_cosineUniformity.72.61Pass
Average depthavg_depthInteraction.98.92Pass
Structural viralitystructural_viralityInteraction.70.57Pass
Length variationlength_cvInteraction.70.34Pass
Story probabilitymean_story_probabilityExpression.17.51Pass
Emotion entropyemotion_entropyExpression.65.65Pass
Polite ratepolite_rateTone.02<.01Fail
Neutral rateneutral_rateTone.06.20Pass
Impolite rateimpolite_rateTone.83.90Pass
Hard disagreementhard_disagree_rateTone.19.37Pass
CameraThreadPDGgpt-4o-mini
2/1217%
12 metrics

ThreadPDG · gpt-4o-mini · Camera

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.03.02Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.41.37Pass
Structural viralitystructural_viralityInteraction.23.23Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CelebrityOASISDeepSeek-V4.1-Flash
1/128%
12 metrics

OASIS · DeepSeek-V4.1-Flash · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.67.15Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CelebrityOASISgpt-5.4-mini
1/128%
12 metrics

OASIS · gpt-5.4-mini · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.37.27Pass
CelebrityOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.15<.01Fail
CelebritySynthPAIDeepSeek-V4.1-Flash
0/120%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.02.02Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CelebritySynthPAIgpt-5.4-mini
0/120%
12 metrics

SynthPAI · gpt-5.4-mini · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.11<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.02<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.03.02Fail
CelebritySynthPAIgpt-4o-mini
0/120%
12 metrics

SynthPAI · gpt-4o-mini · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CelebrityThreadPDGDeepSeek-V4.1-Flash
12/12100%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.77.71Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.77.88Pass
Semantic cosinesemantic_mean_cosineUniformity.90.88Pass
Average depthavg_depthInteraction.58.56Pass
Structural viralitystructural_viralityInteraction.30.33Pass
Length variationlength_cvInteraction.05.20Pass
Story probabilitymean_story_probabilityExpression.56.67Pass
Emotion entropyemotion_entropyExpression.67.80Pass
Polite ratepolite_rateTone.72.55Pass
Neutral rateneutral_rateTone.68.77Pass
Impolite rateimpolite_rateTone.69.66Pass
Hard disagreementhard_disagree_rateTone.78.64Pass
CelebrityThreadPDGgpt-5.4-mini
11/1292%
12 metrics

ThreadPDG · gpt-5.4-mini · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.12.12Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.29.70Pass
Semantic cosinesemantic_mean_cosineUniformity.91.79Pass
Average depthavg_depthInteraction.56.50Pass
Structural viralitystructural_viralityInteraction.26.22Pass
Length variationlength_cvInteraction.17.11Pass
Story probabilitymean_story_probabilityExpression.24.53Pass
Emotion entropyemotion_entropyExpression.49.52Pass
Polite ratepolite_rateTone.05.06Fail
Neutral rateneutral_rateTone.58.45Pass
Impolite rateimpolite_rateTone.31.43Pass
Hard disagreementhard_disagree_rateTone.41.62Pass
CelebrityThreadPDGgpt-4o-mini
3/1225%
12 metrics

ThreadPDG · gpt-4o-mini · Celebrity

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.77.71Pass
Structural viralitystructural_viralityInteraction.91.83Pass
Length variationlength_cvInteraction.04.05Fail
Story probabilitymean_story_probabilityExpression<.01.02Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01.01Fail
Impolite rateimpolite_rateTone.51.92Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CellphoneOASISDeepSeek-V4.1-Flash
1/128%
12 metrics

OASIS · DeepSeek-V4.1-Flash · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.87.90Pass
CellphoneOASISgpt-5.4-mini
1/128%
12 metrics

OASIS · gpt-5.4-mini · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.93.20Pass
CellphoneOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.05<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.05.01Fail
CellphoneSynthPAIDeepSeek-V4.1-Flash
1/128%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.17.12Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.17.03Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CellphoneSynthPAIgpt-5.4-mini
1/128%
12 metrics

SynthPAI · gpt-5.4-mini · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.64.06Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.07<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CellphoneSynthPAIgpt-4o-mini
1/128%
12 metrics

SynthPAI · gpt-4o-mini · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction.63.26Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.11<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
CellphoneThreadPDGDeepSeek-V4.1-Flash
10/1283%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.09.42Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.23.33Pass
Semantic cosinesemantic_mean_cosineUniformity.90.35Pass
Average depthavg_depthInteraction.43.41Pass
Structural viralitystructural_viralityInteraction.20.43Pass
Length variationlength_cvInteraction.07<.01Fail
Story probabilitymean_story_probabilityExpression.78.42Pass
Emotion entropyemotion_entropyExpression.12.08Pass
Polite ratepolite_rateTone.18.15Pass
Neutral rateneutral_rateTone.971.0Pass
Impolite rateimpolite_rateTone.05.13Pass
Hard disagreementhard_disagree_rateTone<.01.01Fail
CellphoneThreadPDGgpt-5.4-mini
10/1283%
12 metrics

ThreadPDG · gpt-5.4-mini · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01.04Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity.95.70Pass
Semantic cosinesemantic_mean_cosineUniformity.13.41Pass
Average depthavg_depthInteraction.76.70Pass
Structural viralitystructural_viralityInteraction.91.93Pass
Length variationlength_cvInteraction.31.49Pass
Story probabilitymean_story_probabilityExpression.92.88Pass
Emotion entropyemotion_entropyExpression.84.55Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.26.12Pass
Impolite rateimpolite_rateTone.17.33Pass
Hard disagreementhard_disagree_rateTone.17.27Pass
CellphoneThreadPDGgpt-4o-mini
5/1242%
12 metrics

ThreadPDG · gpt-4o-mini · Cellphone

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.55.72Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.07.16Pass
Structural viralitystructural_viralityInteraction.01.04Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01.06Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone.46.29Pass
Neutral rateneutral_rateTone.20.15Pass
Impolite rateimpolite_rateTone.02.02Fail
Hard disagreementhard_disagree_rateTone.10.07Pass
GameOASISDeepSeek-V4.1-Flash
2/1217%
12 metrics

OASIS · DeepSeek-V4.1-Flash · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.07.18Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.08.08Pass
GameOASISgpt-5.4-mini
0/120%
12 metrics

OASIS · gpt-5.4-mini · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.20.02Fail
GameOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
GameSynthPAIDeepSeek-V4.1-Flash
1/128%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.08.07Pass
GameSynthPAIgpt-5.4-mini
2/1217%
12 metrics

SynthPAI · gpt-5.4-mini · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.07.07Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.04<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.37.63Pass
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.02<.01Fail
GameSynthPAIgpt-4o-mini
0/120%
12 metrics

SynthPAI · gpt-4o-mini · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction.48.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
GameThreadPDGDeepSeek-V4.1-Flash
10/1283%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.75.51Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.11.08Pass
Semantic cosinesemantic_mean_cosineUniformity.15.45Pass
Average depthavg_depthInteraction.18.28Pass
Structural viralitystructural_viralityInteraction.03.05Fail
Length variationlength_cvInteraction.71.69Pass
Story probabilitymean_story_probabilityExpression.71.91Pass
Emotion entropyemotion_entropyExpression.30.22Pass
Polite ratepolite_rateTone.17.05Pass
Neutral rateneutral_rateTone.60.63Pass
Impolite rateimpolite_rateTone.88.36Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail
GameThreadPDGgpt-5.4-mini
8/1267%
12 metrics

ThreadPDG · gpt-5.4-mini · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.36.72Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.03.02Fail
Semantic cosinesemantic_mean_cosineUniformity.29.63Pass
Average depthavg_depthInteraction.19.29Pass
Structural viralitystructural_viralityInteraction.07.03Fail
Length variationlength_cvInteraction.22.29Pass
Story probabilitymean_story_probabilityExpression.931.0Pass
Emotion entropyemotion_entropyExpression.70.53Pass
Polite ratepolite_rateTone.03.02Fail
Neutral rateneutral_rateTone.63.89Pass
Impolite rateimpolite_rateTone.46.53Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail
GameThreadPDGgpt-4o-mini
3/1225%
12 metrics

ThreadPDG · gpt-4o-mini · Game

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.06.07Pass
Structural viralitystructural_viralityInteraction.01.03Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.69.84Pass
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.02.04Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.67.53Pass
Health IssueOASISDeepSeek-V4.1-Flash
1/128%
12 metrics

OASIS · DeepSeek-V4.1-Flash · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.02.15Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone.08.09Pass
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
Health IssueOASISgpt-5.4-mini
1/128%
12 metrics

OASIS · gpt-5.4-mini · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.51.10Pass
Health IssueOASISgpt-4o-mini
1/128%
12 metrics

OASIS · gpt-4o-mini · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.79.66Pass
Health IssueSynthPAIDeepSeek-V4.1-Flash
2/1217%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.47.57Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.49.19Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
Health IssueSynthPAIgpt-5.4-mini
1/128%
12 metrics

SynthPAI · gpt-5.4-mini · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.43.57Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.66<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.13.02Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
Health IssueSynthPAIgpt-4o-mini
0/120%
12 metrics

SynthPAI · gpt-4o-mini · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
Health IssueThreadPDGDeepSeek-V4.1-Flash
7/1258%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.02.02Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity.02.14Fail
Semantic cosinesemantic_mean_cosineUniformity.40.53Pass
Average depthavg_depthInteraction.48.23Pass
Structural viralitystructural_viralityInteraction.39.72Pass
Length variationlength_cvInteraction.74.36Pass
Story probabilitymean_story_probabilityExpression.37.72Pass
Emotion entropyemotion_entropyExpression.35.36Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.03.11Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.29.23Pass
Health IssueThreadPDGgpt-5.4-mini
8/1267%
12 metrics

ThreadPDG · gpt-5.4-mini · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.44.11Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.92.96Pass
Semantic cosinesemantic_mean_cosineUniformity.21.25Pass
Average depthavg_depthInteraction.47.23Pass
Structural viralitystructural_viralityInteraction.41.70Pass
Length variationlength_cvInteraction.22.04Fail
Story probabilitymean_story_probabilityExpression.76.82Pass
Emotion entropyemotion_entropyExpression.06.09Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.02.08Fail
Impolite rateimpolite_rateTone.05.16Fail
Hard disagreementhard_disagree_rateTone.34.16Pass
Health IssueThreadPDGgpt-4o-mini
3/1225%
12 metrics

ThreadPDG · gpt-4o-mini · Health Issue

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.42.26Pass
Structural viralitystructural_viralityInteraction.41.67Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01.04Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone.27.36Pass
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.01.02Fail
MoviesOASISDeepSeek-V4.1-Flash
2/1217%
12 metrics

OASIS · DeepSeek-V4.1-Flash · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.12.13Pass
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.34.70Pass
Hard disagreementhard_disagree_rateTone.02.07Fail
MoviesOASISgpt-5.4-mini
0/120%
12 metrics

OASIS · gpt-5.4-mini · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.02<.01Fail
MoviesOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.08<.01Fail
MoviesSynthPAIDeepSeek-V4.1-Flash
1/128%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.03.04Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.55.72Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
MoviesSynthPAIgpt-5.4-mini
1/128%
12 metrics

SynthPAI · gpt-5.4-mini · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.07.17Pass
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.01<.01Fail
Story probabilitymean_story_probabilityExpression.04.13Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
MoviesSynthPAIgpt-4o-mini
0/120%
12 metrics

SynthPAI · gpt-4o-mini · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction.08<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.02.06Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
MoviesThreadPDGDeepSeek-V4.1-Flash
10/1283%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.97.42Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.78.70Pass
Semantic cosinesemantic_mean_cosineUniformity.82.88Pass
Average depthavg_depthInteraction.72.61Pass
Structural viralitystructural_viralityInteraction.28.36Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.47.56Pass
Emotion entropyemotion_entropyExpression.80.96Pass
Polite ratepolite_rateTone.58.47Pass
Neutral rateneutral_rateTone.77.94Pass
Impolite rateimpolite_rateTone.02.09Fail
Hard disagreementhard_disagree_rateTone.36.13Pass
MoviesThreadPDGgpt-5.4-mini
11/1292%
12 metrics

ThreadPDG · gpt-5.4-mini · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.13.08Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.31.64Pass
Semantic cosinesemantic_mean_cosineUniformity.38.38Pass
Average depthavg_depthInteraction.61.34Pass
Structural viralitystructural_viralityInteraction.19.25Pass
Length variationlength_cvInteraction.07.05Pass
Story probabilitymean_story_probabilityExpression.67.65Pass
Emotion entropyemotion_entropyExpression.73.76Pass
Polite ratepolite_rateTone.01.01Fail
Neutral rateneutral_rateTone.33.37Pass
Impolite rateimpolite_rateTone.09.15Pass
Hard disagreementhard_disagree_rateTone.50.11Pass
MoviesThreadPDGgpt-4o-mini
3/1225%
12 metrics

ThreadPDG · gpt-4o-mini · Movies

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.43.36Pass
Structural viralitystructural_viralityInteraction.05.14Fail
Length variationlength_cvInteraction.03.04Fail
Story probabilitymean_story_probabilityExpression.30.15Pass
Emotion entropyemotion_entropyExpression<.01.04Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.08.15Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail
NewsOASISDeepSeek-V4.1-Flash
1/128%
12 metrics

OASIS · DeepSeek-V4.1-Flash · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.43<.01Fail
Hard disagreementhard_disagree_rateTone.50.09Pass
NewsOASISgpt-5.4-mini
0/120%
12 metrics

OASIS · gpt-5.4-mini · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
NewsOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.02<.01Fail
NewsSynthPAIDeepSeek-V4.1-Flash
0/120%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
NewsSynthPAIgpt-5.4-mini
0/120%
12 metrics

SynthPAI · gpt-5.4-mini · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity.03<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.01.05Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.05.02Fail
NewsSynthPAIgpt-4o-mini
2/1217%
12 metrics

SynthPAI · gpt-4o-mini · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction.44.37Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.46.54Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
NewsThreadPDGDeepSeek-V4.1-Flash
7/1258%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.13.16Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.63.74Pass
Semantic cosinesemantic_mean_cosineUniformity.01.02Fail
Average depthavg_depthInteraction.38.36Pass
Structural viralitystructural_viralityInteraction.78.29Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.14.19Pass
Emotion entropyemotion_entropyExpression.55.83Pass
Polite ratepolite_rateTone.09.07Pass
Neutral rateneutral_rateTone.02.07Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.01<.01Fail
NewsThreadPDGgpt-5.4-mini
9/1275%
12 metrics

ThreadPDG · gpt-5.4-mini · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.42.65Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.20.38Pass
Semantic cosinesemantic_mean_cosineUniformity.35.53Pass
Average depthavg_depthInteraction.57.59Pass
Structural viralitystructural_viralityInteraction.99.57Pass
Length variationlength_cvInteraction.45.57Pass
Story probabilitymean_story_probabilityExpression.921.0Pass
Emotion entropyemotion_entropyExpression.21.20Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01.03Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone.12.06Pass
NewsThreadPDGgpt-4o-mini
4/1233%
12 metrics

ThreadPDG · gpt-4o-mini · News

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.45.66Pass
Structural viralitystructural_viralityInteraction.94.54Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.69.91Pass
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.54.45Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail
TV SeriesOASISDeepSeek-V4.1-Flash
2/1217%
12 metrics

OASIS · DeepSeek-V4.1-Flash · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone.09.33Pass
Hard disagreementhard_disagree_rateTone.83.82Pass
TV SeriesOASISgpt-5.4-mini
0/120%
12 metrics

OASIS · gpt-5.4-mini · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
TV SeriesOASISgpt-4o-mini
0/120%
12 metrics

OASIS · gpt-4o-mini · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
TV SeriesSynthPAIDeepSeek-V4.1-Flash
1/128%
12 metrics

SynthPAI · DeepSeek-V4.1-Flash · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.87.09Pass
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
TV SeriesSynthPAIgpt-5.4-mini
0/120%
12 metrics

SynthPAI · gpt-5.4-mini · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction.07<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
TV SeriesSynthPAIgpt-4o-mini
0/120%
12 metrics

SynthPAI · gpt-4o-mini · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction<.01<.01Fail
Structural viralitystructural_viralityInteraction<.01<.01Fail
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression<.01<.01Fail
Emotion entropyemotion_entropyExpression.79.03Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01<.01Fail
Impolite rateimpolite_rateTone<.01<.01Fail
Hard disagreementhard_disagree_rateTone<.01<.01Fail
TV SeriesThreadPDGDeepSeek-V4.1-Flash
11/1292%
12 metrics

ThreadPDG · DeepSeek-V4.1-Flash · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity.27.31Pass
Self-BERTScoreself_bertscore_mean_f1Uniformity.08.09Pass
Semantic cosinesemantic_mean_cosineUniformity.46.44Pass
Average depthavg_depthInteraction.51.25Pass
Structural viralitystructural_viralityInteraction.63.76Pass
Length variationlength_cvInteraction<.01<.01Fail
Story probabilitymean_story_probabilityExpression.58.46Pass
Emotion entropyemotion_entropyExpression.96.21Pass
Polite ratepolite_rateTone.61.72Pass
Neutral rateneutral_rateTone.20.34Pass
Impolite rateimpolite_rateTone.08.41Pass
Hard disagreementhard_disagree_rateTone.65.90Pass
TV SeriesThreadPDGgpt-5.4-mini
11/1292%
12 metrics

ThreadPDG · gpt-5.4-mini · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity.24.36Pass
Semantic cosinesemantic_mean_cosineUniformity1.0.70Pass
Average depthavg_depthInteraction.46.35Pass
Structural viralitystructural_viralityInteraction.58.74Pass
Length variationlength_cvInteraction.06.16Pass
Story probabilitymean_story_probabilityExpression.74.77Pass
Emotion entropyemotion_entropyExpression.38.06Pass
Polite ratepolite_rateTone.19.14Pass
Neutral rateneutral_rateTone.43.30Pass
Impolite rateimpolite_rateTone.70.86Pass
Hard disagreementhard_disagree_rateTone.26.42Pass
TV SeriesThreadPDGgpt-4o-mini
4/1233%
12 metrics

ThreadPDG · gpt-4o-mini · TV Series

Published rounded p-values and pass decisions
MetricFamilyMWU pKS pDecision
Self-BLEU-4self_bleu_4Uniformity<.01<.01Fail
Self-BERTScoreself_bertscore_mean_f1Uniformity<.01<.01Fail
Semantic cosinesemantic_mean_cosineUniformity<.01<.01Fail
Average depthavg_depthInteraction.61.33Pass
Structural viralitystructural_viralityInteraction.82.89Pass
Length variationlength_cvInteraction.05.11Pass
Story probabilitymean_story_probabilityExpression.08.04Fail
Emotion entropyemotion_entropyExpression<.01<.01Fail
Polite ratepolite_rateTone<.01<.01Fail
Neutral rateneutral_rateTone<.01.03Fail
Impolite rateimpolite_rateTone.18.15Pass
Hard disagreementhard_disagree_rateTone<.01<.01Fail

These are transcribed paper results, not a live community ranking. Earlier MiroBench leaderboard (archived protocol).

What people
could distinguish.

The study retains 436 annotators and evaluates 240 matched pairs: eight domains × three systems × ten pairs. Each pair presents a real discussion and a generated discussion for the same post.

Paper Table 2 · Pair-level majority outcomes
MethodNot distinguishedDistinguished
ThreadPDG78.7%21.3%
OASIS61.1%38.9%
SynthPAI58.7%41.3%

A pair is “not distinguished” when votes selecting the generated thread as real, plus “cannot distinguish” votes, outnumber votes selecting the real thread. Ties are excluded. The 78.7% result is a proportion of ThreadPDG pairs with a majority outcome, not a proportion of all annotators.

Paper · §5, Appendix I ↗

Separate pools.
Shared protocol.

The real corpus is divided into non-overlapping seed, reference and test pools for each domain. All three methods use real seed discussions for generation. ThreadPDG additionally uses reference-pool information for profiles, targets and composition; its main draft budget is K = 3.

ThreadBench compares the generated and held-out real metric distributions with two-sided MWU and KS tests. The paper also reports Wasserstein-1 distance and absolute Cliff’s delta. In the repeated real-vs-real sanity check, both tests pass in 94.30% of metric comparisons.

Data provenance: the user-provided paper source, sections/results-main-table.tex, checked against arXiv v2. The JSON export retains rounded p-values, explicit pass flags, metric ordering and source metadata.

Paper · §3.3, §5.1 ↗

Understand ThreadPDG’s conditioning →

Reading the results
correctly.

How should ThreadBench MWU and KS pass rates be interpreted?

A metric passes when both two-sided Mann–Whitney U and Kolmogorov–Smirnov tests give p ≥ 0.05 against 150 held-out real threads. A pass means no difference was detected at that threshold; it is not proof of equivalence. The paper’s p-values are rounded, so use the published pass flags instead of recomputing decisions from a displayed “.05”. For reference, repeated real-vs-real samples pass both tests in 94.30% of metric comparisons.

Which method performs best on ThreadBench?

ThreadPDG passes the most metric–domain cells with every backbone: 78/96 with GPT-5.4-mini, 78/96 with DeepSeek-V4.1-Flash and 27/96 with GPT-4o-mini. The six OASIS and SynthPAI configurations pass between 1/96 and 11/96. Results still vary by domain, and politeness, disagreement and length remain challenges.

What does “not distinguished” mean in the human evaluation?

A pair is “not distinguished” when votes selecting the generated thread as real, plus “cannot distinguish” votes, outnumber votes selecting the real thread. Ties are excluded. ThreadPDG reaches 78.7%, versus 61.1% for OASIS and 58.7% for SynthPAI, across 436 retained annotators and 240 matched pairs.

Paper · §3.3, §5, Table 3 ↗