Benchmark
Why this benchmark exists
Signal treats evaluation as falsification, not aesthetics. This page records where models fail, how failures appear in outputs, and what safeguards are applied so analyst decisions remain transparent and auditable.
Failure modes and mitigations
| Area | Known limitation | Current mitigation |
|---|---|---|
| Stance detection | Zero-shot NLI marks most informal posts as ambiguous. | Expose ambiguity as a finding and pair with blind-spot examples + subreddit priors. |
| Coordination | Temporal+URL synchrony can include non-malicious cross-post behavior. | Show confidence labels, normalization mode, and explicit innocent explanations. |
| Trend ranking | Viral score can overweight one-off posts. | Show score + velocity + spread + post_count rationale side by side. |
| Origin inference | First observed subreddit may differ from true narrative birthplace. | Report confidence and time-to-mainstream with explicit uncertainty tier. |