Illuminating Logical Fallacies with the CAMPFIRE Corpus

Misinformation detection remains today a challenging task for both annotators and computer systems.While there are many known markers of misinformation-e.g., logical fallacies, propaganda techniques, and improper use of sources-labeling these markers in practice has been shown to produce low agreement as it requires annotators to make several subjective judgments and rely on their own knowledge, external to the text, which may vary between annotators.In this work, we address these challenges with a collection of linguisticallyinspired litmus tests.We annotate a schema of 25 logical fallacies, each of which is defined with rigorous tests applied during annotation.Our annotation methodology results in a comparatively high IAA on this task: Cohen's in the range .69-.86.We release a corpus of 12 documents from various domains annotated with fallacy labels.Additionally, we experiment with a large language model baseline showing that the largest, most advanced models struggle on this challenging task, achieving an F1-score with our gold standard of .08 when excluding non-fallacious examples, compared to human performance of .59-.73.However, we find that prompting methodologies requiring the model to work through our litmus tests improves performance.Our work contributes a robust fallacy annotation schema and annotated corpus, which advance capabilities in this critical research area.