Overview of the CLEF 2024 SimpleText Task 4: SOTA? Tracking the State-of-the-Art in Scholarly Publications

This talk offers an overview of the CLEF 2024 SimpleText Task 4 on SOTA? Tracking the State-of-the-Art inScholarly Publications, asking systems to perform two tasks: 1) classification – given the full text of an AI scientific paper, classify whether the paper indeed reports model scores on benchmark datasets, and if so, 2) information extraction – extract all pertinent (Task, Dataset, Metric, Score) tuples from the content of the scientific paper to automatically populate leaderboards used to keep track on the latest and greatest AI models. The details of the task set-up are given. First, the “SOTA?” task corpus comprising over 14K AI scientific papers, their corresponding annotations, and detailed corpus statistics. Second, the Evaluation Metrics used and the online Codalab Evaluation Platform to accept participant submissions. Third, the Results of the runs submitted by our participants.