The Black Stories Experiment: Two Groups are Trying to Solve a Riddle Game Behind a Screen, Only One Group Is Alive

Studying large language models (LLMs) can provide valuable insights into their strengths and limitations. This study explores problem-solving capabilities of GPT-4 by comparing the model's performance in solving Black Stories riddles, to human performance. The study utilized a set of 12 adjusted Black Stories, each tested twice within the human and GPT-4 group. The experiment was conducted through text messaging for a comparable set-up. The primary measure of performance was the number of questions and hints needed to solve the riddle. Results indicated no significant difference between the groups. Qualitative results showed that GPT-4 excelled in precise questioning and creativity but often fixated on details. Humans covered broader topics and adapted the focus quickly but struggled with uncommon details. This research suggests that despite different approaches, GPT-4's performance was comparable to that of humans, demonstrating its potential as a capable participant in these types of problem solving games.