Testing AI Chatbot Instructions in Simulation: Where ChatGPT, Gemini, and Claude Misstep When Guiding Arduino Beginners

When beginners turn to artificial intelligence (AI) chatbots for Arduino help, it is unclear whether wiring and code work when a beginner follows every step exactly. To evaluate this, I tested ChatGPT, Gemini, and Claude across a range of beginner projects. The free versions were tested on 12 projects across 4 difficulty levels in Wokwi, a free online simulator, and I followed all instructions exactly. For tests that did not work as given, I sent up to 5 symptom-only complaints and repeated those pairs, so there were 52 counted tests. I also performed 5 tests on real hardware. Out of 36 initial tests, 28 (78%) worked as given, with ChatGPT and Gemini getting 9 out of 12 and Claude getting 10 out of 12. All 36 initial tests worked in the end, though across these 52 counted tests, 1 test never worked. The 22 failures came from part assumptions, specifically button legs (9), display type (9), and buzzer type (4). No fix created new problems, nothing was unsafe, and real hardware matched Wokwi in all 5 tests. The main problem is that chatbots assume specific part types or leg connections without telling the beginner. This makes it hard to tell what went wrong. To prevent this, beginners should name the exact part they are using, and chatbots should state their assumptions or ask questions before giving instructions.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24321888
Primary Topic
Teaching and Learning Programming
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Testing AI Chatbot Instructions in Simulation: Where ChatGPT, Gemini, and Claude Misstep When Guiding Arduino Beginners

Abdul Rahman Siddiqi
2 links
Zenodo (CERN European Organization for Nuclear Research)
Teaching and Learning Programming
preprint

Testing AI Chatbot Instructions in Simulation: Where ChatGPT, Gemini, and Claude Misstep When Guiding Arduino Beginners

Abdul Rahman Siddiqi
preprint en

Abstract

When beginners turn to artificial intelligence (AI) chatbots for Arduino help, it is unclear whether wiring and code work when a beginner follows every step exactly. To evaluate this, I tested ChatGPT, Gemini, and Claude across a range of beginner projects. The free versions were tested on 12 projects across 4 difficulty levels in Wokwi, a free online simulator, and I followed all instructions exactly. For tests that did not work as given, I sent up to 5 symptom-only complaints and repeated those pairs, so there were 52 counted tests. I also performed 5 tests on real hardware. Out of 36 initial tests, 28 (78%) worked as given, with ChatGPT and Gemini getting 9 out of 12 and Claude getting 10 out of 12. All 36 initial tests worked in the end, though across these 52 counted tests, 1 test never worked. The 22 failures came from part assumptions, specifically button legs (9), display type (9), and buzzer type (4). No fix created new problems, nothing was unsafe, and real hardware matched Wokwi in all 5 tests. The main problem is that chatbots assume specific part types or leg connections without telling the beginner. This makes it hard to tell what went wrong. To prevent this, beginners should name the exact part they are using, and chatbots should state their assumptions or ask questions before giving instructions.

Zenodo (CERN European Organization for Nuclear Research)
Teaching and Learning Programming
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon