Interpreting text corpora from androids-related stories using large language models: “Machines like me” by Ian McEwan in generative AI

Ian McEwan’s “Machines like me” presents a scenario where androids demonstrate a limited understanding of the world. In this study, we feed the content of “Machines like me” novel into a chatbot powered by LLMs to analyze how it interprets various questions posed as prompts. We create a private chatbot based on OpenAI API using in-context learning and LangChain orchestration for evaluation. Our focus is on assessing the chatbot’s accuracy in responding, its level of world understanding, and the consistency of its answers. Our findings reveal that chatbots, similar to the android in the novel, still exhibit a lack of world understanding. This limitation, coupled with their tendency to produce hallucinatory responses, can hinder their ability to correctly interpret and respond to textual input. Further, we compare the private chatbot with ChatGPT-4 and evaluate the results.