Trustworthiness in AI: on SciCompBot—the scientific computing chatbot

This paper explores the prospect of a scientific computing chatbot (SciCompBot): an artificial intelligence (AI) system designed to address advanced problems in optimization, spectral computations (eigenvalues, eigenvectors, spectra), differential equations, inverse problems and related areas. With powerful chatbots like ChatGPT-5.2, it is no longer science fiction for AI to accept these problems; ChatGPT-5.2 does so willingly. Yet, current chatbots suffer from a critical flaw: they often produce incorrect answers with unwarranted confidence, making them fundamentally> untrustworthy for scientific computing. We investigate the mathematical challenges involved in developing a trustworthy SciCompBot and demonstrate that tools from the foundations of computational mathematics, particularly the solvability complexity index (SCI) hierarchy, are necessary in this endeavour. Our results present new strategies for overcoming central obstacles for building a trustworthy SciCompBot: the issues of non-computability and the phenomenon of generalized hardness of approximation (GHA). We identify a key feature that a trustworthy SciCompBot must have: it must be 'chatty'. That is, it must initiate a dialogue with the user. This work fits in the broader programme of ensuring trustworthiness and safety in AI systems, which is now a major topic of interest owing to explosive growth in AI development in recent years. This article is part of the theme issue 'Safe, secure and robust AI for safety-critical systems'.