Initial Reliability of a Special Education Law Chatbot

There is a shortage of school psychologists across the nation (National Association of School Psychologists, n.d.), resulting in a need for practitioners to perform all aspects of their jobs as efficiently as possible to avoid burnout while ensuring adequate service delivery. While school psychologists are trained in 10 Domains of practice, Domain 10: Legal, Ethical, and Professional Practice, permeates all aspects of school psychological practice. Specifically, school psychologists are “state actors,” meaning they are an extension of their state’s authority, supported by state and federal laws, to educate and serve school-aged children, who are among our most vulnerable populations (NASP, 2020). As such, all aspects of school psychological practice heavily rely on the interpretation and application of federal and state special education laws (SpEd Law) to ensure processes, procedures, and services remain in compliance with these regulations. However, SpEd Law can often be cumbersome to navigate due to nuanced differences from state to state, ongoing changes to interpretation due to updated case law and legislative updates, and the cumbrous nature of legal documentation. Like many professionals, school psychology practitioners have explored technology-based tools, such as Artificial Intelligence (AI), as efficiency solutions to the growing shortage despite increasing demand for services (Krach & Corcoran, 2023). Generative AI and large language models (Zewe, 2023; Stöffelbauer, 2023) may offer a means for school psychologists to more efficiently navigate SpEd law. To that end, an FHSU team of school psychologists developed a chatbot, Kansas SpEd Law, to assist in navigating federal and Kansas special education law. Although casual interaction with the chatbot yielded output that appeared reliable, a more in-depth investigation of accuracy yielded an unacceptable amount of error for practical application of the chatbot. An overview of the chatbot, analysis of initial accuracy testing, and considerations for future development are offered.