Colorbot: Interactive Smartphone Assistant for Color Blindness Detection and Realtime Assistance using LLM-based Chatbots and Machine Learning

Color blindness, or color vision deficiency (CVD), impacts a large percentage of the population, making it difficult for them to differentiate between some colors. In addition to difficulties in everyday activities such as understanding traffic lights or recognizing colored objects, people with CVD usually experience social embarrassment, miscommunication, or even humiliation when incorrectly identifying colors in work, school, or social environments. This research puts forth an interactive smartphone app meant to grant color-blind individuals autonomy, self-esteem, and a dependable aid in facing everyday problems without needing the help of others. The application incorporates sophisticated color recognition and live help capabilities using the Ishihara test for customized CVD analysis. A tailored, interactive, real-time chatbot based on Gemini and Chat GPT-4 Vision models facilitates color recognition, while traffic light detection offers added road safety. The solution employs image processing techniques and machine learning models, developed using the Flutter framework and FastAPI backend, and is protected with Firebase Authentication. Using user testing and feedback, we assess the app's efficiency in offering accurate color detection and helpful support. The findings show that our method is a user-friendly, effective, and accessible solution for the color-blind, paving the way for future advancements in assistive technology.