Lighthouse Bot: A Platform for Evaluating LLMs for Agentic Maritime Data Analysis

This study evaluates whether open-source Large Language Models (LLMs) can serve as practical alternatives to proprietary commercial models for AI-driven maritime data analytics. We compared multiple open-source LLMs against a commercial alternative using a custom-built evaluation framework with synthetic test cases derived from real maritime ferry data, employing Retrieval-Augmented Generation (RAG) systems and RAGAS/custom evaluation metrics. Results demonstrate a performance difference, though this varies by query complexity and model size.