Generating Actionable Insights from Integration Testing Data and Product Documentation in B2B SaaS Using ChatGPT

In the Software as a Service (SaaS) industry, web APIs are a common method for enabling integrations between a SaaS provider and its customers. Ensuring good customer API integrations is important for optimal product performance. However, achieving an optimal API integration is often challenging and time-consuming, due to the complexity of these systems. While previous research has focused on developing domain-specific chatbots, less focus has been on how LLMs can be integrated into existing enterprise workflows to generate actionable insights. This research contributes to addressing that gap by designing, building, and evaluating an artifact that generates natural language insights from integration test data and product documentation. Design Science Research is used as the methodology, supported by semi-structured interviews with domain experts. Findings indicate that the artifact has potential to provide valuable insights based on integration tests, though the quality of the insights can vary. This thesis shows that LLMs, using Retrieval-Augmented Generation, can generate actionable insights to help business-to-business SaaS providers improve customers’ API integrations. Further research is needed to explore the full potential and limitations of such systems.