User Experience Analysis when Finding Information: Comparative Study of RAG and Traditional Search Engine

The rise of natural language processing techniques, especially large language models (LLM), has driven innovations in various areas. In the realm of information retrieval, LLMs have emerged as an alternative solution to traditional search engines when combined with retrieval-augmented generation (RAG). This study aimed to analyze the system usability scale (SUS) to compare the perplexity-a RAG application focused on information retrieval-with Google. The study involved 120 respondents with IT-related job backgrounds and used a 10-question survey based on a five-point Likert scale. The results showed that the SUS score for Google Search Engine was 82.29, while the SUS score for Perplexity AI was 73.98. Although Google had a higher score than Perplexity AI, both platforms are considered acceptable in terms of user experience. Google's SUS score indicates a higher level of user satisfaction and usability. ANOVA and Cronbach's alpha were used to measure the significance and reliability of the SUS analysis based on the questionnaire results. Both platforms-Google Search Engine and Perplexity AI-demonstrated acceptable internal consistency, with Cronbach's Alpha scores of 0.7787 for Google and 0.7782 for Perplexity AI. The study found that Google consistently outperformed Perplexity AI in terms of usability, task performance, and user satisfaction, mainly due to its familiar and straightforward design. While Perplexity AI offers innovative features, its steeper learning curve and complexity resulted in lower usability scores. Despite these challenges, both platforms achieved “acceptable” ratings on the SUS scale, indicating a baseline level of usability.