Web Scraping for Business Intelligence: Analyzing Higher Education Courses in Brazil

One of the most significant challenges facing higher education institutions is the continual improvement of teaching quality. This is particularly evident in Brazil, where despite the growing number of students enrolled in undergraduate courses, a high dropout rate persists. Among the various factors contributing to this phenomenon, the quality of the courses plays a pivotal role. In light of the aforementioned considerations, this study presents the QualiAcad tool, which performs data mining through web scraping of the Guia da Faculdade survey. This survey evaluates higher education courses in Brazil, employing the following criteria: (1) Pedagogical Project; (2) Faculty; and (3) Infrastructure. QualiAcad is utilized to scrape data from this research and generate dashboards to analyze the evaluation of higher education courses from a large educational group in Brazil and its competitors. The scraping process was conducted over a five-year period, providing educational group managers with a historical and analytical view of the quality of their courses. This allows for the formulation of plans to contain evasion and enables historical comparisons with competitors to determine more effective competitive strategies. This approach not only improves analyses but also provides robust insights to guide better-informed future decisions.