Research on the Competency of Talents in Big Data Management and Application Based on Text Mining

In the digital and intelligent era, new technologies such as big data, artificial intelligence, cloud computing, 5G, and the Internet of Things are constantly developing, and the demand for talents in data management and application is increasing day by day. This paper aims to explore the market's demand for talents in big data management and application, thereby providing directions and ideas for talent cultivation in schools. This paper crawls recruitment information related to big data management and application from 51job.com, conducts text mining on the collected 1,710 pieces of data, and uses the LDA topic analysis and TF-IDF model to analyze the knowledge and skills that need to be mastered. The research findings show that the knowledge and skill requirements for talents in big data management and application reflect cross-disciplinary commonalities and skill diversity, mainly including professional skills such as data analysis, data mining, databases, visualization, modeling, and algorithms, as well as core project management capabilities.