Grid Search and LLM Assisted Hybrid Approach for Hyperparameter Optimization

AI has become one of the technological stepping stones in most industry verticals including BFSI. The use of AI for tasks like credit risk assessment, fraud detection, and customized client experiences has been become a norm in BFSI space. However, developing AI/ML based decision systems for such applications involves spectrum of tasks such as data gathering and analysis, feature selection, tuning models via hyperparameter optimization and finally testing and deployment. In this paper, we explore hyperparameter optimization with feature analysis for credit risk assessment application in the BFSI industry. Typically, GridSearchCV and Bayesian Optimization are widely acceptable and reliable methods for hyperparameter optimization. Recently, it was proposed that LLMs can be used for optimization problems. Here, we look at a hybrid approach with the structured search capabilities of GridSearchCV in tandem with the optimization strength of LLMs. In the proposed approach, we perform the coarse-grain grid search and use the collected data as input for the LLM. Based on these data, we further prompt LLM to analyze and draw and propose a finer search space. Sampling from this finer search space results in more efficient and accurate optimization. This shows the effectiveness of LLMs to refine the search. Further, we explore feature analysis methods like SHAP values, correlation, significance distribution, etc. and identify important features that are helpful in better modeling and decision explanations.