Modeling user autonomy in recommender systems using Markov perturbation-based multi-armed bandits

Recommender systems play a critical role in e-commerce, streaming media, and other fields, where user engagement is vital to platform success. These systems, however, face challenges like the cold start problem, which limits recommendation accuracy for new users and items due to sparse data. Although multi-armed bandit (MAB) algorithms are considered effective for balancing exploration and exploitation, existing studies lack simulations of algorithm performance in scenarios where recommender systems indirectly influence user choices, limiting insights for performance improvements in real-world applications. This study explores the performance of key MAB algorithmsExplore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS)under a novel framework that models user behavior as an unreliable intermediate using Markov state transitions. This Markov perturbation-based approach provides a foundation for indirectly influencing user decisions. Two algorithmic frameworks are examined: one considering only the recommendation system's influence on user choices, and another also considering user feedback as a dynamic influence on state transition probabilities. Results show that, while considering user feedback increases adaptability, it also causes greater volatility. TS achieves the lowest cumulative regret with strong stability, while Asymptotic-UCB achieves a better balance between exploration and exploitation than standard UCB, with consistent adaptability under varying feedback. In contrast, ETC demonstrates significant volatility, highlighting its limitations in feedback-driven scenarios. These findings provide valuable insights for enhancing MAB algorithms in real-world cold start recommendation applications, especially where systems influence user choices indirectly.