Solution-down procedural content generation via reinforcement learning for hyper-casual puzzle games using data-driven difficulty estimation

Procedural content generation via reinforcement learning (PCGRL) has shown promising results in generating solvable levels for puzzle games. Still, little work has been done in exploring its ability to create engaging levels. In this work, we investigate the use of PCGRLinconjunction with difficulty estimation to create engaging levels by making them more challenging. We also propose a novel process for generating levels, called solutiondown generation, that guarantees solvability, allowing the model to focus on learning diff iculty rather than solvability. To test and compare these methods, we implemented four generators: one that uses random selection and a puzzle-up process, another that also uses random selection but our solution-down process, a third that is based on the original Turtle PCGRL method, and one that uses our proposed solution-down PCGRL method (SDPCGRL). All the generators were developed to generate levels for the hyper-casual puzzle game Longcat, created by Fancade. In this game, the player attempts to fill different boxes with an elongating cat. Its hyper-casual nature allowed it to have a manageable number of states and actions, and its popularity provided us with a substantial amount of data. Wefirst implemented a difficulty predictor to evaluate and train the models for generating engaging and challenging puzzles. This predictor is based on a previous method for estimating difficulty in casual puzzle games and was trained on player data collected from the Longcat application. We conducted an analysis phase before developing the generator to use this data properly. After the generators were developed and/or trained, we tested them by having each generate 20,000 levels. We then compared the solvability, number of duplicates, and difficulty of these levels. We found that both puzzle-up approaches struggled significantly with generating solvable levels, with the Turtle PCGRL generator unable to generate any solvable levels. We found that the solution-down random generator had fewer duplicates than the SD-PCGRL method; however, the vast majority of these levels were of trivial difficulty. In conclusion, the SD-PCGRL generator showed the best result in creating solvable, unique, and challenging levels for Longcat.