Dimensionality Reduction for Enabling Visual Reinforcement Learning with a Classifier System

Originally, the XCS Classifier System, seen as the most popular and most researched Learning Classifier System (LCS), was designed to learn and solve Reinforcement Learning (RL) problems. As has become clear over time, known weaknesses and the curse of dimensionality of Michigan-style LCSs severely limit its applicability to RL problems. As a result, visual RL problems generally appear to be outside the scope of XCS variants. We target this class of RL benchmarks by combining XCSF, XCS with hyperrectangle conditions and computed linear prediction, with dimensionality reduction methods. In particular, we combine deep variational autoencoders (VAE) with XCSF using an automatic dataset collection scheme and offline training of the VAE and online training for the combined system to target several visual RL problems. Our results show that XCSF is able to learn visual RL problems in the latent space of deep VAEs. Further experiments confirm that simple downscaling of images can also enable XCSF to learn several visual RL problems. Our results do not only indicate that XCSF is better at RL than its reputation, but show that significantly more RL problems might be within the scope of XCSF and motivate further investigation of dimensionality reduction methods for LCSs.