Model Weights Reflect a Continuous Space of Input Image Domains

In recent years, there has been a growing body of work around Domain Adaptation (DA) and Domain Generalisation (DG). However, most rely on empirical testing to find new ways of generating invariant features from one domain to another. In reality, DA methods must deal with many more domains that vary continuously from one to the next, such as coping with adverse weather conditions in au-tonomous driving. This work shows that it is better to have experts for each of these specific domains, but this is pro-hibitively expensive (e.g. a model per level of rain combined with each level of darkness). To see whether these experts can be generated from only a few models, this work explores the model weight space and the relationship with respect to varying input data. Analysis indicates that there is a rich and continuous structure in weight space that could be used to find such experts. However, existing linear methods lack the flexibility to capture this information. We demon-strate this using intuitive synthetic data and confirm this in real-world data under adverse weather conditions. Initial steps are taken using non-linear methods, which are better equipped to represent these style variations. The findings have the potential to improve model training under DG and DA, as well as in distributed training and model merging.