RAM: Retrieval Augmented Modelling for Tabular In-Context Few-Shot Domain Adaptation

Transformer architectures have shown great success in natural language processing, sparking interest in their applications on tabular data.However, the potential of using transformer-like architectures for incontext domain adaptation in tabular settings remains underexplored.We introduce Retrieval-Augmented Modelling (RAM), a compact attentionbased architecture specifically designed for this task.RAM utilises a Domain-Aligned Memory training strategy, which ensures that it always processes the data from the same domain at each training step, allowing the model to focus on domain-specific patterns.Evaluated on synthetic data simulating domain shifts, RAM outperforms traditional machine learning models, effectively adapting to unseen domains.