EXPLOITING CALM ATTENTION IN VISION TRANSFORMERS, CALM-VIT

Images are a key component in Artificial Intelligence, enabling tasks like high-quality image creation, image translation, and text-to-image synthesis. These methods are vital for multimodal AI systems, such as ChatGPT, which combines visual and textual data for context-aware interactions. Such systems are not without their negative qualities, including computational overhead, overfitting of data, and complete collapse in functionality. The increasing adoption of AI has only exacerbated issues with increasing demand and an influx of generated information to the pool of information. We implemented a novel yet powerful method for optimizing AI workloads using CALM attention. By extending a Vision Transformer (ViT) to incorporate Variational Autoencoder principles, image data is compressed & expanded into Latent Space, allowing attention mechanisms to operate on lower-dimensional representations. Integrating latent space with Cross-Axial Attention to deep residual connections and learned masking improves the performance of ViTs and demonstrates the potential of hybrid transformer architectures in advancing AI applications. We call this architecture CALM-ViT. Our proposed enhancements allow for better exploitation of latent space within the image domain. Further work can expand for tasks in other domains, such as textual and audio generation, potentially enriching them using latent representations. Even if our architecture is not ideal, researching optimal methods for synergizing latent space with attention mechanisms is crucial for continual model improvement.