Application of ViT models with hyper attention mechanism on skin cancer classification

The ViT model has excellent performance in image classification tasks and has been applied in several fields. However, ViT is difficult to be retrained and applied under limited hardware conditions and data due to the large number of parameters and high computational complexity of self-attention. To solve this problem, this paper introduces an embedded hyper-attention layer to replace the original self-attention layer in skin cancer image classification, aiming to improve computational efficiency by reducing the parameters. In practice, the parameters of the Densenet121 ViT model are greatly reduced with comparable accuracy. Our study shows the potential for improving model classification efficiency in hardware-constrained environments, making it possible to efficiently train and apply ViT models in such situations.