Edge-Collaborative Model Reuse for Low-Latency AI Task Scheduling with Attention-based Soft Actor-Critic

The increasing use of AI and large language models has driven the deployment of Generative AI (GenAI) in cloud data centers. However, the inherent latency and high resource costs associated with large AI models are problematic for latency-sensitive edge users. While edge server deployment can reduce transmission times, it often leads to underutilized resources and suboptimal trade-offs between inference latency and quality. This paper introduces an Edge-Collaborative AI Task scheduling (CAT) algorithm that combines model reuse to reduce latency. We address AI task scheduling for edge servers by segmenting tasks and distributing patches, considering model distribution and cold starts. We propose a soft actor-critic-based algorithm, CAT, which uses an attention layer to learn complex edge system dynamics. Simulations in a task scheduling environment demonstrate that CAT reduces inference latency by up to 25% compared to baseline methods.