Real-time human–machine artistic collaboration generation driven by non-explicit usage needs triggered by multimodal behaviors

The purpose is to improve the problem that the existing image generation methods are difficult to achieve stable, small-scale and low-delay update according to the continuous editing operation of users in the real-time man-machine art collaborative generation scene. The study object is multimodal operation behavior data and its corresponding generation context state. Firstly, in the experimental environment of man-machine art collaborative generation, based on COYO-700M ( https://github.com/kakaobrain/coyo-dataset?utm_source=chatgpt.com ) data set, the study constructs a continuous editing operation sequence, and generates multimodal operation behavior data for generating adjustment through operation sequence playback. The data includes three kinds of information: continuous editing operations and their positions, the time sequence and interval of operations, and the local visual content of the modified area in the currently generated image. Secondly, according to the iterative characteristics of diffuse image generation, a generation adjustment mechanism based on operation behavior is introduced. In the process of generation iteration, the local update area and update intensity are conditionally controlled: the user’s operation position is used to define the spatial range of generation update. The operation frequency and order are used to adjust the local resampling intensity, so that the generated results can realize gradual and stable local changes with continuous operation, rather than overall regeneration. Finally, the online generation process for real-time collaborative scenarios is constructed, and the overhead of repeated calculation is reduced through local iterative updating and reasoning result caching. The results show that the generation adjustment mechanism based on multimodal operation behavior significantly improves the stability and controllability of generation in continuous editing scenes. Compared with the whole regenerated baseline, the variation amplitude of pixels outside the local update area is reduced by 12.4%, and the consistency of adjacent generation results is improved by 15.1%. Additionally, the local iterative updating and caching mechanism can control the average response delay within 68 ms and reduce the overall regeneration times by 23.7%, which verifies the efficiency advantage of this method in real-time man-machine art collaborative generation. This method can effectively alleviate the instability and efficiency problems of existing generation methods under continuous interaction. This study provides a reusable technical idea for the direct adjustment of the generation process through operational behavior in the generation of man-machine art cooperation.