StoryCrafter: Instance-Aligned Multi-Character Storytelling with Diffusion Policy Learning and Test-Time Reinforcement Learning

Open-ended visual storytelling poses a significant challenge for text-to-image models, which often fail to maintain narrative coherence and consistent character appearance across a sequence of images. To address this, we introduce StoryCrafter, a multi-character diffusion model that employs a novel instance-level cross-attention module. This module, trained with supervised fine-tuning, ensures precise alignment between text descriptions and characters, enabling consistent multi-character interactions throughout a story. Furthermore, we propose Direct-Diffusion Group Relative Policy Optimization, a novel reinforcement learning stage that uses story-aligned rewards to select optimal frames from a generated group, mitigating accumulated inconsistencies. We show that this group selection capability can be seamlessly integrated into inference time, effectively ensuring multi-frame style and character consistency, outperforming standard static greedy generation. We evaluate our approach through automated metrics, human assessments and vision-language model (VLM) judgment, measuring text-to-image alignment, style and character consistency, and fine-grained detail quality. Experiments on three benchmarks demonstrate that StoryCrafter outperforms existing methods, achieving 5.2% improvements in storytelling consistency on StorySalon dataset and 3.3%-10.8% in character accuracy on Flintstones-SV and Pororo-SV datasets, and also outperforming baselines in both human and VLM evaluations. The TTRL further brings consistent improvements in human evaluation on story visualization and continuation tasks, improving the average rank from 2.79 to 1.80 and from 1.80 to 1.39, respectively.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Intelligent Systems and Technology
Published
2026-11-16
DOI
https://doi.org/10.1145/3856814
Primary Topic
Generative Adversarial Networks and Image Synthesis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

StoryCrafter: Instance-Aligned Multi-Character Storytelling with Diffusion Policy Learning and Test-Time Reinforcement Learning

Hengyang Lu, Ruiqi Dong, Chenjie Pan, Chenyou Fan et al.
ACM Transactions on Intelligent Systems and Technology
Generative Adversarial Networks and Image Synthesis
article

StoryCrafter: Instance-Aligned Multi-Character Storytelling with Diffusion Policy Learning and Test-Time Reinforcement Learning

Hengyang Lu, Ruiqi Dong, Chenjie Pan, Chenyou Fan, Wenjing Pang, Haishan Sun, Ruihan Wu, Rui Zhang
article en

Abstract

Open-ended visual storytelling poses a significant challenge for text-to-image models, which often fail to maintain narrative coherence and consistent character appearance across a sequence of images. To address this, we introduce StoryCrafter, a multi-character diffusion model that employs a novel instance-level cross-attention module. This module, trained with supervised fine-tuning, ensures precise alignment between text descriptions and characters, enabling consistent multi-character interactions throughout a story. Furthermore, we propose Direct-Diffusion Group Relative Policy Optimization, a novel reinforcement learning stage that uses story-aligned rewards to select optimal frames from a generated group, mitigating accumulated inconsistencies. We show that this group selection capability can be seamlessly integrated into inference time, effectively ensuring multi-frame style and character consistency, outperforming standard static greedy generation. We evaluate our approach through automated metrics, human assessments and vision-language model (VLM) judgment, measuring text-to-image alignment, style and character consistency, and fine-grained detail quality. Experiments on three benchmarks demonstrate that StoryCrafter outperforms existing methods, achieving 5.2% improvements in storytelling consistency on StorySalon dataset and 3.3%-10.8% in character accuracy on Flintstones-SV and Pororo-SV datasets, and also outperforming baselines in both human and VLM evaluations. The TTRL further brings consistent improvements in human evaluation on story visualization and continuation tasks, improving the average rank from 2.79 to 1.80 and from 1.80 to 1.39, respectively.

ACM Transactions on Intelligent Systems and Technology
Jiangnan University (CN), South China Normal University (CN)
Generative Adversarial Networks and Image Synthesis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon