VisHintPrompt : A Visual Hint Prompting Strategy for Numerical Inference from Charts

Numerical inference from charts underpins many visualization tasks, including chart question answering and chart redesign. Although many chart-specific models have been developed for numerical inference, their reliance on the chart types and visual encodings represented in training data limits their generalizability. Modern Multimodal Large Language Models (MLLMs) possess competitive general-purpose visual understanding, offering the potential for numerical inference without additional training. Yet how to reliably elicit such fine-grained numerical inference from pretrained MLLMs via visual prompting remains largely unexplored. To address this gap, we propose VisHintPrompt, a scaffolded visual hint prompting strategy grounded in the explicit extraction of coordinate priors from both Cartesian and polar charts. VisHintPrompt elicits the latent numerical inference abilities of MLLMs by progressively narrowing the search region. It supplies structured visual cues that guide the model toward accurate quantitative interpretation. The strategy consists of three coordinated components: Coordinate-aware Grid Enhancement, which extracts coordinate priors and exposes them as structure-aligned grids; Iterative Visual Feedback, which overlays alignment cues derived from intermediate predictions to enhance the next prediction; and Progressive Zoom-in Refinement, which enlarges the region of interest and applies progressive grid densification to enhance local detail for precise value estimation. VisHintPrompt is visual-centric and applies to both Cartesian charts (e.g., bar, scatterplot) and polar charts (e.g., pie, radar). Experiments across diverse chart types on both proprietary and open-source MLLMs demonstrate improvements over standard prompting. Quantitative and qualitative analyses, along with ablation studies, validate the effectiveness and complementary roles of the individual components. These results underscore the effectiveness of VisHintPrompt in enabling reliable and more accurate numerical inference from charts.

Authors

Institutions

Publication Details

Journal
IEEE Transactions on Visualization and Computer Graphics
Published
2026-11-18
DOI
https://doi.org/10.1109/tvcg.2026.3754816
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

VisHintPrompt : A Visual Hint Prompting Strategy for Numerical Inference from Charts

Yigang Wang, 彭勇乐, Zhiguang Zhou, Yong Wang et al.
1 links
IEEE Transactions on Visualization and Computer Graphics
Multimodal Machine Learning Applications
article

VisHintPrompt : A Visual Hint Prompting Strategy for Numerical Inference from Charts

Yigang Wang, 彭勇乐, Zhiguang Zhou, Yong Wang, Fengling Zheng, Dekun Qian, Chenyun Cai, Yumeng He, Zi Rong
article en

Abstract

Numerical inference from charts underpins many visualization tasks, including chart question answering and chart redesign. Although many chart-specific models have been developed for numerical inference, their reliance on the chart types and visual encodings represented in training data limits their generalizability. Modern Multimodal Large Language Models (MLLMs) possess competitive general-purpose visual understanding, offering the potential for numerical inference without additional training. Yet how to reliably elicit such fine-grained numerical inference from pretrained MLLMs via visual prompting remains largely unexplored. To address this gap, we propose VisHintPrompt, a scaffolded visual hint prompting strategy grounded in the explicit extraction of coordinate priors from both Cartesian and polar charts. VisHintPrompt elicits the latent numerical inference abilities of MLLMs by progressively narrowing the search region. It supplies structured visual cues that guide the model toward accurate quantitative interpretation. The strategy consists of three coordinated components: Coordinate-aware Grid Enhancement, which extracts coordinate priors and exposes them as structure-aligned grids; Iterative Visual Feedback, which overlays alignment cues derived from intermediate predictions to enhance the next prediction; and Progressive Zoom-in Refinement, which enlarges the region of interest and applies progressive grid densification to enhance local detail for precise value estimation. VisHintPrompt is visual-centric and applies to both Cartesian charts (e.g., bar, scatterplot) and polar charts (e.g., pie, radar). Experiments across diverse chart types on both proprietary and open-source MLLMs demonstrate improvements over standard prompting. Quantitative and qualitative analyses, along with ablation studies, validate the effectiveness and complementary roles of the individual components. These results underscore the effectiveness of VisHintPrompt in enabling reliable and more accurate numerical inference from charts.

IEEE Transactions on Visualization and Computer GraphicsVol. PP
University of Southern California (US), Nanyang Technological University (SG), Hangzhou Dianzi University (CN), Beihang University (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon