Agent Pointed Context: letting a person point an AI agent at what they see, without sharing the screen

People who work with an AI agent beside an application constantly say "this": this row, this number, this chart. When the agent lives outside the application, in a chat client connected over a tool protocol, it has no way to know what "this" is, and the person has to describe or paste it. We present Agent Pointed Context (APC), an interaction pattern in which the application exposes three small channels to any connected agent: what the person is looking at (the view), what the person has pointed the agent at (the pin), and what each agent is reading or changing on screen (presence). The application hosts no model and no chat. It shares identifiers and keys only, never pixels, text or amounts; the agent reads the facts behind a pointer through the tools it already has, so every read passes the same authorization as any other call. Because a pointer resolves against the data that produced what the person sees, not against the rendering, the agent answers from the inputs behind a figure (the accounts, statements and transactions that make it up) rather than from a description of it. A pin reaches the agent without repeated model-driven polling, piggybacked on its next tool result, and an unattended agent can wait for one. Pointers resolve not only to records but to parts of computed figures, which the server recomputes with the same functions the page uses, so that for the same data snapshot, parameters and authorization scope the agent's number is the screen's number, and the response says when the scope differs. We describe a reference implementation over the Model Context Protocol (MCP) in a personal finance ledger, where 109 elements of the home page alone, from rows to single steps of computed figures, are pointer targets (a coverage count, not a measured benefit), and discuss the design space, the privacy argument and what is left to evaluate.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-15
DOI
https://doi.org/10.5281/zenodo.23382908
Primary Topic
Usability and User Interface Design
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Agent Pointed Context: letting a person point an AI agent at what they see, without sharing the screen

Tomás Neira
Zenodo (CERN European Organization for Nuclear Research)
Usability and User Interface Design
preprint

Agent Pointed Context: letting a person point an AI agent at what they see, without sharing the screen

Tomás Neira
preprint en

Abstract

People who work with an AI agent beside an application constantly say "this": this row, this number, this chart. When the agent lives outside the application, in a chat client connected over a tool protocol, it has no way to know what "this" is, and the person has to describe or paste it. We present Agent Pointed Context (APC), an interaction pattern in which the application exposes three small channels to any connected agent: what the person is looking at (the view), what the person has pointed the agent at (the pin), and what each agent is reading or changing on screen (presence). The application hosts no model and no chat. It shares identifiers and keys only, never pixels, text or amounts; the agent reads the facts behind a pointer through the tools it already has, so every read passes the same authorization as any other call. Because a pointer resolves against the data that produced what the person sees, not against the rendering, the agent answers from the inputs behind a figure (the accounts, statements and transactions that make it up) rather than from a description of it. A pin reaches the agent without repeated model-driven polling, piggybacked on its next tool result, and an unattended agent can wait for one. Pointers resolve not only to records but to parts of computed figures, which the server recomputes with the same functions the page uses, so that for the same data snapshot, parameters and authorization scope the agent's number is the screen's number, and the response says when the scope differs. We describe a reference implementation over the Model Context Protocol (MCP) in a personal finance ledger, where 109 elements of the home page alone, from rows to single steps of computed figures, are pointer targets (a coverage count, not a measured benefit), and discuss the design space, the privacy argument and what is left to evaluate.

Zenodo (CERN European Organization for Nuclear Research)
Usability and User Interface Design
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.