Input paper: S. Wang, H. Zhu, J.-H. Huang, Y. Shen, C. Zeng, S. Rudinac, M. Kackovic, N. Wijnberg, M. Worring, "A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding", ICMR '26, arXiv:2604.19689.
Reference model: M. Worring, J. Zahálka, S. van den Elzen, M. T. Fischer, D. A. Keim, "Multimedia and Visual Analytics in the Agentic Era" (MMA) — Figure 1 (framework), Figure 2 (user interface), Figure 3 (human-AI teaming), Figure 4 (visual analytics agent).
The headline: this mapping is inverted. The two other papers in this project are visual analytics systems with no agentic layer. A-MAR is the opposite — an agentic system with no visual analytics layer. It realises the right-hand side of MMA Figure 1 almost completely (foundation models, AI agents, factual knowledge, prompting, strategy) and the left-hand side almost not at all (expert user, interface, guidance and trust loop). Reading it against MMA is therefore less a matter of finding gaps than of noticing that the two halves of the framework have been built by different papers, in the same research group, and have not yet been joined.
Figures (SVG):
| # | File | Follows |
|---|---|---|
| 1 | wang_fig1_offline_stage.svg |
offline stage, data and knowledge in / out |
| 2 | wang_fig2_online_stage.svg |
MMA Figure 1 — components and loops |
| 3 | wang_fig3_user_interface.svg |
MMA Figure 2 — the four UI pillars |
| 4 | wang_fig4_human_ai_teaming.svg |
MMA Figure 3 — action agents and coordinator |
| 5 | wang_fig5_va_agent_extension.svg |
MMA Figure 4 — a VA agent built on A-MAR's planner |
Figure conventions (shared across all four analyses in this project): every figure is generated from one parameterised template, so the five figures can be read side by side across papers. Zone colours follow the MMA paper itself — green for Foundation model-based AI, red for Human-AI Teaming, yellow for Human Understanding and the User Interface, blue for (un-)structured data. Status is encoded consistently: a white or solid box is realised in the paper, a yellow box partially realised, a dashed grey box absent, an orange border an annotation or a loop, and a purple box something the extension in section 5 proposes rather than something the paper contains.
Understanding a painting is not image description. It requires reasoning over symbolic motifs, stylistic conventions and cultural–historical context that are not visible in the image — and, crucially, it is an ordered, multi-step process in which different steps need different kinds of evidence. Multimodal LLMs do this implicitly, from internalised and sometimes hallucinated knowledge. Retrieval-augmented generation grounds them in external sources, but almost all RAG systems retrieve once, from the user's question, returning a flat, unordered context that treats all evidence as equally relevant to every part of the answer. Prior art-specific work (KALE, ArtRAG) adds structured knowledge but keeps the static, query-driven retrieval.
A-MAR's move is to condition retrieval on an explicit reasoning plan. Given an artwork image I, its metadata M and a question Q, a vision–language planner produces
Φ_plan : (I, M, Q) → P_gen = {(r_t, e_t)}_{t=1..T}
where each r_t is a reasoning sub-goal and e_t is the type of evidence that step requires (visual, metadata, structured knowledge). From the plan a retrieval intent q_ret = ψ(P_gen) is derived — deliberately not a paraphrase of Q but a structured statement of evidence needs, so that ψ "abstracts what evidence is required from how the question is phrased". Retrieval then runs in two stages: coarse top-k similarity in a shared embedding space, then a rerank scoring each candidate jointly against the intent and the image, with an optional structural-importance term for graph sources (s(c) = λ·s_sem(c) + (1−λ)·s_str(c), λ = 0.5). Finally, generation is plan-constrained: follow the order in P_gen, ground each step in evidence matching its e_t, and introduce no new evidence.
The paper is careful about one thing that matters for the MMA mapping: the planner performs meta-reasoning, not answer generation. It emits a machine-readable plan in a restricted output space, so reasoning structure can be inspected independently of any content. The plan is described as a control interface.
Alongside the method, the paper contributes ArtCoT-QA: 227 interpretive questions over 227 unique SemArt paintings (174 high / 53 medium difficulty, 100% multi-hop, 4–5 reasoning steps, mean 4.7), each annotated with a chain of thought where every step carries exactly one grounding tag from a fixed schema — Visual, Metadata, Description, KG-Background, Common-Knowledge. That single-tag rule is what makes step-level diagnosis possible.
Results, under a fixed backbone and a fixed retrieval budget: on Artpedia and SemArt, A-MAR beats static ArtRAG and prior art-specific baselines across BLEU/METEOR/SPICE/ROUGE-L/CLIP with both Claude 4.5 Haiku and Mistral-3 Large backbones (e.g. Artpedia BLEU-1 43.3 vs 39.2, SPICE 12.3 vs 10.4). On ArtCoT-QA the more interesting result appears: a text-only planner already lifts retrieval measures (subgraph relevance 2.37 → 2.78, evidence coverage 2.16 → 2.72) but barely moves reasoning; only the full multimodal planner lifts step completeness (3.23 → 3.58) and faithfulness (3.12 → 3.45) together with answer quality. The paper's own conclusion from this is the sharpest sentence in it: evidence must be retrieved for the appropriate reasoning step, not merely retrieved accurately — "retrieval control alone cannot ensure grounded multi-step reasoning without a plan that specifies how retrieved evidence is to be used."
In. Generic data: the web-scale corpora behind Claude 4.5 Haiku and Sonnet, Mistral-3 Large and GPT-5.2, plus CLIP and the standard NLG metric definitions. Domain-specific data: SemArt (images, structured metadata, descriptions organised by content / form / context perspective) and Artpedia — both public, curated and already expert-written. Factual knowledge: the Art Context Knowledge Graph (ACKG, adopted unchanged from ArtRAG); the art-historical interpretive method that motivates the whole design — Panofsky and Baxandall, analysis proceeding from visual observation to contextualisation to interpretation; and the fixed grounding schema. Implicit knowledge: the frozen models, including Claude 4.5 Sonnet in the double role of annotator and judge.
Processing. O1 index the ACKG into a shared embedding space. O2a filter SemArt so every retained painting has non-empty metadata and informative descriptions (length threshold, reasoning-relevant keywords, both content and context perspectives present) — a filter designed to exclude paintings that could only support factual lookups. O2b annotate with Claude 4.5 Sonnet under a strict prompt contract (multi-step questions not answerable from metadata alone; answers fully supported by the given sources; exactly one grounding tag per step; diverse phrasings), validate as JSON, discard off-spec output, flag inconsistencies with GPT-5.2, then validate manually with a domain-expert researcher. O3 design the plan schema, the evidence-type space and the generation contract. O4 fix the retrieval configuration (k = 10, m = 5, λ = 0.5) identically for every method. O5 design the evaluation: three baselines and an LLM-as-a-Judge protocol with the judge held out from generation.
Out. Factual knowledge — the embedded, searchable ACKG (the one genuinely pre-computed artefact inference depends on), the released ArtCoT-QA benchmark, the plan and grounding schemas, the prompt templates, the hyperparameters, the judge rubric. Generic data given back — the benchmark and the comparison tables. Implicit knowledge — unchanged: nothing is trained anywhere in this paper. Every model is frozen and off-the-shelf, and all domain adaptation lives in the prompt and the plan. That is a real contrast within the project: one paper fine-tunes its extractor, one uses frozen embedders, and this one adapts nothing at all.
Two things are worth flagging honestly. First, the offline stage produces no user-facing anything: no interface, no visual encoding, no session state, and no collection-level structure, because A-MAR reasons over one artwork at a time. Second, ArtCoT-QA's ground truth is model-generated: an MLLM writes the reasoning chains from expert-written SemArt descriptions, another model checks consistency, and one domain-expert researcher validates. In MMA's terms the offline stage uses a generation action to manufacture factual knowledge. That is efficient and clearly documented — and it is the paper's most load-bearing assumption, in a domain where interpretation is legitimately contested. The extension in section 5 happens to fix this as a side effect.
The only closed feedback loop in the paper lives here: benchmark → judge scores → ablation over the planning strategy → revised schema. It is a research loop, running at the timescale of a project, and the human in it is a co-author.
Foundation models: Claude 4.5 Haiku as primary backbone for both planning and generation; Mistral-3 Large (MoE) as a second backbone to show the gains are not specific to one model family, scale or architecture; a vision–language scorer for reranking; an encoder for coarse retrieval; Claude 4.5 Sonnet as a held-out judge. All frozen — but used for reasoning and generation, not as feature extractors. The planner is itself a vision–language model: it sees the painting before it plans.
AI agents: realised — the only paper in this project where MMA's agent box is genuinely occupied. The planner is goal-directed, autonomously decomposing open-ended queries into interpretable sub-goals and evidence requirements; retrieval is multi-hop and plan-guided; a multimodal reranker scores candidates against the intent and the image; the generator is contractually bound to the plan. What is not there: no observation agent, no verifier, no critic, no multi-agent team, and no tool use beyond the fixed retrieval stack.
Expert modules: partial. The retrieval stack (encoder, ACKG traversal, structural importance via degree or centrality, softmax normalisation, reranker) does the work of expert modules, but as hard-wired pipeline stages rather than registered capabilities an agent could choose among or extend at runtime.
External knowledge: factual knowledge is realised — the ACKG, metadata and expert descriptions, explicitly retrievable and held identical across every compared method so the ablation isolates planning. Implicit knowledge is the frozen weights plus the art-historical method baked into the plan schema.
The planner is a genuine action model in MMA's sense: it holds a goal, decomposes it, types each sub-goal's evidence requirement, and controls the components downstream. Prompt templates and strategy are realised — Φ_plan and ψ are exactly the prompting machinery MMA describes, with a typed output contract, and the plan is an externalised, inspectable strategy. The multimodal structure parser is partially realised: results are typed, the answer is emitted step-wise so evidence alignment makes claims traceable, and the plan doubles as a rationale.
What is missing is the entire visual half. No VA mapper — no prompting from an interface, no provision mapping from response to presentation, no recommendation mapping from feedback to guidance. No session memory. No feedback translation. And no visual analytics grammar — which is the sharpest omission, because the plan already carries the type that would drive an idiom choice: e_t = Visual wants the image with regions marked, KG-Background wants the retrieved subgraph, Metadata wants a timeline. The typing exists; the mapping does not.
The expert user is notional. The paper motivates itself by "particular relevance to cultural industries" and positions itself as "a strong foundation for interactive and explainable multimodal reasoning systems" — a foundation for interactivity, not an interactive system. There is no user study, no participant, no human in any loop; in the system as built, the user is a dataset row.
Goals are present, but as a single string supplied once — no refinement, no follow-up. Hypotheses are absent: a user cannot state one, test one, or ask the system to argue against an interpretation. Tasks are fixed: explain one artwork. Of the four actions, Generation is fully realised (and this is the only paper in the project where MMA's fourth action exists at all); Analysis and Search happen but are performed by the agent, not by a human; Query is absent as a user action — q_ret is authored by the planner. The Finding → Insight → Knowledge chain terminates at delivery: nothing accumulates, no annotation, no growing structure, no carry-over to the next artwork.
The user interface is absent — and this is worth stating precisely rather than dismissively, because A-MAR computes the raw material for three of MMA's four interface pillars and then discards it: the plan is an ordered, typed process description (a ready-made progress display and navigation spine); the per-step evidence tags plus the retrieved subgraph are provenance, traceable by construction; the plan is a rationale, available before the answer; and s_sem, s_str and s(c) are per-candidate confidence signals. All four exist in memory at inference time. None reaches a human.
| MMA component | In A-MAR |
|---|---|
| Expert user | absent — no participant, no user study |
| Goals | present, as one question string, once |
| Hypothesis | absent |
| Tasks | fixed: explain one artwork |
| Actions: generation | realised — the only paper here where it is |
| Actions: analysis / search | performed, but by the agent, not the human |
| Actions: query | realised as an agent, absent as a user action |
| Finding → insight → knowledge | terminates at delivery; nothing accumulates |
| User interface | absent — though the material for it is computed |
| Visual analytics agents | half — a real action model, no visual half |
| Visual analytics grammar | absent (though the plan is already typed) |
| Prompt templates / strategy | realised — Φ_plan and ψ |
| Multimodal structure parser | partial — typed results and rationale, no confidence |
| Foundation models | realised — frozen, but for reasoning and generation |
| AI agents | realised — a genuine planning agent |
| Expert modules | partial — hard-wired pipeline stages |
| Factual knowledge | realised — the ACKG, shared across all methods |
| Guidance and trust loop | absent |
| Strategy loop | realised, but open — retrieval fires once |
Figure 3 — the (missing) user interface against MMA Figure 2. Rather than four empty boxes, each cell names the
artefact A-MAR already computes that would fill it. Outputs: the step-wise explanation, which is the richest
output in the project — the cheapest win in the paper is to render it as steps linked to evidence rather than as a
paragraph. Process: the plan is a progress display and a navigation spine waiting to be drawn; in the paper it
appears only in figures, as an illustration for readers. Knowledge: the ACKG is a structured knowledge base and
the retrieved subgraph a genuine provenance record — but the knowledge base is read-only and nothing a user
concludes is written back. Trust: the strongest raw material anywhere in the project — a rationale available
before the answer, claim-level rather than document-level traceability, per-candidate confidence, and a
contractual prohibition on introducing new evidence. The guidance row is empty across all four columns, and the
sharpest case is Process: the plan makes "where next?" trivially answerable, since any step whose evidence
requirement is thinly covered is exactly where a user should look.
Figure 4 — human-AI teaming against MMA Figure 3. The inverse of the other two papers: three and a half of the
four action-specific agents exist. Search is the substantive contribution; Query is realised as ψ, which is
precisely the query-formulation agent both visual analytics papers in this project lack; Generation is
plan-constrained and real; Analysis exists as meta-reasoning rather than data analysis. The coordinator is
partial in an instructive way — it holds a goal, decomposes it and governs the downstream stages, but it does not
schedule, monitor, re-plan, aggregate across rounds, track progress or remember. It plans once and hands off.
Figure 5 — a visual analytics agent built on A-MAR's planner. See section 5.
A-MAR is the one paper in this project that already owns the Action Model. Extending it is therefore not a matter of adding intelligence — it is a matter of adding everything that connects an action model to a person. Most of it is unusually cheap, because the plan is already typed.
The plan is already a user interface. P_gen = {(r_t, e_t)} is an ordered, typed, machine-readable structure that states in advance what will be done, in what order, and what each step needs. Render it: four or five steps, each either grounded or not yet grounded, each clickable to reveal the evidence retrieved for it and the claim it produced. That single change fills MMA's Process pillar and most of Outputs, with no new modelling at all.
Make the plan editable, and the paper's open question becomes a feature. The discussion notes that the design choices governing agent behaviour — "how reasoning is decomposed, ordered, and expressed" — "are not unique", and that comparative, contrastive, analogical and theme- versus form-driven strategies are all plausible. Today that is future work for the authors. In a visual analytics version it is a user choice: a curator asks for a contrastive reading against the artist's earlier work, or reorders the steps to put context before motif, and the plan changes. This is exactly MMA's prompting mapping — specification from the interface into a prompt template.
Provision is nearly free, and it is the highest-value addition. Nothing about presentation is modelled, yet the plan already carries the type that determines it: e_t = Visual → the painting with the relevant region highlighted; KG-Background → the retrieved ACKG subgraph with nodes ranked by s(c); Metadata → a timeline or an oeuvre strip; Description → the passage with the claim linked to it. Declare the idioms once and the mapping from evidence type to view follows mechanically.
Close the strategy loop with per-step retrieval and re-planning. Retrieval currently fires once on the aggregated intent. Make each step retrieve for its own e_t, observe whether the requirement was met, and re-plan when it was not — an observation-based agent in MMA's sense. The paper's own ablation is the argument for this: a text-only planner improves retrieval measures but barely moves reasoning, so better access to evidence is not the binding constraint. The binding constraint is the coupling between what a step needs and what it gets, and only a loop can enforce that.
Surface the confidence you already compute. s_sem, s_str and s(c) exist per candidate, are used to rank, and are then dropped. Expose them per step: which ACKG nodes supported step r_3, how strongly, and what almost made the top-m. That converts the Trust pillar from a claim into a display — and it is the same pattern found in both other papers in this project, where a computed confidence signal is discarded before any human sees it.
Guidance falls out of the plan–evidence mismatch. When a step's required evidence type is thinly covered, that is a legible, actionable signal. Orienting: which steps are solidly grounded. Directing: where to look next. Prescriptive: accept, re-tag, supply a source, or re-plan. The authors' closing request for "evaluation protocols for more controllable and transparent agent-based multimodal AI systems" is, read in MMA's terms, a request for exactly this row.
Feedback translation, using a vocabulary that already exists. The grounding schema — Visual, Metadata, Description, KG-Background, Common-Knowledge — is precisely the alphabet an expert needs to correct a step: "this step is not visual evidence, it is assumed background." Every accept, reject, re-tag and re-order is simultaneously a preference signal for the planner and a new ArtCoT-QA item. This moves the annotation loop from a one-off offline pipeline into the working product — and it repairs the paper's most load-bearing assumption, since the ground truth stops being model-generated-and-validated-once and starts being expert-corrected continuously, in a domain where interpretation is contested and a single validator is thin.
Session memory, and the jump from one artwork to a collection. A-MAR reasons over one artwork, once. Art-historical work is collection-level: tracing a motif across an oeuvre, comparing a school, following an attribution dispute, positioning a work within a period. Store plans, retrieved subgraphs, accepted and rejected steps and the expert's own phrasing across a session, and the second artwork can be planned in the light of the first. This is also where the project's two visual analytics papers become directly relevant — they already know how to make a large collection navigable, and A-MAR already knows how to reason about one item in it.
A verifier, and richer knowledge. There is no self-check, no critic and no revision pass; a verifier agent scoring a claim against its cited evidence before it reaches an expert is standard agentic practice and directly addresses the hallucination risk the paper cites as its motivation. On knowledge: the ACKG is the only source. Iconclass and other iconographic vocabularies, provenance and exhibition records, museum collection APIs, and the art-market and career data that this author group's own economics co-authors work with would all extend the factual-knowledge box — and, unlike the ACKG, some of them carry rights and confidentiality constraints, which argues for on-premise or local models and therefore, as MMA points out, for visual analytics carrying more of the weight rather than less.
Evaluation has to change with the extension. LLM-as-a-Judge over step completeness, faithfulness, subgraph relevance, evidence coverage and answer quality measures system properties, and does so well. A visual analytics version needs measures of the human-AI team: time to insight, plan edits per session, expert acceptance rate per grounding type, and something Analytic-Quality-shaped over the pair rather than the model. ArtCoT-QA's 227 items with one question each is an excellent diagnostic and not a workflow benchmark. One caution to keep in view: the annotator and the judge are both Claude 4.5 Sonnet, and the generators are frontier models from overlapping families. The paper mitigates self-evaluation bias by holding the judge out from generation, which is the right move — but the deeper fix is putting experts in the loop, which the extension does anyway.
What the extended version would feel like. A curator opens a painting and states an interest rather than a question. The agent proposes two or three candidate reading strategies; the curator picks one and reorders a step. The plan appears as a spine: motif → context → symbolism → synthesis. Each step fills in as its evidence arrives, displayed in the idiom its evidence type implies — regions on the canvas, a subgraph, a timeline — with confidence shown and thin steps flagged. The curator rejects one grounding, re-tags another as assumed background rather than documented fact, and supplies a source. The agent re-plans the affected step. At the end, the explanation is defensible claim by claim, the corrections have improved both the knowledge base and the planner, and the session carries forward to the next painting in the collection.