Multimedia and Visual Analytics in the Agentic Era: Additional Material
Extra information for the paper Marcel Worring and Jan Zahálka and Stef van den Elzen and Maximilian T. Fischer and Daniel A. Keim, Multimedia and Visual Analytics in the Agentic Era, arXiv 2504.06138 (2026).
To show the potential of the model, we have used Claude Opus 5 to automatically map existing papers, taken from the ones mentioned in the multimedia analytics model paper, to the framework to put the method in the context of the model and to consider how the multimedia analytics model helps in identifying directions to extend these existing methods. Below we give a set of example results. Note that these results are directly taken from Claude they are not checked against the original papers.
Description of the procedure
Claude was queried with the following query "Analyze the paper by "first author" et.al based on the instructions provided" where the instructions are defined as:Steps:
- Consider the multimedia analytics model (MMA) by worring et.al. especially by considering the figures in the MMA paper.
- Take the scientific paper given as input and consider all the processing of the data that is done in the offline stage
- Consider the output of the offline stage and then analyze the online stage which you should then map to the different components and loops in the MMA model.
- Having mapped the main elements of the paper into the MMA model identify how the methods in the input paper could be extended based on the components in the model that have not yet been used.
- As output provide the following elements with all figures in SVG format:
- a short textual description of the paper
- a figure showing the offline stage where it is clear what data goes in and what data goes out wherever appropriate decomposed into implicit knowledge, factual knowledge, domain specific data and generic data.
- an overview figure showing the online stage closely following figure 1 of the MMA model including the different loops.
- when appropriate use figures 2, 3, and 4 in the MMA model to create more detailed figures for describing the online stage of the scientific paper.
- Provide a textual description of how the method in the paper could be extended when it would be based on visual analytics agents.
Papers
- Gisolf et al. (2026), Interactive Hypergraph Visual Analytics for Exploring Large and Complex Image Collections
- Fischer et al. (2024), MULTI-CASE: A Transformer-based Ethics-aware Multimodal Investigative Intelligence Framework
- Wang et al. (2026), A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding
- Tran et al. (2026), NII-UIT at VBS2026: Towards Effective Visual Question Answering for Interactive and Multimodal Video Retrieval
- Feng et al. (2024), Interactive Prompt Engineering for Text-to-Image Creation
Overview of all mappings (according to Claude)
Five papers, one framework, five near-disjoint realisations.
| Gisolf et al. (hypergraphs, images) | Fischer et al. (MULTI-CASE, multimodal) | Wang et al. (A-MAR, art) | Tran et al. (NII-UIT, video) | Feng et al. (PromptMagician, generation) | |
|---|---|---|---|---|---|
| What it is | visual analytics, pre-agentic | visual analytics, pre-agentic | agentic pipeline, pre-visual | interactive retrieval under a clock | creation support; the collection is an output |
| Human in the loop | 8 experts, studied | 11 experts, studied | none | live, timed, competing | 12 novices, studied — expertise inverted |
| User interface | five coordinated views | graph, text, timeline, modules | none | timeline, previews, answer panel | four views, semantic zoom, BioFabric |
| Foundation models | frozen embedders, offline | one fine-tuned, online | frozen, reasoning + generation | frozen, embedders and generators | frozen; CLIP in three roles + SD generating |
| AI agents | absent | absent | realised | absent | absent (named as future work) |
| Prompt templates / strategy | absent | absent | realised (Φ_plan, ψ) | prompting without a plan | one real template with a contract; no strategy |
| Actions: generation | absent | thin | realised | realised, in four roles | primary — the framework's exemplar |
| Actions: query formulation | absent | absent | realised as ψ | absent — the human routes | realised, and novice-operable |
| Factual knowledge | EXIF only | 50+ types, ontology, 6x6 | the ACKG | self-generated captions/ASR/OCR | the crowd's prompts — a consumed commons |
| Expert modules | barely | realised — plugins | partial | substantial, hard-wired | present as code, not callable |
| Coordinator | absent | mechanical | plans once, hands off | absent — the human, at speed | absent — a novice, unsupported |
| Session memory | a view history | a provenance archive | none | none | none — the AI side is stateless |
| Trust pillar | essentially empty | realised, designed for | material computed, never shown | partial — linked evidence | partial — verification by example |
| Guidance pillar | thin | thin, named as a gap | absent | realised | realised in Outputs only |
| Strategy loop | absent | absent (substrate exists) | realised, but open | absent | absent |
| Sensemaking loop | realised | realised | absent | realised (explore ⇄ verify) | realised (generate ⇄ evaluate ⇄ refine) |
| Best single extension hook | CES re-targeted at analyst edits | the schema as an output contract | the plan as a user interface | a coordinator under the clock | slots on the prompt, values from the corpus |
The three patterns from the earlier reading hold, and PromptMagician sharpens each.
Every system computes a signal it throws away. Gisolf et al. threshold away membership probabilities and discard analyst edits; Fischer et al. write gradings into an archive nothing reads; Wang et al. rank with per-candidate scores and drop them; Tran et al. generate captions, transcripts and detections that disagree and never compare them. Feng et al. discard the most valuable signal of the five, because their models are frozen: the interaction record is the only channel through which this system could ever learn anything, and it is not even observed. The highest-value extension is once again a reader for a record the system already produces.
No paper has both halves. Gisolf and Fischer have the human, the interface and the sensemaking loop and no agent. Wang has the agent, the plan and the generation action and no human. Tran comes closest and still has no agent to route its own five indices. Feng et al. add a new variant: a rich interface and a genuine guidance channel built on top of two entirely stateless models — everything that would constitute learning or planning is done by the person.
The complementarity is almost exact, and now includes generation. A-MAR's planner is the missing router for NII-UIT's five indices and the missing decomposer for PromptMagician's single-vector retrieval. MULTI-CASE's provenance archive is the missing session memory for A-MAR, NII-UIT and PromptMagician alike. Gisolf's collection-scale overlapping structure is the missing Knowledge pillar for the retrieval and creation systems both. NII-UIT's hotspot-and-candidate guidance and PromptMagician's keyword-and-criterion guidance are two independent solutions to the same problem — how to make a machine suggestion checkable before it is accepted — arrived at in different media. And PromptMagician contributes the piece none of the other four has: generation as an analytic action, with an interface designed for judging its output rather than merely receiving it.
The framework's contribution, read across five papers rather than one, is to make visible that these are not five research agendas but five parts of one — and that in every case the missing part has already been built somewhere else in the same list.