Gisolf, Geradts & Worring (2026) analysed through the Multimedia Analytics model

Input paper: F. Gisolf, Z. J. M. H. Geradts, M. Worring, "Interactive hypergraph visual analytics for exploring large and complex image collections", Information Visualization, 2026.

Reference model: M. Worring, J. Zahálka, S. van den Elzen, M. T. Fischer, D. A. Keim, "Multimedia and Visual Analytics in the Agentic Era" (MMA), 2026 — Figure 1 (framework), Figure 2 (user interface), Figure 3 (human-AI teaming), Figure 4 (visual analytics agent).

Figures (SVG):

Figure conventions (shared across all four analyses in this project): every figure is generated from one parameterised template, so the five figures can be read side by side across papers. Zone colours follow the MMA paper itself — green for Foundation model-based AI, red for Human-AI Teaming, yellow for Human Understanding and the User Interface, blue for (un-)structured data. Status is encoded consistently: a white or solid box is realised in the paper, a yellow box partially realised, a dashed grey box absent, an orange border an annotation or a loop, and a purple box something the extension in section 5 proposes rather than something the paper contains.

1. What the paper does

Domain experts — forensic and accident investigators, investigative journalists, social-media analysts — regularly have to draw high-stakes conclusions from image collections of thousands to tens of thousands of images that arrive without any annotation. Gisolf et al. call these complex image collections (CICs) and identify two properties that break the usual toolchain. First, the relations among images are overlapping and non-exclusive: one image may belong at once to a content grouping, a location, a time and an event, so single-assignment clustering misrepresents the collection and any annotation would have to be multi-label. Second, the content is specialised and unique, so pre-trained models trained on ImageNet or MS COCO face a semantic gap — and the usual remedy, fine-tuning, is unavailable precisely because there are no labels.

Their answer is a hypergraph: images are vertices, and each hyperedge is a grouping of images sharing content or context, so an image can sit in many hyperedges at once. Around that representation the paper makes four contributions.

1. A construction pipeline from raw pixels. Frozen pre-trained embeddings (Swin v2 as primary, plus Places365 and OpenCLIP) feed an unsupervised constructor. Fuzzy and possibilistic c-means and Gao et al.'s multi-granularity k-means are tried and found wanting; the winner is an adaptation of TEMI — a self-distilled, PMI-based clustering method — in which the usual argmax assignment is replaced by a threshold t on the soft memberships, so an image can join several clusters and the clusters become hyperedges. Two parameters remain: k and t.
2. CoverEdge Similarity (CES). Because construction is unsupervised, it can only be validated offline against annotated benchmarks. Existing hypergraph similarity measures do not scale past ~100 nodes, and clustering measures such as NMI/ARI assume exclusive membership. CES scores how well each ground-truth hyperedge is covered by a set of generated hyperedges, applies a 1/k diminishing-returns factor so over-segmentation is punished, and multiplies by the used-hyperedge ratio R so a solution cannot cheat by generating every possible subset. It is validated on synthetic hypergraphs alongside hNMI (an overlapping-community adaptation of NMI), and shown to be micro-level where hNMI is macro-level.
3. A scalable visual analytics system. Five coordinated views: a hyperedge list with size, status, origin and coherence indicators; a hyperedge grid with Ward-based consolidation and a six-image representative summary per hyperedge; a dynamic list of intersecting hyperedges; a spatial hypergraph built from a UMAP of hyperedge centroids with a second, per-hyperedge UMAP revealed on zoom and links between copies of the same image; and a hypergraph matrix of pairwise overlaps coloured by the harmonic mean of the two directional overlap ratios. The design inverts the usual node-link encoding — hyperedges are the nodes, vertex detail appears only on demand — which is what lets it hold 10,000+ images and 200–400 hyperedges on consumer-grade hardware. The analyst names, merges, splits, creates and removes hyperedges and queries the collection by text, image, region of interest, clipboard or hyperedge.
4. Evaluation with real investigators. A think-aloud study with eight Dutch Safety Board experts on three real accident collections (MH17, 12,621 images; marine, 42,960; parking garage, 4,417). Most tasks were completed in minutes. The overview and text search were the dominant entry points; the spatial view was valued for "where next?" navigation; the matrix was rarely used spontaneously and, in the authors' own case work, only became useful later in an investigation.

Read in MMA terms, the paper is a strong, carefully engineered realisation of the Human Understanding side of the framework — and, deliberately, a pre-foundation-model one. That is what makes it a clean test case.

2. The offline stage (figure 1)

Everything that could be called intelligence in this system happens offline, and it happens once.

In. Generic data: ImageNet-1k, Places365 images and web image–caption pairs, as pre-training corpora, plus the public multi-label benchmarks (CUB-200, MLRSNet, DSEG660) and synthetic Erdős–Rényi / Barabási–Albert / Watts–Strogatz hypergraphs that serve as surrogate ground truth. Domain-specific data: the three confidential DSB collections and the EXIF metadata inside the files — raw, unannotated, with unique categories. Factual knowledge: the hypergraph formalism, the multi-label ground truth of the benchmarks, hNMI, the EXIF field schema. Implicit knowledge: the frozen weights of Swin v2, OpenCLIP and Places365 — generic visual semantics only, which is exactly the semantic gap the paper is about.

Processing. P1 frozen-model feature extraction → P2 unsupervised construction (the TEMI adaptation with k and t) → P3 design and validation of CES → P4 a k × t sweep with CES/hNMI plus a qualitative judgement of whether the overlaps are analytically useful → P5 pre-computation of everything the interface needs: the two-level UMAP layout with iterative overlap removal, hyperedge centroids, cosine-similarity indices, six-image summaries, Ward hierarchies, the overlap matrix, thumbnail tiers, the EXIF index.

Out. As derived domain-specific data, the hypergraph itself plus all the pre-computed structures — this is what the online stage runs on. As factual knowledge, CES and the benchmarked ranking of constructors and embedders (TEMI > FCM > PCM/Gao; Swin v2 > OpenCLIP > Places365), together with the honest finding that the optimal (k, t) is collection-dependent. As implicit knowledge, the collection-specific TEMI heads — and the per-image soft membership probabilities, which are collapsed by the threshold and then discarded. That discarded probability is the only confidence signal the system ever computes, and it never reaches the analyst.

Two consequences matter for the mapping. Every foundation model is used offline, frozen, and purely as an embedder: the only live inference in a session is embedding a text, region-of-interest or clipboard query, which is then cosine-ranked against the pre-computed store. And no factual domain knowledge is ever assembled — no ontology, no case file, no prior report, no detections, captions or OCR (the authors name these as future work). The online stage therefore begins from a static data model.

3. The online stage mapped onto MMA Figure 1 (figure 2)

Human Understanding — fully present

The expert user is a domain investigator, explicitly not a computer-vision or visual-analytics specialist. Goals are stated in domain language ("find all hull damage", "find images that are not from MH17"). Hypotheses are concrete ("there are images showing the fuselage of the cockpit specifically"). Tasks decompose into the paper's own explore/search alternation. Of MMA's four actions, three are present — Search (five query types), Query (filter, sort, intersect, EXIF-field hyperedges) and Analysis (in a thin, statistical form: coherence standard deviation, overlap matrix, top-k intersections) — and Generation is entirely absent.

The Finding → Insight → Knowledge chain is realised in an unusually direct way: knowledge is externalised as the hypergraph itself. Naming a hyperedge "Cockpit", splitting a mixed one, creating "cockpit fuselage" from an intersection — these are the knowledge artefacts. This matches MMA's own claim that categorisation is a rudimentary form of knowledge on which (hyper)graph structure can then be built.

Foundation model-based AI — present only as frozen embedders

Foundation models: Swin v2, OpenCLIP, Places365, all frozen, no generation and no reasoning. Expert modules: partially — the EXIF metadata module genuinely is one (curated, factual, auto-naming its hyperedges, tagged by origin), as are the similarity/UMAP/Ward computations; but there is no knowledge graph or curated domain module. AI agents: absent — no planning, decomposition, tool use, MCP, RAG, observation or verifier agents. External knowledge: implicit knowledge is the frozen weights; factual knowledge is EXIF and nothing else. The mechanisms MMA prescribes for reaching factual knowledge and unseen domain data are not used.

Human-AI Teaming — the empty column

There are no visual analytics agents, no coordinator agent, no action model, no online goal or progress measure, no session memory, no reasoning or optimisation. In their place sits a hard-coded coordinated-view controller plus direct manipulation. The visual analytics grammar exists as behaviour — four idioms, five query types, seven edit operations, three data transformations — but never as an abstraction, so nothing can select an idiom, compose a workflow or explain a presentation decision. Prompt templates and strategies do not exist: the system's only prompt is a free-text string turned into one OpenCLIP vector and one cosine ranking, single-shot and stateless.

The loops

• L1 · Guidance and Trust loop — half-realised. The upward channels are strong: actions, data and parameters (queries, selections, edits) and explicit feedback (name/merge/split/add/remove) both flow, and implicit feedback is genuinely captured (visited, modified, time-since-visit, hide-bordered). The downward channels split in two. Structured result and context of result are excellent — the six-image summary, the ranked intersecting-hyperedge list, the matrix cell that opens an intersection. Recommendations and rationale / confidence are absent. Crucially, the feedback that is collected only edits the model or re-colours the interface; nothing learns from it. The system throws away its richest supervision signal.
• L2 · Strategy loop — not realised. There is no prompting/responding cycle, no prompt or AI strategy, no chain-, tree- or graph-of-thought, nothing to explain.
• L3 · Exploration ⇄ search alternation — realised. The paper's own central loop, and the study confirms analysts move fluidly between the two modes and between views.
• L4 · Model re-construction loop — realised, but out-of-band. The analyst reads a diagnostic cue off the overlap matrix (no meaningful intersections, or near-total overlap), concludes that (k, t) is wrong for this collection, and re-runs construction. This is a real feedback loop from human insight into the model — but it runs offline, takes minutes to hours, and effectively restarts the session. In MMA terms it is a strategy loop with a human standing in for the agent.

Component-by-component summary

4. Detailed views (figures 3, 4, 5)

Figure 3 — the user interface against MMA Figure 2. Of the four pillars, Outputs is the system's strongest (grid, consolidation slider, six-image summaries designed to expose internal variation rather than hide it, intersecting-hyperedge list). Process is strong on navigation (spatial view, minimap, lasso, lazy loading, back/forward history) and empty on progress — nothing shows movement towards a goal, and nothing is progressive because nothing is asynchronous. Knowledge is strong on structure (list + matrix) and rudimentary but real on provenance: the origin column (embedding model / metadata / user) and status column (new / modified / original), mirrored in node colour, are a genuine first step towards MMA's provenance requirement. Trust is essentially empty: a coherence standard deviation and a harmonic-mean overlap score, no explanation, no confidence, no uncertainty, no performance depiction. Across all four pillars the guidance layer is thin — the one honest exception being visit-recency colouring, which is a real "where next?" cue and, tellingly, was added because users asked for it.

Figure 4 — human-AI teaming against MMA Figure 3. Each of the four action-specific agents is replaced by a fixed mechanism: analysis by statistics, search by five one-shot cosine rankings, query by filtering and sorting, generation by nothing. The coordinator is the human being; what is automated is only cross-view coordination. The cost is visible in the study: the same task took under 10 seconds or up to 18 minutes depending purely on how the analyst phrased a text query; participants asked for a way back to previous views and for recency colouring — both session-memory functions they were otherwise performing in their heads.

Figure 5 — a visual analytics agent for this system, following MMA Figure 4. See section 5.

5. Extending the method with visual analytics agents

The striking thing about this system, read against the MMA model, is how much of the required machinery it already has — just pointed in the wrong direction. An agentic extension is therefore less a rewrite than a re-targeting, and it can be organised exactly along the four actions plus a coordinator.

An Analysis agent turns statistics into characterisation. At start-up the analyst faces 200–400 unlabelled hyperedges; the paper reports that some were "more opaque, limiting their analytic value" and that participants coped by hunting for a seed image instead. An analysis agent takes the six-image summary that already exists, prompts a multimodal model with it plus the neighbouring hyperedges as context, and returns a typed proposal — a candidate name, a judgement of whether the hyperedge is internally mixed, a proposed split with its member sets, and a rationale naming the images that drove the conclusion. The prompt is a template, not a free string, so the output has a contract and can be parsed rather than read. Because the collection is confidential and must run on consumer hardware, this argues for small local models — which is precisely the case MMA makes for keeping visual analytics central rather than delegating to the largest available model.

A Search agent turns a phrase into a strategy. Today a text query is one string, one OpenCLIP vector, one ranking. A search agent would decompose "the bridge of a fishing vessel" into a multi-step plan: expand the query with domain vocabulary, issue several complementary queries (text, plus region-of-interest queries seeded from the best hits, plus hyperedge queries over near neighbours), fuse the rankings, and verify candidates before presenting them. It would remember what has already been tried in this session. The 10-seconds-versus-18-minutes spread in the study is the exact symptom this addresses: it is prompt-engineering variance, borne by the investigator.

A Query agent gives the hypergraph the algebra it deserves. The representation was chosen because membership is multi-label, yet the interface only exposes pairwise intersections through the matrix. A query agent would translate domain-language constraints into set expressions over visual and metadata hyperedges — "third floor AND beam joint NOT vehicle", "same location as this hyperedge but a different date" — combining EXIF-derived and visually-derived hyperedges automatically. This is where participants struggled most (floor-specific garage views, the bridge of a specific vessel type), and it needs no new models at all, only structure.

A Generation agent supplies the action the paper does not have. Concretely: names and descriptions for the unlabelled hyperedges; a written account of the investigation with the evidence hyperedges cited, which is the actual deliverable of a Safety Board investigation; synthetic query images for concepts too rare to find by example; and, via a code-generation agent, case-specific views composed from the declared grammar rather than shipped in advance.

A Coordinator agent restores asynchrony and memory. An overarching goal — "establish the cause of the hull breach" — decomposes into sub-goals per action. Because agents run asynchronously, work that today must be offline can move into the session: captioning 300 hyperedges, or re-clustering. The coordinator aggregates results, tracks progress against sub-goals, and keeps the human as the final authority, which is how MMA positions it. This also closes the gap the study exposed: session memory of prompts, results, contexts and feedback belongs in the system, not in the analyst's head.

Closing the strategy loop is the highest-value single change. The paper's most candid limitation is that the optimal (k, t) is collection-dependent, that a naive grid search "may result in unwanted hypergraphs", and that distinguishing a bad constructor from an intrinsically low-overlap collection "requires analyst assessment". That is a reinforcement-learning problem with a reward model waiting to be used. CES is already a coverage and over-segmentation measure — it is simply pointed at benchmark ground truth instead of at the analyst. Re-target it: every merge is evidence of over-segmentation, every split evidence of an impure hyperedge, every added image evidence of a missed member, every rename evidence about semantics. Read as evaluative, instructive and descriptive feedback, these signals let an agent tune k and t online, per collection, and eventually adapt the constructor itself — human-in-the-loop reinforcement learning in the sense MMA describes. The system already logs all of it in the status and origin fields. It simply discards it.

Fill the Trust pillar with signals already computed. The TEMI membership probability is calculated for every image and every hyperedge and then thresholded away. Expose it: membership confidence per image, uncertainty on proposed hyperedges, an ordering that lets an analyst check the borderline cases first. Add attention or counterfactual views for "why is this image here", and a per-suggestion rationale from the analysis agent. This matters more as collections grow: inspecting every image is workable at 4,417 and not at 42,960.

Turn the fixed mapping into a declared grammar. The four idioms, five query types, seven edit operations and three transformations already exist as code. Declaring them makes three things possible at once: an agent can choose a view (matrix when the question is about overlap, spatial when it is about "where next?") and justify the choice; the interface can be recomposed per case; and the mapping from response to presentation becomes an inference rather than a design-time constant. The provision and recommendation mappings of MMA's VA agent are precisely this.

Extend provenance into a shared knowledge base. The origin and status columns are a real head start. Extend them to record, per hyperedge and per membership, whether a human or an agent proposed it, on what evidence, with which model, and when. This is what the paper's own future work — multi-week investigations and several analysts building on a shared hypergraph — actually requires, and it is what MMA means by a knowledge base shared between the expert user and the AI with explicit provenance.

Reach factual knowledge through expert modules and RAG. The semantic gap the paper diagnoses is a gap in factual knowledge about the domain, and RAG plus curated expert modules is MMA's answer to exactly that: an aircraft-part or ship-type taxonomy, the investigation protocol, prior published reports, the case file. With those in reach, a naming agent can propose "port-side hull plating, frame 42" rather than a generic ImageNet word — closing the gap without any annotation, which was the constraint that motivated the whole design. The richer machine-generated cues the authors list as future work (object detection, OCR, semantic segmentation) then arrive naturally as MCP tool calls issued on demand, rather than as another offline pre-processing pass — so a hyperedge like "images containing a readable serial number" becomes something the analyst can ask for mid-session.

What the agentic version would feel like. Instead of receiving a static hypergraph and reverse-engineering what it means, the analyst states a goal in domain language. The coordinator decomposes it; an analysis agent has already characterised the hyperedges and flags the mixed ones with proposed splits; a search agent runs several query strategies in parallel and reports which worked; a query agent answers multi-label constraints directly; a generation agent drafts the findings section with its evidence linked. Every suggestion carries a rationale and a confidence, and every accept or reject tunes the agents and, through them, the hypergraph itself. The hypergraph stops being a fixed artefact delivered by an offline pipeline and becomes what MMA's discussion of this very paper calls for: a dynamic structure supporting on-the-fly information extraction and incremental understanding, with the balance between human and AI negotiated continuously rather than fixed at design time.

Two cautions worth keeping. First, the paper's evaluation is insight-oriented and qualitative for good reason; the agentic version needs evaluation methods that do not yet exist — insight-based measures combined with something like Analytic Quality, extended to human-AI teaming. Second, adding agents adds surface for hallucination and bias in a domain where conclusions carry legal weight. The paper's instinct to keep the analyst looking at actual images is the right foundation to build trust machinery on, not something to design away.