MULTI-CASE (Fischer et al.) analysed through the Multimedia Analytics model

Input paper: M. T. Fischer, Y. Metz, L. Joos, M. Miller, D. A. Keim, "MULTI-CASE: A Transformer-based Ethics-aware Multimodal Investigative Intelligence Framework", arXiv:2401.01955, 2024.

Reference model: M. Worring, J. Zahálka, S. van den Elzen, M. T. Fischer, D. A. Keim, "Multimedia and Visual Analytics in the Agentic Era" (MMA) — Figure 1 (framework), Figure 2 (user interface), Figure 3 (human-AI teaming), Figure 4 (visual analytics agent).

A note worth stating up front: Maximilian Fischer and Daniel Keim are authors of both papers. MULTI-CASE is therefore not an arbitrary test case but part of the lineage the MMA model generalises — which shows in how precisely its strengths line up with MMA's knowledge and trust pillars, and how precisely its gaps line up with MMA's guidance and agent layers.

Figures (SVG):

Figure conventions (shared across all four analyses in this project): every figure is generated from one parameterised template, so the five figures can be read side by side across papers. Zone colours follow the MMA paper itself — green for Foundation model-based AI, red for Human-AI Teaming, yellow for Human Understanding and the User Interface, blue for (un-)structured data. Status is encoded consistently: a white or solid box is realised in the paper, a yellow box partially realised, a dashed grey box absent, an orange border an annotation or a loop, and a purple box something the extension in section 5 proposes rather than something the paper contains.

1. What the paper does

Intelligence analysts — investigative journalists, criminal investigators, law-enforcement analysts — have to combine many heterogeneous, uncertain and sometimes conflicting pieces of intelligence into one incomplete picture. Fischer et al. identify two obstacles. First, ethics and privacy: the data is sensitive, the stakes of error are high, and the users are typically not machine-learning experts, so they may hold unrealistic expectations or misplaced trust. Second, multimodal integration: analysing network structure, entity content and timing in isolation gives a partial view and forces analysts to hunt for cross-matches between modalities by hand.

MULTI-CASE answers both with one architectural idea: a fully-integrated graph data model that acts simultaneously as an abstraction layer between analysis modules and as a shared store of knowledge for humans and machines. Around it:

1. A plugin architecture. Analysis modules (NER, Whisper transcription, speaker detection, image and video analysis, geo, search) register at startup with the Orchestration Layer, declaring the data types they ingest, the graph changes they listen for, the context actions they contribute and their preview handlers. Heavy computation runs server-side; the client is a thin web UI. Modules are change-agnostic as long as the types they use are untouched.
2. A designed data model. Generalised from the police information model IMP with domain experts and aimed at being mutually exclusive and collectively exhaustive: 50+ hierarchical entity types (root Thing → Entity, Event, Datetime, Location, Document, each with subtypes such as Person or PhoneCall) and 10+ relationship types. Relationships also carry source attribution (which document, which module) and a confidence score on the 6x6 intelligence scale analysts already know.
3. A domain-specific NER transformer. GottBERT (RoBERTa, German) fine-tuned on a mixed corpus of intelligence-specific text and a re-tagged GermanNER, labelled with the authors' own MULTI-CASE-compatible labelling tool. Generic PER/ORG/LOC/MISC schemes are too coarse for investigations, so custom labels (EVENT, PRODUCT, LAW, LANGUAGE, DATETIME, QUANTITY, NUMBERS) were added. Model and dataset are open-sourced.
4. A visual interface. A GPU-rendered 2D/3D knowledge graph holding 30k nodes and 100k edges at 60+ fps, a neighbourhood-exploration spotlight showing 3–4 steps of local context, an ontological search with exact / substring / fuzzy / ontological modes and distance-based scoring, an annotated text view with entities colour-coded by type and a per-document entity summary, a brushable timeline, specialised module UIs, in-situ context overlays, and a printable chain-of-evidence report.
5. Four evaluations. A war-crime-investigation case study (the journalist "Alisa"); an assessment against the authors' own ethics design guidelines (C1-6, R1-5, A1-6); an assessment against a 52-criterion intelligence capability classification; a per-label NER benchmark against five baselines; and a formative study with eleven domain experts in law enforcement, research, security industry, law, ethics and policy (60-minute presentation plus 120 minutes hands-on, 38 semi-structured questions).

The design principle throughout is equal joint agency: automation is checked and balanced by visual analytics. The single sharpest expression of it is a policy, not a widget — an automated decision that has not been manually reviewed is never rated above C ("fairly reliable") on the 6x6 scale. Trust is capped by design.

2. The offline stage (figure 1)

The most important structural fact about MULTI-CASE is what its offline stage does not contain: the case. Nothing about a specific investigation is pre-computed. The offline stage builds capability; the online stage does the analysis.

In. Generic data: the German corpora behind GottBERT, BERT-base-german-cased and XLM-RoBERTa; GermEval2014; the GermanNER corpus; Whisper's weakly-supervised speech data. Domain-specific data: intelligence-specific text labelled with their own tool, the UNODC Criminal Intelligence: Manual for Analysts, expert interviews, and existing police case models such as IMP. Factual knowledge: the entity/relation taxonomy to be designed, the ontology database of synonyms and hyponyms with distance scoring, the 6x6 scoring scheme, the UNODC task taxonomy (link / event / flow analysis answering who–what–how–where–why–when), the ethics guidelines, the 52-criterion capability scheme. Implicit knowledge: GottBERT as fine-tuning base, four baseline models, Whisper, and partner models for speaker recognition and vision.

Processing. O1 requirement elicitation → three stated requirements (a centralised multimodal platform for collaborative case work; assistance in labour-intensive big-data tasks; transparency and reliability). O2 data model design with experts. O3 interactive labelling and a 70/15/15 split. O4 fine-tuning — Adam, weight decay, dropout 0.1, slanted triangular learning rate, early stopping on validation F-score with patience 10, one RTX 4000. O5 benchmarking. O6 ontology population, the plugin contract, and the GPU renderer.

Out. As factual knowledge: the fully-integrated data model, the ontology database, the 6x6 confidence policy including the cap on unreviewed automation, and the plugin registration contract. As implicit knowledge: the fine-tuned NER model (open-sourced for transparency, C3, and so others can detect automated inequality, R3) plus the transcription, speaker and vision models. As generic data given back: a re-tagged GermanNER benchmark and a six-model comparison table. Benchmark reality is reported honestly — strong on DATETIME (.99), NUMBERS (.98), QUANTITY (.97), LANGUAGE (.96), PERSON (.91), LOCATION (.89), weak on LAW (.60), EVENT (.57), PRODUCT (.54) and MISC (.18), and below GermEval-level scores for every model including the baselines.

Two observations matter for the mapping. First, this is the one paper in the project where a foundation model is genuinely adapted rather than used frozen — and it is adapted for extraction, not generation. Second, there is a loop the paper designs for and does not close. It lists "supervising and validating the model in deployment and adapting it in the face of changing language patterns, terms, or requirements" as a necessary step, and it built the labelling tool to be MULTI-CASE-compatible. Yet an analyst correcting an entity, confirming a relation or grading it B in the 6x6 scheme produces exactly the supervision the NER model needs — and it is only archived. Retraining stays a manual, out-of-band act. Call it the model-improvement loop: designed for, not closed.

3. The online stage mapped onto MMA Figure 1 (figure 2)

Human Understanding — fully present, and unusually well grounded

The expert user is an investigative journalist or criminal investigator, explicitly not an ML expert. The goals are the UNODC six questions, which is the closest thing to a formal goal model in any of these papers — it is written down, it is domain-standard, and the analysts already think in it. Tasks are UNODC's link, event and flow analysis (the last only partially supported by the shipped visualisations, as the authors note). Hypotheses are concrete: A and A′ are the same person; this image is disinformation.

Of MMA's four actions, three are strongly realised — Analysis (NER, transcription, speaker separation, image and video matching, spotlight, timeline), Search (four match modes with ontology-distance scoring), Query (timeline brushing, type selection, confidence threshold, cross-match-only mode). Generation is the weak one, but not empty: Whisper transcription is generative, and the physical report renders a chain of evidence from a template. What is missing is summarisation, hypothesis generation and narrative — and the authors point there themselves.

The Finding → Insight → Knowledge chain is the strongest in the project. Knowledge is externalised in the shared knowledge graph: confirmed relations, 6x6 gradings, merged aliases, comments, disproved items archived rather than deleted, and a printable chain of evidence. One expert described the graph view as "a supportive mental map and a large digital notebook" for what "currently is often only in one's head". That is MMA's shared knowledge base with provenance, built and used.

Foundation model-based AI — expert modules and factual knowledge, realised

Foundation models: the fine-tuned RoBERTa NER model, Whisper, and partner models for speaker, image and video — running online, on every ingested document. No generative or reasoning model is deployed.

Expert modules: the strongest realisation of this MMA component anywhere in the project. Every capability is a registered plugin with a declared contract; modules can be swapped or added; the ontology database is itself a curated factual-knowledge module, editable by the analyst at runtime.

External knowledge: factual knowledge is richly present — 50+ entity and 10+ relation types, the ontology with hyponym/synonym distances, the IMP-derived case model, 6x6 scoring, source attribution on every relation. Implicit knowledge is the tuned weights. What is missing is retrieval over external corpora: OSINT reports, prior cases, legal texts are never brought in.

AI agents: absent. The nearest thing is the event-driven module chain, which fires on data types and graph events rather than on analytical intent, and cannot decide to do something else.

Human-AI Teaming — the most interesting column in the project

There are no visual analytics agents. But unlike a system that replaces them with hard-coded coordinated views, MULTI-CASE has built much of what an agent would stand on:

• The Orchestration Layer is a coordinator's body without its head. It selects modules by data type, dispatches them, writes their results into the shared model, re-triggers downstream modules through graph change listeners, hides superseded inferences, aggregates everything into one collaboratively edited structure, and logs every change. What it lacks is a goal, decomposition, a progress measure, analytical grounds for choosing between modules, and any rationale for what it ran.
• The visual analytics grammar is partly declared. The plugin contract is genuinely machine-readable — ingest types, change listeners, context actions, preview handlers — and the main graph UI composes itself from those declarations at runtime. What is missing is a reader: nothing reasons over the declarations to pick an idiom for a question, and the encodings and interactions themselves stay hard-coded inside each module.
• Prompt templates and strategies are absent. What crosses into the AI side is a MIME type (dispatch) and a search term (retrieval). The authors' outlook is explicit about the alternative: three experts tried ChatGPT on crafted case material and were "astonished" by the querying workflow and the quality of the leads; the authors call interactive prompting through LLMs "potentially very useful" and "very promising".

The loops

• L1 · Guidance and Trust loop — trust realised, guidance missing. The trust channel is genuinely there: in-situ explanations, transparent source attribution on every relationship, a 6x6 confidence score with default F (unknown), a confidence-threshold slider, the policy cap at C for unreviewed automation, an open-sourced model, and a deliberate refusal to offer "solve investigation" buttons (against exaggerated expectations, C4). Explicit feedback genuinely reaches the models — confirming a relation, deselecting a speaker, editing the ontology all change subsequent results. Implicit feedback is fully captured by the provenance archive and used only for audit. Recommendations are absent, and the authors name the gap: "one aspect to further improve upon is automated guidance to facilitate critical reflection (R4), for example, by automatically trying to detect biased behavior by human operators."
• L2 · Strategy loop — absent. No prompting, no responding, no plan, no explanation of a plan.
• L2′ · Inter-module refinement loop — realised, and the key finding. Changing a parameter in one module re-runs everything downstream: deselect two background speakers → resynthesise the audio → re-transcribe → re-run NER → the graph is enriched again, the old inference archived. The paper says this comes "even closer to the traditional visual analytics loop". Architecturally it is the asynchronous, re-triggerable substrate an agentic strategy loop requires — with no agent on top of it.
• L3 · Collaborative case-working loop — realised. Several analysts work one shared graph, with comments, shared filters, 6x6 gradings and a provenance trail; the terminal artefact is the printed chain of evidence.
• L4 · Progressive ingestion loop — realised. New material arrives mid-investigation (a SecureDrop submission, a stream, an import from the ICC evidence platform), is dispatched by type, and enriches the same graph. The paper classes the analysis as online and incremental and the interaction strategy as both iterative and progressive.
• L5 · Model-improvement loop — designed for, not closed. See section 2.

Component-by-component summary

4. Detailed views (figures 3, 4, 5)

Figure 3 — the user interface against MMA Figure 2. Outputs is strong (GPU knowledge graph, annotated text with per-document entity summaries, in-situ context overlays, scored search results, specialised module views). Process is strong on navigation — the neighbourhood spotlight was the single most praised feature, "a must, especially when many data items are loaded", and integrated navigation lets an analyst jump from a graph node straight to the passage in the document viewer — and the streamed, incremental ingestion makes the process genuinely progressive; what is missing is any display of progress towards a goal. Knowledge is the flagship: structure in the fully-integrated data model, provenance in a write-once-read-many archive plus source attribution on every relationship, and collaborative editing of one case. Trust is unusually strong and, notably, designed for rather than retrofitted. The guidance row is the consistent gap across all four pillars — the cross-matches-only display mode and the ontological search's synonym surfacing are the only guidance-like cues, and the authors themselves ask for more.

Figure 4 — human-AI teaming against MMA Figure 3. Each action-specific agent is present as a mechanism and absent as an agent: modules run because a type matched, not because they would answer the question. Search is the best of the three papers (four modes, editable ontology) and still leaves query formulation entirely to the analyst. Query is expressive but cannot compile a constraint stated in the analyst's own words. Generation is nearly empty. The coordinator section is the substantive one: it lists, side by side, the seven things the Orchestration Layer already does and the six things that make it a pipeline rather than an agent.

Figure 5 — a visual analytics agent for MULTI-CASE, following MMA Figure 4. See section 5.

5. Extending the method with visual analytics agents

MULTI-CASE is unusually well positioned for an agentic extension, for one reason that is easy to miss: it already has the output contract. The hardest practical problem in putting a generative model into an analytical system is that its output is prose while the system needs structure. Here the structure is already specified — 50+ entity types, 10+ relation types, attributes, source attribution, a confidence scale. Constrain generation to that schema and an LLM emits graph deltas, not paragraphs. Everything else follows from that.

An Analysis agent that explains itself. Today a module runs because a MIME type matched, and nothing accounts for its output. Experts asked precisely the missing question — why was that speaker recognised as that speaker? An analysis agent would take a subgraph or a document, prompt a multimodal model with it, and return a typed proposal plus the passage, frame or speaker turn that supports it. In this domain the rationale is not a nicety: a chain of evidence is what the deliverable is. It also directly addresses the weak NER categories — EVENT at .57 and PRODUCT at .54 are exactly the broad, semantically open categories where an LLM with case context should beat a fine-tuned tagger, and the authors' own GPT-4 zero-shot experiment reached "only slightly less quality than manual, human labeling" on the core types.

A Search agent that formulates, not just matches. The four match modes plus an editable ontology are already strong. What is missing is strategy: decomposing "who else was near this village in the week before the invasion" into ontology expansion, entity queries, timeline constraints and cross-document corroboration; running them; and fusing the results. The case study shows the analyst doing this by hand, step by step, and doing it well — which is exactly the labour an agent should absorb.

A Query agent that compiles constraints. The schema can already answer "calls between anyone near this village and anyone linked to A, in the three weeks before the invasion, corroborated by at least two documents". The interface cannot express it. A query agent translating domain language into graph queries over the existing typed model is a low-risk, high-value addition that needs no new models at all.

A Generation agent — the largest missing action. Summarise a document, a speaker's contributions, or a subgraph. Draft the article section or the case-file entry with every claim linked to its source and its 6x6 grade. Generate candidate hypotheses from an under-explored region. Draft the report the analyst currently assembles by hand. The report function already exists as a template; making it a drafting agent is a small step with a large effect on the "extremely high manual workload, which currently requires much personnel" that LEA 1 and LEA 2 both named as their core problem.

A Coordinator agent that wraps, not replaces, the Orchestration Layer. Keep the type dispatch, the change listeners, the graph writes, the archiving of superseded inferences. Add a goal — and the goal is already written down, in the UNODC six questions and the link/event/flow taxonomy. Add decomposition into sub-goals, a progress measure, and a rationale per invocation. This is the cheapest structural win in the paper: the body exists, and the extension adds the head.

Close the two open loops. They are the same loop seen from two ends. Every 6x6 grading is a labelled example; so is confirming a relation, merging two aliases, deselecting a background speaker, adding a codeword to the ontology, or marking an image disproved. The system already re-runs downstream modules on these edits — the mechanism is there — but it never learns from them, and retraining stays out-of-band. This is the richest supervision signal in the project, and it is currently write-only. Human-in-the-loop learning over it would tune the NER model, the ontology and the confidence calibration continuously, per case, which is exactly the "supervising and validating the model in deployment" the paper says is necessary.

Turn the provenance archive into session memory. It is already a complete, revision-safe record of every creation, update and hiding — kept for accountability (C6) and human oversight (R5), and read by nothing. Extending it with prompts, results, contexts and agent feedback buys three things at once: agent memory across a multi-week case, the collaborative history colleagues asked for ("show a modification and usage history from co-workers"), and a full audit trail of what the agent did — which in this domain is the precondition for using an agent at all.

Build the guidance layer out of data already held. The archive knows where the analyst has and has not been. That yields orienting guidance ("this subgraph has not been visited since the new import"), directing guidance ("nine high-impact links are still graded F"; "every claim about A traces to a single document — corroboration is thin"), and the reflective guidance the authors ask for under R4: flagging the analyst's own pattern of attention as possibly biased. None of this needs a new model. It needs something to read the log.

Reach factual knowledge through retrieval. The ontology and data model are curated factual knowledge, but nothing external is retrieved. RAG over prior case files, published OSINT reports, sanctions and equipment databases, and the relevant legal texts would let an extraction agent resolve a vague mention into a specific equipment model or unit designation rather than a generic PRODUCT token — and would let a query agent answer questions the current case data alone cannot.

Extend the grammar so agents can read it. The plugin contract already declares capabilities. Extend it to declare idioms, interactions and tasks, and the provision mapping becomes an inference: the agent chooses graph versus text versus timeline versus module, and says why. The registry of context actions and preview handlers is already the mechanism — today the analyst picks from a context menu; tomorrow the agent proposes and the analyst still decides.

The constraints this domain imposes on agents — from the paper's own findings

This is where MULTI-CASE gives the agentic discussion something the other papers cannot: a vocabulary for the limits.

• F4, limited acceptance of unreasoned decisions, makes the Rationale channel mandatory, not optional. An agent that cannot produce a chain of evidence is unusable here regardless of its accuracy.
• F3, reduce false negatives for VA and false positives for AI, is a concrete objective function. An interactive, human-verified agent should optimise recall — a missed lead can break an investigation. An agent acting autonomously should optimise precision — a wrong attribution harms an innocent person. The paper found experts holding both preferences simultaneously, indexed to the automation level. An agentic system should therefore switch its own operating point according to how much human review a given output will receive.
• F2, unobtrusive support systems are accepted, warns against the obvious failure mode of agentic interfaces: "tormenting approaches hindering the workflow" destroy trust that took a long time to build. Guidance should nudge and answer on request, not interrupt.
• The 6x6 policy cap generalises directly. "No unreviewed automated decision above C" becomes "no agent-produced entity, relation or merge above C without human review" — a single rule that keeps an agentic system inside the same accountability envelope the paper already defends, and that prevents automated inequality (R3) by construction.
• The authors' own LLM cautions apply. Hallucinations "hard to detect" argue for a verifier agent and for schema-constrained output; the limited context window argues for retrieval rather than stuffing the case into a prompt; and legal constraints argue for on-premise or specially contracted deployment — which pushes toward smaller local models and therefore, as MMA argues, toward visual analytics carrying more of the weight rather than less.

What the agentic version would feel like. The analyst states one of the six questions. The coordinator decomposes it, dispatches extraction, search and query agents across the modalities, and reports progress against the sub-goals. Every proposal arrives as a typed graph delta with a rationale, a source, and a 6x6 grade that cannot exceed C until a human confirms it. The analyst confirms, corrects or rejects — and that act is simultaneously a knowledge-graph edit, a training signal, and an audit-log entry. Guidance points at what has not been corroborated and at where the analyst's own attention has been narrow. The chain of evidence assembles itself as a by-product of the work rather than as a final chore.