Input paper: B. Tran, T. Do, T. D. Ngo, D.-D. Le, S. Satoh, "NII-UIT at VBS2026: Towards Effective Visual Question Answering for Interactive and Multimodal Video Retrieval", MMM 2026, LNCS 16415, pp. 238–244.
Reference model: M. Worring, J. Zahálka, S. van den Elzen, M. T. Fischer, D. A. Keim, "Multimedia and Visual Analytics in the Agentic Era" (MMA) — Figure 1 (framework), Figure 2 (user interface), Figure 3 (human-AI teaming), Figure 4 (visual analytics agent).
The headline: this is the fullest online loop of the four papers in this project, and the only one whose Guidance row is populated. Everything MMA asks for on the human side is present and under extreme pressure — a real expert user, a real interface, a standardised task taxonomy, and a live clock. Foundation models are used on both sides of the divide in the same session: as embedders for retrieval and as generators for hotspots and candidate answers. What is missing is the agent that would choose among the system's own five indices, and any memory that outlives a single task.
One caveat up front, because it governs how much weight the mapping can bear: this is a seven-page VBS system description, not an evaluated study. The authors write that the contributions are "expected to improve efficiency and accuracy". No measurement is reported. What follows therefore maps a design, and the design's effect on the analytics loop is a claim rather than a finding.
Figures (SVG):
| # | File | Follows |
|---|---|---|
| 1 | tran_fig1_offline_stage.svg |
offline stage, data and knowledge in / out |
| 2 | tran_fig2_online_stage.svg |
MMA Figure 1 — components and loops |
| 3 | tran_fig3_user_interface.svg |
MMA Figure 2 — the four UI pillars |
| 4 | tran_fig4_human_ai_teaming.svg |
MMA Figure 3 — action agents and coordinator |
| 5 | tran_fig5_va_agent_extension.svg |
MMA Figure 4 — a VA agent for a timed session |
Figure conventions (shared across all four analyses in this project): every figure is generated from one parameterised template, so the five figures can be read side by side across papers. Zone colours follow the MMA paper itself — green for Foundation model-based AI, red for Human-AI Teaming, yellow for Human Understanding and the User Interface, blue for (un-)structured data. Status is encoded consistently: a white or solid box is realised in the paper, a yellow box partially realised, a dashed grey box absent, an orange border an annotation or a loop, and a purple box something the extension in section 5 proposes rather than something the paper contains.
The Video Browser Showdown is the field's live benchmark for interactive video retrieval: systems are operated by a human under time pressure, in front of an audience, against heterogeneous collections — V3C (everyday web video), Marine Video Kit (scientific footage) and LapGynLHE/GynSurg (laparoscopic surgery). NII-UIT won the overall VBS 2025 competition; this paper describes what the team changed for 2026.
The change is driven by one task type. Alongside Known-Item Search and Ad-hoc Video Search, VBS has introduced Question Answering, where the evidence supporting an answer may appear only fleetingly inside minutes of video — and for 2026 QA appears more often and is combined with harder KIS variants: KIST, where several textual descriptions must be comprehended, remembered and linked across time, and KISC, where the player may query the organiser for hints that come back partial and indirect, requiring inference.
Against that, the authors name three concrete failures of current systems: pipelines fail to localise short-lived evidence, forcing manual browsing; even once a segment is found, verification is expensive and error-prone under time pressure; and retrieval is optimised for video-level search, leaving a gap to the frame-level localisation QA actually needs. Their answer has four parts.
The stated design goal is worth quoting because it is MMA's own vocabulary arriving from a different direction: "shifting the user's role from open-ended searching to guided verification."
In. Generic data: the LAION-scale pairs behind CLIP and SigLIP, BEIT-3's joint pre-training, NVILA's training data, latent-diffusion data for synthesising query images, and ASR/captioning/detector corpora. Domain-specific data: the three VBS collections — raw video and audio, no labels. Factual knowledge: the VBS task taxonomy itself (KIS, KIST, KISC, AVS, QA) as an externally defined task model, the competition protocol, the detector's label vocabulary, and the profiling results from the 2025 system. Implicit knowledge: SigLIP, BEIT-3, NVILA, a captioner, ASR, a detector, OCR and a diffusion model — all frozen.
Processing. Keyframe segmentation → cross-modal embedding into a vector database → and then the step that decides everything downstream: machine-generated description of the collection. Dense captions over short temporal units, ASR transcripts, OCR strings, detected objects. Then index construction — including, new for 2026, per-video keyframe sub-indices so search can be scoped once a video is identified. Then pipeline acceleration, explicitly motivated by the clock. Then interface design around the new artefacts.
Out. The most interesting output in MMA terms is factual knowledge that the system generated about itself. The captions, transcripts, OCR strings and object labels are factual assertions about the collection produced by generative and extractive models rather than by people — and they are what the online stage actually reasons over. At query time the LVLM aligns the question against the caption timeline, not against the pixels. The semantic gap is bridged offline, by generation. The corollary is inherited error: the paper acknowledges "noisy captions or transcripts" as a live risk, and the n-candidate design is a direct response to it.
Also out: keyframes, embeddings, the object/OCR/transcript indices, the per-video sub-indices, and a query cache. Implicit knowledge is unchanged — nothing is fine-tuned anywhere, exactly as in the art-retrieval paper in this project.
Not produced: no knowledge structure over the collection, no session store, no confidence calibration, and — worth dwelling on — no domain knowledge for the specialist collections. Two of the three datasets are marine biology and laparoscopic surgery, and nothing supplies species names, anatomy or surgical phases. A generic captioner is weakest precisely where the tasks are hardest, and the whole online stage reads off that captioner's output.
The one closed loop here is the annual engineering loop: compete → observe failure under live conditions → profile, swap CLIP for SigLIP, add answer-span prediction, redesign the UI → compete again. Real human experience genuinely reaches both the model and the interface — but at the timescale of a year, mediated by the team. Nothing a player does during a run changes anything.
The expert user is a competition player working against a stopwatch, and the system carries both a novice mode (simplified interaction, for accessibility) and an expert mode where a practised operator can adjust model weights, reformulate queries and combine multimodal evidence. This is the only paper in the project whose user is real, present, and timed.
Goals are externally supplied by the competition rather than formed by the user — sharp, verifiable, and bounded. Hypotheses are rapid and, in KISC, explicitly required: inference from partial, indirect hints is part of the task. Tasks form a genuine, externally standardised taxonomy, the strongest in the project.
All four of MMA's actions are realised. Analysis: hotspot inspection, timeline scanning, cross-checking caption against transcript against frame. Search: text, image, query expansion by paraphrase, synthetic query images from diffusion, object filters, temporal search across consecutive query contexts, and in-video search. Query: object, OCR and speech constraints, temporal context, plus expert-mode weight tuning. And Generation — realised in four distinct roles: dense captions, hotspot reasoning, candidate answers, and query paraphrase/synthesis. Notably, generation here serves the human rather than replacing them: the output is a small verifiable set, not a confident single answer.
Finding → insight → knowledge is where it stops. Finding and insight are fast; knowledge is a submission — one video, one frame, one answer — and then the clock resets. Nothing is annotated, grouped, or carried to the next task. KIST is the paper's own evidence for the gap: it requires linking several descriptions across time, and the authors note that retrieval systems "typically process only isolated queries."
The user interface is realised and purpose-built for verification speed. Preview keyframes rendered directly on the timeline let a hotspot be judged without playback, which skips the single most time-expensive operation in video search.
SigLIP and BEIT-3 as embedders; NVILA as an online reasoner over the caption timeline; a captioner, an LLM paraphraser and a diffusion model as generators. All frozen. This is the only paper here that uses foundation models as both embedders and generators in the same live session.
Expert modules are substantially present as specialised retrieval services — keyframe vector search, object filtering, OCR search, spoken-content search, in-video search, query expansion, visual query generation, caching — though hard-wired in the pipeline rather than registered as capabilities. External knowledge is realised but self-generated: the caption, transcript, OCR and object indices are the system's only knowledge of the collection, and it produced them itself. AI agents are absent.
There are no agents: nothing holds a goal, decomposes the question, chooses which index to consult, or revises after seeing results. NVILA is called once per question; candidates are aggregated once from the hotspots. What stands in is a fixed three-stage pipeline — span prediction → answer suggestion → user verification — plus a human coordinating it at speed.
The visual analytics grammar is implicit and hard-wired, but with a genuine wrinkle worth recording: a real provision mapping exists here. Hotspots go to the timeline; candidate answers go to the answer panel with linked evidence; caption and transcript go to synchronised side panels; ranked frames take positions along the timeline. That is precisely the response-to-idiom mapping MMA asks a VA agent to infer. Here it is correct — and it is a constant, decided once for every task and every user alike.
Prompt templates and strategy: prompting without a plan. Four prompts genuinely cross into the models — NVILA with the question and the caption timeline, the answer module with the question and collected cues, an LLM to paraphrase, a diffusion model to synthesise. None has an output contract, none is aware of the others, and none is reissued after a failure.
The multimodal structure parser is partially realised and thoughtfully so. Result: n discrete candidates rather than one free-form answer, explicitly to protect the user from a confidently wrong generation. Rationale: each candidate is shown with its linked evidence, so the user verifies against frame, transcript and caption rather than trusting text — a working explanation channel. Context of results: partial — frames are ranked by similarity, but no confidence attaches to a hotspot boundary or a candidate, and nothing flags when caption and transcript disagree.
| MMA component | In NII-UIT VBS 2026 |
|---|---|
| Expert user | realised — real, present, and timed; novice and expert modes |
| Goals | realised, externally supplied by the competition |
| Hypothesis | realised — and required by KISC |
| Tasks | realised — KIS, KIST, KISC, AVS, QA |
| Actions: analysis / search / query | realised, and broad |
| Actions: generation | realised in four roles, serving the human |
| Finding → insight → knowledge | stops at submission; nothing accumulates |
| User interface | realised — timeline, previews, panels, answer panel |
| Visual analytics agents | absent |
| Visual analytics grammar | implicit, but with a real, constant provision mapping |
| Prompt templates / strategy | prompting without a plan |
| Multimodal structure parser | partial — n candidates + linked evidence, no confidence |
| Foundation models | realised — as embedders and generators, live |
| AI agents | absent |
| Expert modules | substantially present, hard-wired |
| Factual knowledge | realised, but entirely self-generated offline |
| Guidance and trust loop | guidance realised — the only paper here; trust partial |
| Strategy loop | absent |
Figure 3 — the user interface against MMA Figure 2. Outputs is strong and tuned for one thing, verification
speed. Process is excellent on navigation and empty on progress — nothing shows time remaining, timeline
coverage, or unverified candidates, which is a sharp omission in a task defined by a time limit. Knowledge is the
weakest pillar and structurally so: no structure over the collection, and provenance limited to which modality a
cue came from, with nothing recording that a caption is a machine guess rather than an observation. Trust is
partial but real: linked evidence is a working explanation, and the n-candidate design is itself a trust mechanism
— the system refuses to commit precisely because its inputs are noisy. The guidance row is populated in two of
four columns, which no other paper in this project manages at all.
Figure 4 — human-AI teaming against MMA Figure 3. All four action-specific agents exist as mechanisms, and
generation is the strongest of the four papers. The coordinator is absent, and this is the costliest absence in the
project because of what the task is: the system owns five distinct indices — embedding, object, OCR, speech,
caption — and any given question is answerable through some and not others. "What number is on the runner's bib?"
is an OCR question; "what city does the narrator name?" is a speech question; "is there a dog in the shot?" is
an object question. Choosing among them is exactly a coordinator's job, and it is left to a person with seconds to
decide.
Figure 5 — a visual analytics agent for a timed session. See section 5.
This system is the closest of the four to what MMA describes, which changes the character of the extension. There is nothing to add on the human side — the user, the interface and the guidance channel are all there and working. What is missing is an agent that spends the system's existing capabilities well, under a constraint no other paper in this project faces: five minutes.
A coordinator under a clock — the single highest-value addition. Today everything is synchronous and serial: the player issues one query, waits, judges, issues another. The scarcest resource in the whole system is the player's attention over five minutes, and it is spent choosing which single thing to try next. An agent should run several strategies concurrently — text search, OCR search and speech search issued in parallel, results merged onto the one timeline — while the human verifies the first to return. The paper's own acceleration work (profiling, caching, removing duplicate feature extraction) buys latency; asynchrony would buy far more of it, and is a scheduling change rather than a modelling one.
Route the question to the right index instead of guessing. The five indices exist; the choice among them is manual. Decomposing a question into typed evidence requirements and dispatching each to the index that can serve it is exactly the formalism the art-retrieval paper in this project already writes down — a plan of ordered sub-goals, each tagged with the evidence type it needs. Dropping that planner in front of NII-UIT's five indices is close to a direct transplant, and it converts the system's breadth from a burden on the operator into an asset the agent can spend.
Close the loop on span prediction. Answer-span prediction runs once per question. Nothing checks whether a hotspot actually held the answer, and nothing narrows a hotspot that turned out too wide. An observation step — retrieve for the hotspot, judge whether the required evidence type is actually present, re-predict if not — is the same open-loop-to-closed-loop change that the art-retrieval paper also needs, and here it has an immediate operational payoff: a wrong hotspot costs the player thirty seconds of verification.
Session memory, for the two tasks that demand it. The paper names both. KIST asks the player to "comprehend, remember, and link together multiple textual descriptions across time", and the authors observe that retrieval systems "typically process only isolated queries" — a memory that accumulates descriptions into a growing constraint set, and conditions retrieval on the conjunction, is the direct answer. KISC asks the player to reason from partial, indirect hints supplied by the organiser; today there is nowhere to put a hint and no way to combine two. A hint is a specification arriving from a third party, and MMA has a place for it.
Read the interaction log back. VBS already logs everything for post-hoc analysis, so this is unusually cheap here. Which hotspots were opened and which skipped, how long each verification took, which candidate was chosen, which query was abandoned and rewritten — all generated by the interaction, none read back. Every rejected hotspot is a label saying the span predictor was wrong here; every chosen candidate is a label saying the aggregator was right. This is the same pattern as in the other three papers, and here the data is already being written to disk.
Give the guidance a clock. Guidance already works — extend it in two directions. Coverage: "you have inspected 3 of 11 hotspots; these 4 are unlike anything you have opened." Time: with forty seconds left, stop advising exhaustive verification and start advising a best-guess submission. In a task scored on speed, time-aware guidance is the guidance that matters most, and it is the one thing the interface never mentions.
Turn a constant provision mapping into an inferred one. Hotspots → timeline, candidates → answer panel, caption and transcript → synchronised panels: all correct, all fixed. Declare them as a grammar and the agent can vary them per question. An OCR-answered question wants the frame zoomed on the text; a speech-answered one wants the transcript line foregrounded and the frame secondary. Same components, chosen per question rather than once for all questions.
Fill in confidence, cheaply. No score attaches to a hotspot boundary or a candidate. The cheapest reliability signal available is disagreement: when the caption says one thing, the transcript another, and the detector a third, that is precisely the case where the machine-generated factual knowledge is unreliable — and it is exactly the failure mode the n-candidate design was built to hedge against. Computing and showing it costs almost nothing.
Supply the domain knowledge the collections need. Two of three datasets are specialist — marine biology and laparoscopic surgery — and the entire online stage reads off a generic captioner's output. Species vocabularies, anatomical terms and surgical phase taxonomies, reachable as expert modules or by retrieval, would improve the caption timeline at its weakest points, which are also the points where the hardest tasks live.
And measure it. The paper reports no evaluation, which is normal for a VBS system description but leaves the central claim untested. VBS supplies task success and time-to-submission; a guidance-oriented system deserves guidance-attributable measures alongside them — how often the submitted answer was among the suggested candidates, how often a hotspot actually contained the answer, recall-at-hotspot, and time-to-verification with and without the answer panel. Those are cheap to log in the same runs that already log everything else.
What the extended version would feel like. A QA task appears. The agent reads the question, decides it is answerable from on-screen text and speech rather than visual similarity, and issues both searches in parallel while also running span prediction. Hotspots appear on the timeline within a second, each with a confidence and a note on which modality supports it; two are flagged because caption and transcript disagree there. Candidate answers appear with linked evidence. The player rejects the first hotspot; the agent narrows the remaining spans and re-ranks. With forty seconds left, the guidance changes tone: two candidates remain plausible, this one is better supported, submit. And every one of those interactions is logged as training signal for next year's span predictor.