Overview

VLM hallucination benchmarking, multimodal safety, AI-generated content detection, and knowledge-graph-grounded vision-language reasoning. Foundation work for recent developments in vision-language model safety and grounding.

Details

This research cluster investigates the reliability, safety, and structural grounding of Vision-Language Models (VLMs). Work spans hallucination taxonomy and benchmarking across text, image, and video modalities; safety alignment for multimodal systems; detection of AI-generated visual content; and knowledge-graph-conditioned reasoning to reduce model drift and factual error. This foundational work forms the basis for our recent developments in vision-language model safety and alignment.

Project Motive

VLMs inherit the hallucination and alignment vulnerabilities of LLMs while introducing new failure modes through the visual modality — unsafe outputs triggered by images, identity drift in video generation, and AI-generated content that evades detection. This project builds the diagnostic tools, benchmarks, and grounding methods needed to make VLMs safer and more factually reliable. The insights and methods developed here continue to inform our work on advanced safety mechanisms and grounding strategies for modern multimodal systems.

Where we are

• Introduced the first fine-grained taxonomy of hallucination across degree, orientation (factual mirage vs. silver lining), and category for both LLMs and multimodal models.
• Built ViBe, the first large-scale benchmark specifically for hallucination in text-to-video generation, annotating five distinct failure types across 3,782 videos.
• Introduced VCT², a 166K-image benchmark revealing that state-of-the-art AI-generated image detectors achieve only ~58% accuracy on outputs from contemporary models — exposing a critical generalization gap.
• Proposed IMAGINATOR, a joint image+text embedding that extends the distributional hypothesis to the visual modality for the first time at object-level grounding.
• Applied causal knowledge graphs to ground visual scene understanding in autonomous driving and VQA, bridging neuro-symbolic reasoning with VLM outputs.

Status

Active and ongoing. Current open threads include hallucination mitigation in video generation, prompt-level interventions for multimodal factuality, and knowledge-graph-conditioned grounding for VLMs.