📄 中文摘要
AMVICC 是一个新颖的基准，专注于系统性地比较视觉语言模型（VLMs）和图像生成模型（IGMs）在图像到文本和文本到图像任务中的故障模式，从而实现视觉理解的跨模态评估。尽管机器学习领域发展迅速，但现有的视觉语言模型在理解或生成基本视觉概念（例如物体方向）方面仍存在缺陷。通过创建特定的测试场景和评估指标，AMVICC 旨在揭示这些模型在处理空间关系、属性识别、逻辑推理以及复杂场景理解上的局限性。基准设计涵盖了多种视觉推理挑战，包括但不限于物体位置、数量、形状、颜色及其相互关系。通过对 MLLMs 和 IGMs 在这些任务上的表现进行定量和定性分析，该研究旨在识别不同模态之间共享的或特有的视觉理解障碍。例如，一个 VLM 可能难以准确描述一个“倒置的杯子”，而一个 IGM 可能无法根据“一个红色的球在蓝色的盒子里”的描述生成正确的图像。AMVICC 提供的系统性故障模式剖析有助于研究人员更深入地理解当前视觉AI模型的瓶颈，为开发更鲁棒、更具泛化能力的下一代视觉智能系统提供指导。这项工作强调了在视觉推理任务中，特别是对于细粒度视觉概念的理解和生成，当前模型仍有显著的改进空间。
📄 English Summary
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
AMVICC introduces a novel benchmark designed for systematically comparing the failure modes of both multimodal large language models (MLLMs) and image generation models (IGMs) across image-to-text and text-to-image tasks, thereby enabling cross-modal evaluation of visual understanding. Despite rapid advancements in machine learning, existing vision language models (VLMs) frequently struggle to comprehend or generate basic visual concepts, such as object orientation. By establishing specific test scenarios and evaluation metrics, AMVICC aims to uncover the limitations of these models in processing spatial relationships, attribute recognition, logical reasoning, and complex scene comprehension. The benchmark's design encompasses a diverse range of visual reasoning challenges, including but not limited to object position, quantity, shape, color, and their interrelations. Through quantitative and qualitative analysis of MLLMs' and IGMs' performance on these tasks, the research seeks to identify shared or modality-specific visual understanding obstacles. For instance, a VLM might struggle to accurately describe an 'inverted cup,' while an IGM might fail to generate an image correctly depicting 'a red ball inside a blue box' based on the textual description. The systematic failure mode profiling provided by AMVICC assists researchers in gaining deeper insights into the bottlenecks of current visual AI models, offering guidance for developing more robust and generalizable next-generation visual intelligence systems. This work underscores that significant room for improvement remains in visual reasoning tasks, particularly concerning the understanding and generation of fine-grained visual concepts by current models.