多模态模型、视觉语言模型与跨媒体理解

Gemma 4 12B features an encoder-free multimodal architecture that runs locally on 16GB VRAM devices with native audio support. By eliminating separate vision and audio encoders, it reduces latency and pairs with a dedicated MTP model for faster inference, marking the first mid-sized multimodal model with a macOS desktop app for fully offline interaction.

NVIDIA launches Cosmos 3, the first unified multimodal model integrating language, video, sound, and action inputs/outputs, built on Mixture of Transformer architecture, open-sourced with weights available on Hugging Face, achieving top scores across physical AI benchmarks including Robo Lab, PiBench, and Vintage.

OpenAI introduces three real-time audio models to enhance voice applications.

全球首个医疗视频理解大模型uAI Nexus MedVLM开源，包含6k+精标测试集，覆盖多种手术场景，准确率远超通用大模型。

IBISAgent通过多步交互决策重新定义医学图像分割，解决了隐式token导致的推理退化问题，显著提升分割精度。

商汤开源了全新架构模型SenseNova-U1，通过NEO-unify架构统一图像理解与生成，支持连续性图文创作和高密度信息图处理。

OpenAI released three new voice models: GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper, significantly enhancing dialogue, translation, and real-time transcription capabilities.

Gemma-4 12B is an encoder-free, unified multimodal model that runs directly on laptops with 16GB VRAM. It matches the performance of the 26B MOE with less than half the memory footprint, ships with Hermes and agent tools, macOS Edge Gallery, and RTLM, and is released under Apache 2.0.

Gemma 4 12B is a unified, encoder-free multimodal model bringing high-performance multimodal intelligence to your laptop. It matches the performance of our 26B MoE at less than half the memory footprint, supports native audio inputs, and runs locally on 16GB VRAM hardware with low-latency multi-step reasoning.

Tomofun significantly reduced inference costs for vision-language models in pet behavior detection using AWS Inferentia2 chips, while maintaining high accuracy and throughput for large-scale real-time monitoring.

Meta发布SAM 3.1，通过对象多路复用和全局推理实现更快、更高效的实时视频检测与跟踪。

美团开源原生多模态模型LongCat-Next，通过离散Token统一建模视觉、语音与文本，实现理解与生成的对称架构。

ARC Prize 2026宣布ARC-AGI-3里程碑奖，Tufa Labs的The Duck方案通过Python REPL实现交互式推理，Reki使用视觉语言模型。

苹果提出VICIS任务，解决视觉语言模型从图像集合推断概念的难题，新框架在ImageNet数据上实现更准确的生成。

苹果提出LVSum基准，揭示当前多模态大模型在长视频摘要任务中存在时间定位和跨模态一致性缺陷。

阶跃在WAIC 2026展示智能体手机和多模态模型，推动AI落地终端。模型矩阵覆盖端云协同，STEPX Neo手机实现跨设备任务自动化。

Thinky发布975B参数多模态模型Inkling，支持文本/图像/音频，开放权重且兼容Huggingface等平台。

AI在学习效率上仍不如婴儿，婴儿大脑的机制可能为AI提供新方向。Meta等机构通过EgoBabyVLM挑战赛揭示当前AI模型处理婴儿视角数据时的局限性。

Inkling是Thinking Machines Lab推出的多模态模型，支持高效推理和跨任务能力，Together AI提供生产级部署服务。

Bonsai 27B通过1-bit量化技术首次实现27B级模型在iPhone 17 Pro上的运行，性能保留全精度90%。