多模态模型、视觉语言模型与跨媒体理解

Gemma 4 12B features an encoder-free multimodal architecture that runs locally on 16GB VRAM devices with native audio support. By eliminating separate vision and audio encoders, it reduces latency and pairs with a dedicated MTP model for faster inference, marking the first mid-sized multimodal model with a macOS desktop app for fully offline interaction.

NVIDIA launches Cosmos 3, the first unified multimodal model integrating language, video, sound, and action inputs/outputs, built on Mixture of Transformer architecture, open-sourced with weights available on Hugging Face, achieving top scores across physical AI benchmarks including Robo Lab, PiBench, and Vintage.

OpenAI introduces three real-time audio models to enhance voice applications.

全球首个医疗视频理解大模型uAI Nexus MedVLM开源，包含6k+精标测试集，覆盖多种手术场景，准确率远超通用大模型。

IBISAgent通过多步交互决策重新定义医学图像分割，解决了隐式token导致的推理退化问题，显著提升分割精度。

商汤开源了全新架构模型SenseNova-U1，通过NEO-unify架构统一图像理解与生成，支持连续性图文创作和高密度信息图处理。

OpenAI released three new voice models: GPT-Realtime-2, GPT-Realtime-Translate, and GPT-Realtime-Whisper, significantly enhancing dialogue, translation, and real-time transcription capabilities.

Gemma-4 12B is an encoder-free, unified multimodal model that runs directly on laptops with 16GB VRAM. It matches the performance of the 26B MOE with less than half the memory footprint, ships with Hermes and agent tools, macOS Edge Gallery, and RTLM, and is released under Apache 2.0.

Gemma 4 12B is a unified, encoder-free multimodal model bringing high-performance multimodal intelligence to your laptop. It matches the performance of our 26B MoE at less than half the memory footprint, supports native audio inputs, and runs locally on 16GB VRAM hardware with low-latency multi-step reasoning.

Tomofun significantly reduced inference costs for vision-language models in pet behavior detection using AWS Inferentia2 chips, while maintaining high accuracy and throughput for large-scale real-time monitoring.

Meta发布SAM 3.1，通过对象多路复用和全局推理实现更快、更高效的实时视频检测与跟踪。

美团开源原生多模态模型LongCat-Next，通过离散Token统一建模视觉、语音与文本，实现理解与生成的对称架构。

Google在Gemini中推出动态视频理解功能，减少88% token和66%成本。

Google推出Gemini 3.8 Flash模型，专为复杂任务和多步骤代理设计，提升推理与视频理解能力。

Google DeepMind推出Gemini模型的agentic视频理解功能，可减少88%的token消耗，降低成本66%，并提升7%准确性。