license: mit language:

• en
• zh tags:
• multimodal
• vision-language
• audio
• agent
• video-understanding
• long-context

hf download XiaomiMiMo/MiMo-V2.5 config.json tokenizer_config.json --local-dir ./MiMo-V2.5

MiMo-V2.5

1. Introduction

MiMo-V2.5 is a native omnimodal model with strong agentic capabilities, supporting text, image, video, and audio understanding within a unified architecture. Built upon the MiMo-V2-Flash backbone and extended with dedicated vision and audio encoders, it delivers robust performance across multimodal perception, long-context reasoning, and agentic workflows. Key features include:

• Hybrid Attention Architecture: Inherits the hybrid design from MiMo-V2-Flash, interleaving Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and 128 sliding window. This reduces KV-cache storage by nearly 6× while maintaining long-context performance via learnable attention sink bias.

• Native Omnimodal Encoders: Equipped with a 729M-param Vision Transformer (ViT) featuring hybrid window attention and a dedicated audio encoder initialized from the weights of MiMo-Audio, enabling high-quality image, video, and audio understanding.

• Multi-Token Prediction (MTP): Three lightweight MTP modules with dense FFNs accelerate inference via speculative decoding and improve RL training efficiency.

• Efficient Pre-Training: Trained on a total of ~48T tokens using FP8 mixed precision. The context window supports up to 1M tokens.

• Agentic Capabilities: Post-training incorporates SFT, large-scale agentic RL, and Multi-Teacher On-Policy Distillation (MOPD), achieving strong performance on agentic tasks and multimodal understanding benchmarks.

Model Summary

• Architecture: Sparse MoE (Mixture of Experts), 310B total / 15B activated parameters
• Context Length: Up to 1M tokens
• Modalities: Text, Image, Video, Audio
• Vision Encoder: 729M-param ViT (28 layers: 24 SWA + 4 Full)
• Audio Encoder: 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full)
• Multi-Token Prediction (MTP): 329M parameters, 3 layers

2. Downloads

3. Evaluation Results

Multimodal Benchmarks

Coding & Agent Benchmarks

Long Context Benchmarks

4. Model Architecture

LLM Backbone

MiMo-V2.5's core language backbone inherits from the MiMo-V2-Flash architecture, a sparse MoE model with hybrid sliding window attention.

Vision Encoder

We train a dedicated MiMo ViT that adopts sliding-window attention to enable efficient visual encoding.

Window pattern notation: -1 = full attention, 0 = 1-D row window, 1 = 1-D column window.

Audio Encoder

Our audio encoder is initialized from the weights of MiMo-Audio-Tokenizer and further finetuned to support high-quality audio understanding.

5. Training Process

MiMo-V2.5 is trained on a total of ~48T tokens.

1. Text Pre-training: We collect diverse text data for pre-training the LLM backbone.
2. Projector Warmup: Short-duration warmup of multimodal projectors (audio and visual MLP projectors).
3. Multimodal Pre-training: High-quality multimodal data collected for large-scale pretraining.
4. SFT & Agentic Post Training: Supervised fine-tuning with diverse agentic data. During this stage, the context window is progressively extended from 32K → 256K → 1M.
5. RL & MOPD Training: Reinforcement learning for improving perception, reasoning, and agentic capabilities.