Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresight

Mantis is a versatile vision-language-action model that empowers robots to perform complex manipulation tasks through innovative disentangled visual foresight, progressive training, and adaptive temporal integration mechanisms. The key features of Mantis include: disentangled visual foresight efficiently captures latent action trajectories without overburdening the backbone network; a progressive training strategy introduces multimodal data in stages, effectively preserving and enhancing the language understanding and reasoning capabilities of the vision-language model backbone; and adaptive temporal integration dynamically adjusts the integration strength, ensuring control stability while reducing inference costs. Mantis demonstrates outstanding performance in robotic manipulation tasks, including strong generalization capabilities for both in-domain and out-of-domain instructions. It is meticulously fine-tuned based on multi-stage pretraining and multiple datasets.

• Disentangled Visual Foresight: Automatically captures latent actions depicting visual trajectories without adding extra burden to the backbone network.
• Progressive Training: Introduces modalities in stages to preserve the language understanding and reasoning capabilities of the Vision-Language Model (VLM) backbone.
• Adaptive Temporal Integration: Dynamically adjusts the temporal integration strength to maintain stable control while reducing inference costs.

Models and Datasets

Model Versions

Datasets Used

Please first download the LIBERO datasets and the base Mantis model.