VerseCrafter

• Dynamic Realistic Video World Model: VerseCrafter learns a realistic and controllable video world prior from large-scale in-the-wild data, handling challenging dynamic scenes with strong spatial-temporal coherence.
• 4D Geometric Control: A unified 4D control state provides direct, interpretable control over camera motion, multi-object motion, and their joint coordination, improving geometric faithfulness.
• Frozen Video Prior + GeoAdapter: We attach a geometry-aware GeoAdapter to a frozen Wan2.1 backbone, injecting 4D controls into diffusion blocks for precise control without sacrificing video quality.
• VerseControl4D Dataset: We introduce a large-scale real-world dataset with automatically rendered camera trajectories and multi-object 3D Gaussian trajectories to supervise 4D controllable generation.

Framework of VerseCrafter. We render 4D geometric controls (camera and 3D Gaussian trajectories) as multi-channel maps and inject them into a frozen Wan2.1 backbone via GeoAdapter for geometry-consistent generation.

VerseCrafter is a controllable video world model for real-world scenes. It provides explicit 4D geometric control by allowing users to specify a target camera trajectory and multi-object 3D Gaussian trajectories, so the generated video follows both viewpoint changes and object motions with strong spatiotemporal consistency. Given a single reference view, we lift the scene into geometry-aware controls and render them into per-frame maps that serve as conditioning signals for diffusion-based video synthesis. VerseCrafter is trained on VerseControl4D, a large-scale in-the-wild dataset with automatically derived camera trajectories and multi-object 3D Gaussian trajectories, enabling robust control across diverse dynamic and static scenes.

Frozen Wan2.1 Backbone

We adopt Wan2.1 as a frozen latent video diffusion prior. Keeping the backbone unchanged preserves its strong realism and generalization, while our geometry-aware controls learn to steer generation without degrading visual quality.

GeoAdapter for 4D Control

A lightweight GeoAdapter encodes the rendered 4D control maps and injects them into selected diffusion blocks as zero-initialized residual modulations. This design enables precise camera and multi-object motion control while maintaining sharp, geometrically coherent videos.

Training a video world model with precise 4D geometric control requires large-scale data with accurate annotations. We introduce VerseControl4D, a dataset constructed from Sekai-Real-HQ and SpatialVID-HQ with complete geometric supervision.

Dataset construction pipeline. Starting from Sekai-Real-HQ and SpatialVID-HQ, we extract 81-frame clips and apply quality filtering. For each retained clip, Qwen2.5-VL-72B, Grounded-SAM2, and MegaSAM provide captions, object masks, depth, and camera poses, which are lifted into background/object point clouds, fitted with 3D Gaussian trajectories, and rendered as background/trajectory maps plus a merged mask that constitute our 4D Geometric Control.

VerseControl4D contains 35,000 training clips and 1,000 validation/test clips. Overall, 26% of the clips come from Sekai-Real-HQ and 74% from SpatialVID-HQ, reflecting their complementary scene coverage.

To support both camera-only world exploration and coordinated camera-object control, VerseControl4D includes dynamic scenes (clips with salient foreground object motion together with camera motion) and static scenes (clips with negligible object motion and only camera movement). About 20% of the training clips are static scenes, and the validation set additionally includes 250 static-scene clips for dedicated camera-only evaluation.

Dataset Examples