Motionctrl: A unified and flexible motion controller for video generation

Z Wang, Z Yuan, X Wang, Y Li, T Chen, M Xia… - ACM SIGGRAPH 2024 …, 2024 - dl.acm.org
Z Wang, Z Yuan, X Wang, Y Li, T Chen, M Xia, P Luo, Y Shan
ACM SIGGRAPH 2024 Conference Papers, 2024dl.acm.org
Motions in a video primarily consist of camera motion, induced by camera movement, and
object motion, resulting from object movement. Accurate control of both camera and object
motion is essential for video generation. However, existing works either mainly focus on one
type of motion or do not clearly distinguish between the two, limiting their control capabilities
and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion
controller for video generation designed to effectively and independently control camera and …
Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods. Project page: https://wzhouxiff.github.io/projects/MotionCtrl/ .
ACM Digital Library
以上显示的是最相近的搜索结果。 查看全部搜索结果