Intelligent Director:

An Automatic Framework for Dynamic Visual Composition using ChatGPT

Sixiao Zheng*,
Jingyang Huo*,
Yu Wang,
Yanwei Fu,
Fudan University   *Equal Contribution
arXiv

Intelligent Director consists of four main steps: (1) Caption Generation, (2) Music Retrieval, (3) Video Composition, and (4) Style Transfer. In caption generation, LENS generates descriptions for images and video key frames extracted by pHash, and then ChatGPT creates coherent and storytelling captions and recommends suitable music name. In music retrieval, we utilize the music name recommended by ChatGPT to search a large music library for the best-matched music. In video composition, the seamless integration of captions, images, videos, and music is achieved through four steps: caption fusion, material fine-tuning, switching animation, and music fusion. Finally, in style transfer, AnimeGANv2 transforms the video to other styles, such as the animated style of Kon Satoshi.

Abstract

With the rise of short video platforms represented by TikTok, the trend of users expressing their creativity through photos and videos has increased dramatically. However, ordinary users lack the professional skills to produce high-quality videos using professional creation software. To meet the demand for intelligent and user-friendly video creation tools, we propose the Dynamic Visual Composition (DVC) task, an interesting and challenging task that aims to automatically integrate various media elements based on user requirements and create storytelling videos. We propose an Intelligent Director framework, utilizing LENS to generate descriptions for images and video frames and combining ChatGPT to generate coherent captions while recommending appropriate music names. Then, the best-matched music is obtained through music retrieval. Then, materials such as captions, images, videos, and music are integrated to seamlessly synthesize the video. Finally, we apply AnimeGANv2 for style transfer. We construct UCF101-DVC and Personal Album datasets and verified the effectiveness of our framework in solving DVC through qualitative and quantitative comparisons, along with user studies, demonstrating its substantial potential.

▶ Personal Album Dataset

▶ UCF101-DVC Dataset

▶ Style Transfer

Miyazaki Hayao Kon Satoshi Makoto Shinkai

BibTeX

@misc{zheng2024intelligent,
        title={Intelligent Director: An Automatic Framework for Dynamic Visual Composition using ChatGPT}, 
        author={Sixiao Zheng and Jingyang Huo and Yu Wang and Yanwei Fu},
        year={2024},
        eprint={2402.15746},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }