Motion-Conditioned Image Animation for Video Editing


Wilson Yan1,2, Andrew Brown1, Pieter Abbeel2, Rohit Girdhar1, Samaneh Azadi1

1GenAI, Meta       2UC Berkeley

Paper Dataset

Abstract

We introduce MoCA, a Motion-Conditioned Image Animation approach for video editing. It leverages a simple decomposition of the video editing problem into image editing followed by motion-conditioned image animation. Furthermore, given the lack of robust evaluation datasets for video editing, we introduce a new benchmark that measures edit capability across a wide variety of tasks, such as object replacement, background changes, style changes, and motion edits. In addition, we present a comprehensive human evaluation of the latest video editing methods along with \ours, on our proposed benchmark. MoCA establishes a new state-of-the-art, outperforming notable recent approaches including Dreamix (63%), MasaCtrl (75%), and Tune-A-Video (72%), with especially significant improvements for motion edits.

Style Edits

Several goldfish swim in a tank.
↓
Several goldfish swim in a tank, impressionist style.

Three cows graze together in a green pasture.
↓
Three cows graze together in a green pasture, pointilism style.

Background Edits

A ship sails on the sea during sunset.
↓
A ship sails on the lunar surface during sunset.

An airplane flies through the blue sky leaving a contrail behind it.
↓
An airplane flies over the Taj Mahal leaving a contrail behind it.

Object Edits

A beautiful lotus in river water on a rainy day.
↓
A beautiful poppy flower in river water on a rainy day.

A static shot of red roses in sunlight, gently swaying in the breeze.
↓
A static shot of rainbow roses in sunlight, gently swaying in the breeze.

Multi-Spatial Edits (Combo of style / background / object edits)

The sun setting with clouds moving around it.
↓
The moon setting with clouds moving around it, reflecting on a flooded road.

Rain falling on a stone pathway in super slow motion.
↓
Snow falling on a fantasy landscape in super slow motion, with dramatic lighting.

Motion Edits

Riding a boat over the ocean
↓
Huge waves crash while riding a boat over the ocean

A duckling looking for food in grass
↓
Zooming out from a duckling looking for food in grass

Multi-Motion Edits (Combo of motion + spatial edits)

A bright pink flower
↓
Timelapse of an orange lily fully blooming

Fresh apricots hanging off a tree
↓
Ripe apples fall off a tree


Comparisons Against Baselines

A panda sitting down and eating from a pile of bamboo → A panda sitting down and playing in a pile of colorful ribbons

A duck swimming on water → A duck shaking wings on water

Two grey sharks swim in the blue ocean on a coral reef. → Two grey sharks swim in the blue ocean on a coral reef, surrounded by various colorful fish.

An audi q7 goes on a snow trail. → An audi q7 goes on a desert trail.

A white jeep driving down a gravel road → A white jeep driving down a dirt road

A turtle laying on a green floor → A turtle laying on a green floor retracts into its shell

A scene of a calm lake → A scene of a lake with crashing waves

The sun setting with clouds moving around it. → The sun setting with clouds moving around it, 2d vector animation.

A time lapsed video of a ferris wheel moving. → A time-lapsed video of a ferris wheel moving with a sunset in the background.

A duck swimming on water → A duck swimming on water with big waves

Fresh apricots hanging off a tree → Ripe apricots fall off a tree

A duckling looking for food in grass → A duckling jumping on a pile of leaves

Three cows graze together in a green pasture. → Three sheep graze together in a green pasture.

A bright pink flower → Timelapse of a bright pink flower fully blooming

A duck swimming on water → A duck dunks its head underwater

A ship sails on the sea during sunset. → A ship sails on the sea during sunset, 2d vector art.

A bird sitting on a rock in a river → A red cardinal takes flight off a rock

A boar sniffs the dirt looking for food → A boar digs a hole in the ground

Rain falling on a stone pathway in super slow motion. → Snow falling on a stone pathway in super slow motion.

Bright red leaves on a tree during autumn → Bright red leaves fall off a tree during autumn

A duck swimming on water → A swan swimming on water

Goldfish swimming around in a lake → Blue fish swimming around in a lake

Bright red leaves on a tree during autumn → Bright red leaves on a tree during autumn, windy day, rustling leaves

A scene of a calm lake → A scene of a left to right pan of a calm lake

A scene of a calm lake → A time-lapse of a calm lake