I Turned Doodles into Video Clips in Minutes with Runway’s New AI Video Tool — Here’s How

Follow ZDNET: Add us as a favorite source On Google.

ZDNET Highlights

• A new feature in Runway lets you create videos from doodles.
• It’s impressive, but it performs best when handled with a little care.
• Access requires payment for (at least) a standard Runway account.

A family of settlers, admiring a herd of bison, run into two monstrous winged creatures flying overhead. The little girl passes through the fence post like a ghost.

This surreal five-second scene was created thanks to a new feature called Motion Sketch within the AI ​​platform Runway. As the name suggests, it creates videos from simple doodles that you can draw directly on top of a still image.

In the case of the clip below, I started by crafting an image using Google’s Nano Banana Pro — one of the text-to-image models offered through Runway’s platform — then simply drew bird-like shapes and two arrows pointing in the direction in which the family and the bison should run. I added some text prompts for a little extra guidance, and the model did the rest.

This feature still needs some help, just as ChatGPT often requires many finely crafted prompts to produce the output you’re looking for. Still, this could be a big step forward for AI-generated videos and professionals in creative industries who prefer to think visually more than verbally.

“This feature is useful for all types of users,” Runway product manager Aditi Poduval told ZDNET. “Ultimately, we are helping people achieve precise movement without the need to prepare a specific written prompt.”

This is what I’ve learned after working with Motion Sketch.

my experiments

After my initial experiment with the image of the settler family lying in bed, I decided to increase the sketchiness, so to speak.

I created the family image using a text prompt. But the Nano Banana Pro via Runway can also create an image based entirely on a doodle, which you can then draw directly on the Microsoft Paint platform. I’m a writer, not a visual artist, but I did my best to depict what I hoped would look like a snake hanging from a tree branch.

Then, I turned to motion sketch. As mentioned, I met the model halfway through the Settler Family experiment by adding some text prompts in addition to my doodles, but this time I wanted to see how well it would perform without any written guidance. I hoped to create a clip of the snake slithering to the right, so I created some simple motion-directed arrows along the length of its body. Any six-year-old child will be able to understand what those arrows mean, but how will the AI ​​perform? I set the length to 10 seconds, selected Runway Gen-4.5 Select the video model (Google’s Veo 3 and Veo 3.1 are also available), and click Generate.

I was expecting a relatively smooth clip of the snake moving along the branch and exiting the bottom-right of the shot, but what I got was much trippier. The snake on the left side of the shot released a part of its body, and the other snake came out of its body and fell out of the tree. At one point, it appeared to have grown crocodile-like legs, which it used to drag itself along the branch. The original arrows I drew also appeared briefly at the beginning of the clip before disintegrating.

Frankly, snakes are a biological marvel, and accurately modeling their movement is no small task for any animator – human or machine. And the model misinterpreted my multiple arrows as instructions for the snake to move in multiple directions at once, clearly confusing it. So I went back and added a simple text prompt with arrows to my original image: “The snake crawls along the branch.”

The final product this time was a little closer to what I had in mind (no sudden duplication of the snake’s body or sprouting of legs). The motion still seemed strange, but again, snake motion is a fairly tall question, involving a lot of complex microscopic physics.

Another experiment: I wanted to see how well the model could work with real photos, so I uploaded one from my laptop: a shot I took of Prospect Bark in Brooklyn a few months ago.

I drew what was effectively a universal symbol for fire: wavy red lines accented with orange and extending upward. Again, I make no claims of being a talented painter, but most humans will be able to recognize what I’m going for with this. The runway models picked it up too, and dropped a huge bonfire in the center of the tranquil lakeside scene captured in my photo. Then, my sketch appeared briefly in the clip before disappearing in a psychedelic motion of gradually shortening squiggly lines. And maybe I’m imagining it, but even the trees next to the bonfire were turning black from its heat.

takeaway

Like any other generic AI tool, Motion Sketch also has its flaws and glitches (the little girl crossing a wooden beam, a snake growing a leg, the wavy lines of my original sketches appear briefly in the final video clip). Still, I’m impressed.

Here’s a glimpse of a whole new format for creating video with AI, one that could lower the barrier between imagination and production than ever before. “This is what I’ve been waiting for,” said one X user. Posted Below the runway’s Tuesday announcement. “Creating your vision and then seeing it presented as a video? Absolutely a game changer for creators who can’t engineer a prompt their entire life.”

Now, it still requires a lot of trial and error if you have a specific vision in mind, but nonetheless, it’s an exciting preview of what the future could hold. And it’s definitely worth playing with it, even – and maybe especially – if you’re just going into it in the spirit of creativity, without any clear vision of what you actually want to produce.

how to reach

Motion Sketch requires (at least) a standard subscription to Runway, which costs $12 per user per month and comes with 625 monthly credits.

To access from your user dashboard, click “Apps” in the left-hand menu, then select Motion Sketch below the “Explore Gen-4.5” collection or below the preview window on the right half of the screen, where you’ll see a button labeled “Try it now.”

From there, simply upload your image and click “Sketch” -> “Export Sketch” -> Generate.