I collaborated with AI to Edit an Animated Music Video

AI has undoubtedly become the buzzword of the year, making a significant impact in our lives. While some people find it exciting, others also find concerns and fears. The widespread emergence of AI in 2023 has sparked numerous questions, also among video editors. Will it render us obsolete, or will it enhance the quality of our work? At Premiere Gal, my goal is to ensure that you can harness the power of AI to your advantage. That's why I've been exploring some of the best AI video tools available. In our latest endeavor, I took on a challenge.

I wanted to test the feasibility of creating a fully animated music video for a real artist using an affordable AI tool, all while examining the amount of work left for the creator. I set a budget constraint of $200, and that's where Gen-1, an AI animation generator developed by Runway, came into play. Gen-1 proved to be the perfect tool for effortlessly adding any animation style to your video, eliminating the need for laboriously animating each and every frame separately.

I was fortunate to discover Heartbreak Kid, a talented band, through Musicbed, my go-to platform for licensing high-quality music for our YouTube videos. Notably, Musicbed is also the generous sponsor of the behind-the-scenes video 😉. When I shared my idea of creating an AI video for their brand new single "Showtime" with the band, they were excited! They entrusted me with creative freedom, allowing me to bring my imaginative vision to life.

PS. If you will get inspired and feel like producing your own AI video get your 10% off a Runway Gen-1 magic tools plan using code GAL.

Step 1. Licensing a song

If you are using somebody else’s music, remember to license it. It is pretty easy with a Musicbed subscription.

Step 2. Develop the concept and story

Before delving into the editing process using software like Premiere Pro, it's essential to determine the story you want to tell and how you intend to convey it through your video. Regardless of the video's genre or purpose, having a well-thought-out plan for both the audio and visual elements is crucial.

Personally, I find it helpful to begin with an Audio-Visual Editing Script, which I typically create using Google Docs. This script takes the form of a two-column table, with the audio descriptions on the left and the corresponding visual elements on the right. This approach allows for a structured and organized workflow, ensuring that the audio and visual components align seamlessly to tell a coherent and engaging story.

When it comes to creating a music video, it's evident that placing the lyrics on the left side of the script is essential. This way, you can have a clear reference to brainstorm ideas for the corresponding visuals on the right side. Having the lyrics in front of your eyes serves as a catalyst for inspiration, allowing you to develop creative concepts that align with the music.

In the case of our cosmic-themed AI music video, this approach proved fruitful in conjuring up the captivating story. The lyrics served as a foundation for the visual narrative, guiding imagination towards a cosmic storyline that complements the music. By leveraging this creative process, I was able to merge the style of the lyrics with visually stunning elements, resulting in an engaging and cohesive music video experience.

Step 3. Create a Video Storyboard

For our music video, I went through a vast selection of clips and carefully chose those that seemed to align well with specific lines from the script. Once I had gathered all the suitable stock video clips, I imported them into Premiere Pro and arranged them on the timeline in the Assembly View, ensuring they fit seamlessly with the desired story. Afterward, I exported the video from Premiere Pro and uploaded it into Runway.

Step 4. Use AI to generate animations

To generate AI animations, I utilized the Gen-1 AI animation tool in Runway. The uploaded footage was automatically split into clips using the Scene Detection tool. This allowed me to work with each shot separately, which was crucial for the main part of the AI video creation process. For the animation style, I chose a "retro cosmic" look, using a written prompt as my reference. Within the advanced settings, I adjusted parameters like "Structural consistency" to determine how closely the animated shot resembled the original footage, and the "Iight" to control the level of reference style applied.

The important part here is ‘seed’. The number you see there is one of the millions of variables that AI can come up with. Therefore, when you are searching for that perfect style you want the ‘seed’ setting to be randomized. Once you choose the style that you want to maintain through all of your shots, remember to fix the ‘seed’. 

Each shot took a few minutes to generate, and I downloaded each AI animation individually. To experiment further with the chosen settings, use the "Generate style previews" feature to explore numerous propositions provided by the AI animation generator.

Considering that I had over 100 individual clips to animate, the process was time-consuming. However, the AI animation generator proved invaluable, as animating such a video manually would have been impractical. Afterward, I imported the generated clips into a separate bin in Premiere Pro, using the icon view to organize them. I then placed the comic book-style animations on top of the live-action footage on the timeline. Adding transitions and special effects from Gal’s very own Premiere Gal Toolkit enhanced the video, emphasizing crucial moments.

Step 5. Animate the lyrics

In the next step, I focused on animating the lyrics or any other text in the video. I employed a plugin called SubMachine for Premiere Pro, which automated the animation of the song lyrics. To see how it works watch the full tutorial on my YouTube channel.

And that was it! Premiere Gal’s first AI video was created! Now, all that remained was to showcase it to the artists. Throughout this entire process, the goal was to see the capabilities of AI tools while observing how much work is still on the creator. Here's the good news: video editors won't become redundant anytime soon (yay!). The majority of the video editing work still fell on us, with the AI animation generator serving as a useful and affordable tool to create animations. However, it's important to acknowledge that AI tools are still in their beta versions, indicating room for improvement (check the video for the feedback I provided for Gen-1). Considering the rapid progress of AI in recent months, it's plausible that the best AI video tools like Gen-1 by Runway will continue to upgrade faster than expected.

There's more good news! I managed to stay within the budget. 

Moreover, the Heartbreak Kid liked the AI video so much that they decided to use it as their official music video! Watch the final music video here:

Many thanks to Heartbreak Kid for trusting our vision and to Musicbed for funding this project!