Hollywood is notorious for its celebrity excess, but in Tinseltown there is strict control over one scandalous indulgence: swearing.
Director Scott Mann encountered these constraints after filming the thriller Falling. Film giant Lionsgate – best known for John Wick, SeenAnd Hunger Games Franchises – wanted to release the film in the US. But the studio had big problems. Thirty-six of them, to be exact.
“They said there was too much shit,” Mann tells TNW in a video call from LA.
All these F-bombs were pushing Falling towards an R rating, which would reduce the potential audience. To secure the PG-13 needed to expand its reach, these profanities needed to go away.
The 💜 of EU technology
The latest rumors from the EU tech scene, a story from our wise founder Boris and questionable AI art. It's free in your inbox every week. Register now!
Easier said than done. Reshoots would cost a bomb and post-production magic couldn't get rid of the dirty words. Luckily, Mann had another trick up his sleeve. The British filmmaker had quietly built a startup called Flawless that develops AI video editing tools. Falling provided a new field test: swapping F-bombs for gentler epithets.
Mann asked the cast to record clearer words. Once the sound was ready, the Flawless system went to work. The software first converted the actors' faces into 3D models. Neural networks The performances were then analyzed and reconstructed. Facial expressions and lip movements were synchronized with the new dialogue.
The experiment proved successful. All 36 F-bombs were replaced without a trace. Well, almost all of them. “I ended up doing shit,” Mann says. “Apparently I’m not allowed to do shit.”
The evaluation committee was satisfied with his restraint Falling the coveted PG-13. The film became a box office hit and earned revenue a reported $21 million on a budget of just $3 million. A sequel is currently being filmed in Thailand.
Inspired by his success, Mann began commercializing the software. The latest version is DeepEditor, an AI tool that refines dialogue and performances.
The studio system
DeepEditor can shorten lines, insert pauses or change delivery times. It's even possible to copy and paste performances from one recording to another. All outputs offer Hollywood-quality 4K resolution, 16-bit color depth and ACES color spaces.
Early access applications for the tool are now possible. A full product release is planned for the first half of this year.
“It’s already changing the places where people shoot,” Mann said. “And as it expands, I think it will completely change the way we make films.”
Nor is it the only tool Mann wants to use to transform films. About a decade ago, he began developing another AI system for filmmaking. Like DeepEditor, it started on a Hollywood set.
The big breakthrough
After completing film school, British television and short films, Mann got his big break in Hollywood in 2014. Lionsgate had offered him the director's chair for the crime film Robbery. Also on board was an all-star cast led by Robert De Niro.
Husband enjoyed the experience. “It was an absolute privilege. We were very close to the film and very happy with the English language version. But then I saw a foreign translation of the film.”
Mann was “horrified” by the synchronization. His script had been rewritten and the actors' gestures had changed. He found that the culprit was widespread throughout the industry.
The problem stemmed from Hollywood's established translation process. When dubbing films, the scripts are usually rewritten to match the original mouth movements. If the new lines still don't match the old gestures, voice actors try this synchronize the two by twisting their delivery in unnatural directions. The results range from amusing to annoying.
“It's really bad for the filmmakers and actors because it's not an authentic representation of their work,” Mann says. “And as an experience, if it’s not in sync, you’re not immersed.”
Mann began to investigate novel synchronization techniques. He examined head scans, but the depiction lacked realism. The synchronization just moved from one uncanny valley to the next.
Mann lost faith in established VFX and began looking beyond the film industry. He soon came across a promising alternative: Profound video portraits.
Hollywood meets GenAI
Deep Video Portraits was introduced in 2018 and was a major breakthrough for the emerging generative AI sector. The technique enables photorealistic reanimation of faces with just an input video. Every facial movement and lip movement can then be synchronized with the speech.
The lifelike results amazed observers, including Mann. “It blew me away,” he says.
Mann turned to the research team. They agreed to work together on a new technical test: having De Niro's character speak German.
The transformation, Mann says, was “like magic.”
“It was really understanding how a particular actor might say a particular line…You keep the performance the same, but you can change the synchronicity.”
Expressions are transmitted digitally from one person to another. Photo credit: Kim et al.
Mann believed the technology was ideal for Hollywood. To turn the idea into a business, he sought advice from technology industry veteran Nick Lynes. The two founded Flawless together in 2018.
The startup's first product was TrueSync, a synchronization tool that studios apply to Hollywood films. Among them is Venom: The Last Dancea Marvel blockbuster released last year.
Flawless also presented a Sizzle roll of AI-translated trailers at this year's Cannes Film Festival. Still, not every customer is ready to brag about the results.
Threatening actions
As the premiere of Fall approached, Lionsgate grew anxious. GenAI was still a novel term at the time, but unions were already concerned about the threat to artists. The studio feared that the film's visual dubbing would spark a backlash.
“They would pull the release if the matter wasn't resolved with the Screen Actors Guild and there was a lot of nervousness,” Mann recalls. “But luckily we had the consent workflows planned and [rights protections] early on.”
Flawless built the plan on several pillars. All data would come from legitimate sources – and not be cracked without permission like so many GenAI companies do. There would be a full rights release for each issue. The acting would remain faithful to the original performances. Additional consent would be required for significant changes.
The launch also limited the system's operation. “We often call our models 'narrow models,'” says Mann. “They are large, but they focus on a specific aspect and are curated for a specific purpose. They are very targeted and based on clean data that can be used for this purpose.”
Flawless presented the plan to the Screen Actors Guild (SAG). “They gave it the thumbs up,” Mann says. “Fall” was released in cinemas in the USA in August 2022. The film – and the dubbing – were huge successes.
Just a few months later, GenAI exploded into the mainstream. The trigger was the launch of ChatGPT in November. A wave of image, text and audio generators followed closely behind her. Suddenly, AI's threats to actors, artists, and copyrights had become a public concern.
Another shot
In July 2023, the actors union SAG-AFTRA began the longest strike in its history. One of the guild's main concerns was the threat of AI.
After months of intense negotiations, the union reached an agreement with Hollywood's leading companies. Under the agreementAny digital changes would require express consent unless they are “substantially consistent with the script, performance and/or recording.”
Mann welcomed the conditions. They would not restrict lip-syncing in foreign language dubs, but would enforce strict consent requirements for any significant changes to the script or performance.
The new rules opened up business opportunities for Flawless. By supporting union regulations, the startup hopes to gain a quick foothold in Hollywood.
Mann has made actor rights a core tenet of the Flawless product line. Credit: Immaculate
A month after the SAG-AFTRA strike began, Flawless unveiled a new rights management platform. Called the Artistic Rights Treasury (ART), the system shares AI-generated edits with artists. If the actor agrees to the changes, they can consent within the app. If they don't like the new versions, they can submit their own takes.
ART has now been integrated into DeepEditor. Mann believes the mix of AI editing and security measures creates a unique product. “DeepEditor will be the first legitimate enterprise AI solution on the market,” he says. “Everything else is riddled with controversy and legal questions.”
A better future for Hollywood?
Mann expects GenAI to open up endless possibilities for filmmakers over time. He envisions falling costs, less drudgery and lower barriers to entry. If everything goes according to plan, Hollywood will have a taste for originality again.
“The key to thriving in this industry is innovation – and the responsible use of innovation,” says Mann.
But he also has to learn a lot about responsible work. During production for the Falling In the sequel, Mann encountered a familiar problem.
“I accidentally wrote way too many f*cks again,” he sighs. “We had to have a conversation: We’re allowed shit, so let’s use it wisely.”
Comments are closed.