OpenAI unveils powerful, creepy new text-to-video generator that it calls 'a foundation for models that can understand and simulate the real world'

Image for OpenAI unveils powerful, creepy new text-to-video generator that it calls 'a foundation for models that can understand and simulate the real world'
(Image credit: OpenAI)

The generative AI company behind ChatGPT and DALL-E has a new toy: Sora, a text-to-video model that can (sometimes) generate pretty convincing 60-second clips from prompts like "a stylish woman walks down a Tokyo street..." and "a movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet..."

A lot of the AI video generation we've seen so far fails to sustain a consistent reality, redesigning faces and clothing and objects from one frame to the next. Sora, however, "understands not only what the user has asked for in the prompt, but also how those things exist in the physical world," says OpenAI in its announcement post (using the word "understands" loosely).

View post on imgur.com"

The Sora clips are impressive. If I weren't looking closely—say, I was just scrolling past them on social media—I'd probably think many of them were real. The prompt "a Chinese Lunar New Year celebration video with Chinese Dragon" looks at first like typical documentary footage of a parade. But then you realize that the people are oddly proportioned, and seem to be stumbling—it's like the moment in a dream when you suddenly notice that everything is a little bit wrong. Creepy.

"The current model has weaknesses," writes OpenAI. "It may struggle with accurately simulating the physics of a complex scene, and may not understand specific instances of cause and effect. For example, a person might take a bite out of a cookie, but afterward, the cookie may not have a bite mark. The model may also confuse spatial details of a prompt, for example, mixing up left and right, and may struggle with precise descriptions of events that take place over time, like following a specific camera trajectory."

My favorite demonstration of Sora's weaknesses is a video in which a plastic chair begins morphing into a Cronenberg lifeform. Behold:

View post on imgur.com"

Sora is not widely available yet, and OpenAI says it's assessing social risks of the model and working on mitigating them, for instance with "a detection classifier that can tell when a video was generated by Sora."

It's fascinating as a research project, but OpenAI isn't just interested in doing cool computer science. If it can outmaneuver copyright critics and legislators, it's here to make bank. The company says it's currently "granting [Sora] access to a number of visual artists, designers, and filmmakers to gain feedback on how to advance the model to be most helpful for creative professionals."

One commenter on X optimistically wondered if models like Sora will one day allow the public to wrest control of filmmaking away from Hollywood by making movies purely with prompts—but I wonder where the source material for all this generated video will come from if not, you know, filmmakers? Big Hollywood movies may already look pretty homogenous, but auto-reproducing Marvel Cinematic Universe-style CGI and car commercial drone shots isn't exactly bringing creative expression to the masses. (The blog post notably doesn't mention Sora's training material.)

View post on imgur.com"

Despite the often clumsy results of current generative AI models and the legal, ethical quagmire it presents, we're already seeing it used in professional creative media. That includes videogames, both in ways that are directly visible to us, like to generate art and voices and on-the-fly dialogue, and in ways that are less obvious, like generating code snippets or early concept art. A recent survey found that 31% of game development professionals use generative AI in some capacity. Combined with other software, I wonder what this kind of machine learning-driven video simulation could do besides generate slightly-off CG-like clips?

I don't think anyone really knows how generative AI will be used in five or ten years or what the consequences of continued development will be, but it isn't slowing down, so it appears we'll find out. OpenAI and other companies are explicitly working not just toward better image and video and text generators, but toward "artificial general intelligence" or AGI—as in, the science fiction idea of what AI is.

"Sora serves as a foundation for models that can understand and simulate the real world, a capability we believe will be an important milestone for achieving AGI," says OpenAI.

Tyler Wilde
Editor-in-Chief, US

Tyler grew up in Silicon Valley during the '80s and '90s, playing games like Zork and Arkanoid on early PCs. He was later captivated by Myst, SimCity, Civilization, Command & Conquer, all the shooters they call "boomer shooters" now, and PS1 classic Bushido Blade (that's right: he had Bleem!). Tyler joined PC Gamer in 2011, and today he's focused on the site's news coverage. His hobbies include amateur boxing and adding to his 1,200-plus hours in Rocket League.

Read more
The OpenAI logo is being displayed on a smartphone with an AI brain visible in the background, in this photo illustration taken in Brussels, Belgium, on January 2, 2024. (Photo illustration by Jonathan Raa/NurPhoto via Getty Images)
OpenAI is working on a new AI model Sam Altman says is ‘good at creative writing’ but to me it reads like a 15-year-old's journal
Aloy
'Creepy,' 'ghastly,' 'rancid': Viewers react to leaked video of Sony's AI-powered Aloy
An Ai face looks down on a human.
Xbox announces 'a generative AI model for gameplay ideation' called Muse, but don't get too excited: Machines aren't about to make games for you just yet
Microsoft Muse-generated gaming in action
'A massive, massive moment of wow.' Microsoft CEO predicts AI-generated games are a 'CGI moment' for the industry
SAN FRANCISCO, CALIFORNIA - NOVEMBER 06: OpenAI CEO Sam Altman speaks during the OpenAI DevDay event on November 06, 2023 in San Francisco, California. Altman delivered the keynote address at the first-ever Open AI DevDay conference.(Photo by Justin Sullivan/Getty Images)
In a mere decade 'everyone on Earth will be capable of accomplishing more than the most impactful person can today' says OpenAI boss Sam Altman
A robot girl from Judas looks skeptical, her synthetic skin peeling off to reveal metal below.
Bioshock's Big Daddy Ken Levine says that while he doesn't want to 'underestimate' AI, he's 'not overly impressed' by it, either
Latest in AI
Otter AI Meeting Agent
As if your work meetings weren't already fun enough, now Otter has a new all-hearing AI agent that remembers everything anyone has said and can join in the discussion
Image for
'No real human would go four links deep into a maze of AI-generated nonsense': Cloudflare's AI Labyrinth uses decoy pages to trap web-crawling bots and feed them slop 'as a defensive weapon'
CHINA - 2025/02/11: In this photo illustration, a Roblox logo is seen displayed on the screen of a smartphone. (Photo Illustration by Sheldon Cooper/SOPA Images/LightRocket via Getty Images)
'Humans still surpass machines': Roblox has been using a machine learning voice chat moderation system for a year, but in some cases you just can't beat real people
OpenAI logo displayed on a phone screen and ChatGPT website displayed on a laptop screen are seen in this illustration photo taken in Krakow, Poland on December 5, 2022.
ChatGPT faces legal complaint after a user inputted their own name and found it accused them of made-up crimes
Public Eye trailer still - dead-eyed police officer sitting for an interview
I'm creeped out by this trailer for a generative AI game about people using an AI-powered app to solve violent crimes in the year 2028 that somehow isn't a cautionary tale
Closeup of the new Copilot key coming to Windows 11 PC keyboards
Microsoft co-authored paper suggests the regular use of gen-AI can leave users with a 'diminished skill for independent problem-solving' and at least one AI model seems to agree
Latest in News
The heroes are attacked by monsters
Pillars of Eternity is getting turn-based combat to mark its 10th anniversary, and that means PC Gamer editors will soon be arguing about combat mechanics again
Image of Ronaldo from Fatal Fury: City of the Wolves trailer
It doesn't really make sense that soccer star Ronaldo is now a Fatal Fury character, but if you follow the money you can see how it happened
Junah beginning a battle in Metaphor: ReFantazio.
Today's RPG fans are 'very sensitive to feeling like they wasted time' when they die, says Metaphor: ReFantazio battle planner—but Atlus still made combat hard anyway
Image of Cersei Lanniser from Game of Thrones: Kingsroad Steam early access trailer
A new Game of Thrones RPG is coming to Steam today with a cast of 'familiar faces,' which is good because it's really the only way to tell it's a GoT game at all
The new Prime Asset featured in the upcoming update for the Outlast Trials.
The Outlast Trials puts its already paranoid players under surveillance for a time-limited story event
A Viera looking confused in Final Fantasy 14.
Old armor continues to fall victim to Final Fantasy 14's bizarre two-channel dye system, unless you're super into changing the colour of teeny-tiny eyelets: 'Why even bother at this point?'