Soon, you will be able to direct a movie without a crew. You will be able to make changes and iterate on your movies with ease. And most importantly, you will be able to experiment and play within the worlds of your stories.
In 2014, I gave a speech at SIGGRAPH Asia. You can read it here. The majority of the people in the room were VFX professionals. In my speech, I made an appeal for these creative technologists to move away from service-based work, and instead to focus on building creative tools to help creatives realize their visions.
“This is the goal: I want to develop a great idea for a movie. I want to direct that movie on my iPad, or Oculus Rift, or phone. I not only want to direct the movie in that context, but also I want to write it in that context. I want to try something crazy only to realize it doesn’t work, but take what I learned and incorporate it moving forward. I want a tool that allows me to synthesize a movie with no technical or user interface friction. I want to be able to use it at my desk, my bedroom or at a cafe.”
After the speech, Richard Chuang came up to me. Richard is a real pioneer in the world of digital animation and visual effects. He’s one of the founders of PDI, which became DreamWorks Animation (Antz, Shrek, Kung Fu Panda). We had lunch and he told me that George Lucas wanted to build what I was talking about. He said that George was obsessed with building a tool to direct movies on the Playstation.
Subsequently, I’ve learned that since the late 70’s, Francis Ford Coppola has also had a similar dream.
“My interest is more in electronic armature to plan films by constructing them in simulated ways, like a wind tunnel test, where I could make the whole movie before it’s shot, to design the actual shooting of the production more efficiently. My gamble is that the money we spend on it will be saved by the fact that we’ll shoot it that much quicker.” - Francis Ford Coppola
To go even further back, Stanley Kubrick had a desire for a tool to diagram shot setups and automatically storyboard his films.
The problem is that, up until now, the technological building blocks weren’t good enough to build such tools.
We built Storyboarder because we needed a simple tool to help us draw and organize storyboards. Story art and storyboarding are a huge part of our iterative story development process.
When we released it publicly, we were very surprised at the response. Today, there are 173,000 users, and just over half a million downloads. It’s great to see the tool being used by professionals for major motion pictures. And it’s so inspiring to see it being used in schools by young artists who will craft great movies in the future.
I like to think that there’s a kid living somewhere like West Virginia, working at a dead-end job, spending all day dreaming up amazing stories. There are probably millions of these kids, each with a unique and interesting point of view. However, due to an inability to move to Hollywood, and an additional inability to practice transforming their ideas into movies, we might never experience their stories.
In order to visualize these stories, you need a lot of money, or at the very least, the ability to draw. We need a tool that anyone can use, regardless of cost, technical proficiency, or ability to draw.
Unfortunately, I think the barrier to entry for storyboarding is way too high for most.
Even at Wonder Unit, working on EXPLORERS, storyboarding can be limiting. Only a fraction of our team can draw, and even fewer can board well. I can draw, but, many times, I can’t draw fast enough to pitch my visual ideas to the team. Furthermore, making changes is a real chore. Storyboarding is a wonderful tool. But, ironically, and in many cases, it’s a real impediment to iteration.
Now is the time to build a tool that anyone can use.
Early on in development of Storyboarder, I built a feature called “Shot Generator.” The idea was that you could type in a kind of shot you wanted, for example, “Medium Shot” or even more complicated shots like, “Wide worms-eye shot of woman running.” The program would generate some shots for you to use in your storyboard sequence. The problem is that you don’t have much control over the generated shot. You can’t move the camera around. You can’t move the characters. It’s a novel feature – but it’s mostly useless.
What are we building?
We started building an extremely simple interface to layout shots. This is built into Storyboarder, so you can easily rearrange and sequence your shots.
How does it work?
You start with the view from a single camera. You can add characters. You can add objects. You can drag them around, moving them where you want. You can pose the characters. With your mobile phone, you can literally reach out, and pose the characters by touching their arm and rotating it the way you want. You can move the camera around to frame the shot exactly how you want, and you can insert that shot into Storyboarder.
After laying out a few shots, if you want to make a change, it’s as easy as going back into the shot editor; the entire world is saved in each shot.
Then, when you are done with a sequence, you can export the shot’s entire top down lighting diagrams, which makes preproduction and production easier.
Choices
We made specific choices around the design of the tool, primarily focused on simplicity. At first glance, the interface looks like a stripped down 3D editor. Most 3D editors are very complicated, frustrating, and in many cases, infuriating. But, in Shot Generator, when you drag objects around the scene, they won’t be lifted off the ground. They won’t accidentally rotate to weird angles. Hopefully everything behaves as you would expect. We don’t want there to be any set-up work to quickly make a shot.
A major problem with using a complicated tool is that sometimes you get so caught up in fiddling with the tool itself, that you’ve forgotten the thing you wanted to do in the first place. (Sort of like this complicated sentence.) The interface should be completely transparent and dovetail seamlessly with your intention. This first attempt is far from that, but it’s the goal.
Silhouette
Visually, the most important aspect in shot layout is silhouette. Who’s in the shot? What do they look like? What are they doing? Where are they?
We came up with characters that are really simple to add and customize. You can pick from four base characters: adult female, youth female, adult male, and youth male. Then you can set how tall they are. Then you can modify their body shape by dragging three sliders: Ectomorph, Muscular, or Obese. You can save a character as a preset, so you can easily drop characters in without doing any setup.
More information on the technicalities of characters.
Simple Characters
You’ll notice that you can’t customize their clothes. It’s irrelevant to laying out a shot. Besides, you can draw clothes on later in Storyboarder. There are no faces on the characters. 3D faces are uncanny. They look dead, and furthermore, that is another thing you would have to set up and spend hours to get to look right. The choice was made to keep the faces blank so that you can draw facial expressions on them in Storyboarder.
Color
There is no color. Color isn’t needed in laying out a shot. Specifying color would just be another set-up task that wastes time.
Boxes
Boxes are the building blocks of objects. While there will be some built-in 3D objects, and you’ll be able to specify your own, we encourage the use of boxes. Need a table? Put a box there. Need a wall? Put a box there. Need a building? Put a box there. Most things are boxes anyways. But even things that are not, you can just put a box for scale and placement, and draw the object you need over the top.
Lighting
Lighting is very minimal. This tool won’t simulate accurate lighting. No tool does that very well. We are trying to layout shots, so we thought it was important to be able to place physical lights, so you can see if they are in a shot. We, roughly, show the motivation of the light (What direction the lights are coming from). We do not want to make the tool hard to use, or even to force you to set up lights.
Control
Controlling the camera is as simple as clicking and dragging the mouse. You can use the arrow keys or “WASD” to move. Click an object to select it. Drag it to move it. Modify properties on the side to augment it.
We implemented the PlayStation 4 controller for control. As a controller, it’s great. For about $50, (or $30 if you buy it used like I did), you get a controller with 18 buttons, analogue sticks, touch pad with multi-touch, and motion controls. If you have Bluetooth 4.2 or greater, you can use it wirelessly.
Posing
An expressive pose is the most important thing in a character’s silhouette. You can select from a library of preset poses. However, you can manually pose characters by selecting a character, and then selecting an underlying bone you’d like to rotate. Just rotate whichever bones you’d like, and you’re done. You can even save the pose as a preset.
But, how does the rotation work?
When I first played with latest wave of virtual reality, I was so surprised how easy it was to manipulate 3D objects in space with the use of wands. I just reached out, touched something and rotated it. It was a joy. I don’t care much for virtual reality entertainment, (I haven’t seen anything that compelling yet). However, as a creative tool, it’s a blue ocean of opportunity. In VR, you can build a world, and live in the world.
I’ve wanted to build the equivalent of shot generator in VR for a while. In fact, we built an early version of it in the Unreal 4 Engine. To be honest, I got really sick working on it, and I haven’t wanted to be in VR much since.
Besides, no one has VR setups. Not real people. I’m friends with a lot of nerds and all anyone talks about, regarding VR, is how long it’s been since they’ve hooked it up.
Yet, manipulating 3D rotations in 2D space really sucks. Rotating an object, something that humans have an innate feeling for, is really hard in 2D. You have to rotate on one or two axes, rotate the camera around the object, and rotate on the third axis. Keep rotating to see if it’s what you wanted.
So I had an idea of using the accelerometer on a smartphone, which everyone has, to control the rotation of the object.
If you’re on the same WiFi network as your computer, you can use you phone to scan a QR code, which will connect it to the computer. Now, all you have to do is select a bone, and touch the screen of your mobile phone and rotate it in any direction. It feels really nice. In fact, you can make a really complicated pose in seconds.
As a better method, you can use a PlayStation 4 controller. The rotation is high fidelity and it feels much more natural. Just select a bone with the right trigger. Then press and hold the circle button and rotate the controller. Release the button when you’re done rotating. Easy.
In the worst-case scenario, you can use 2D UI to rotate a bone.
Additional Stuff
You can toggle guides over the shot: drawn storyboard, centerlines, thirds, and K marks. You can model your own characters and objects and use them in the system. You can specify a model for the environment to import. For example, you can design a set in a 3D app, and use it as your set in the tool.
You will be able to see the whole scene in augmented reality on your phone. You can walk around the scene, and get a feel for the world. This will also be another way you can pose objects.
When you are done with a sequence, you can export shot’s entire top down lighting diagrams. Shot generator saves a top down view of the whole scene with each shot. This is nice to hand off to a cinematographer to light - or to a producer for pre production.
I need your help
We’ve been building this tool for use on our own projects. But I would like to get feedback about this program’s design from live-action directors. If you know any of the following people, could you send them a link to this page, or tell them to get in touch with me? If you are someone on this list, sup? My name is Charles. Wanna call me at: 917-696-5465?
- David Fincher
- George Lucas
- Francis Ford Coppola
- Steven Spielberg
- Ridley Scott
- Spike Jonze
- JJ Abrams
- Wes Anderson
- George Miller
- Peter Jackson
- James Cameron
- Steven Soderbergh
- Alejandro Inarritu
- Bong Joon Ho
- Robert Rodriguez
- Michel Gondry
- Alfonso Cuaron
- PT Anderson
- Zack Snyder
- Brad Bird
- Chris Columbus
- Jean Pierre Jeanet
- Tim Miller
I’d actually like to speak with anyone who has any feedback!
Wonder Unit will be the first studio to direct a live action movie entirely on the computer. We’re using this tool to layout each shot for our first movie, EXPLORERS. We would love to talk with young directors and storytellers who might be interested in joining us on this journey.