Making Of
stereoscopic
preproduction
When Daniel Nogueira (Cake Film & Photography Amsterdam), the director of this project, came to us with his idea to create this musicvideo out of pointcloud-data he emphasized that he wanted to keep everything as flexible as possible.
We knew from the beginning that we would not be doing this with a 2D plugin but rather use real 3D data, as it has advantages such as the ability to do impossible camera moves and the flexibility to change the camera angles later on. Daniel’s initial idea was to shoot it with real 3D scanners so our first priority became to develop a new workflow for this visual idea.
We also decided very early on not to use the pointcloud-data provided by the scanner directly, but to only use the 3D-geometry generated by them. This gave us the ability to cast our own points on this mesh, where we could then decide which parts to cast points on and how dense the pointcloud should be. Later on, these pointclouds could then be manipulated in any way we like to with Autodesk Softimage's particle system "ICE".
Production
The real footage for the intro and outro of the music video was shot with Canon's 5D.
As the 3D scanner had a maximum framerate of 15fps, but the video a framerate of 25fps, we had to retime the song for the playback. This resulted in Krause performing the song at 60% of the original speed.
The scanner weren't able to scan face and color simultaneously at a high enough framerate, so we decided to record Krause's performance with 2 additional cameras. Their footage could then later be projected onto the 3D scans.
We also couldn't scan the hair of Krause, as hair is too shiny and to tiny-detailed for the scanners to read. That's why she had to wear a "hat" to cover her hair and serve as a readable surface for the 3D scanner.
Processing and cleaning of the 3D data
The data provided by the 3D scanners had to be processed to synchronize it with the footage of the reference cameras and with the actual song. In Order to do that, we had to interpolate extra 3D-frames so that the framerates of the different sources would match.
Multi-angle previews of the scanned faces were made and imported into the Avid, so the director could select the takes he wanted to use. Here selected takes where arranged and "lip-synched". We then had an "EDL" for the 3D-sequences.
As the scanner couldn't read shiny surfaces such as eyes and teeth and thus create holes in the 3D geometry, we had to fill these holes in order to make the face look more human.
We then tracked the movement of every head so we could remove the "hat" of Krause and replace it with our virtual hair.
Virtual Hair
To make a consistent mesh matching the face-scans, we decided to go for polygon modeled hair instead of using conventional hair-systems.
The animation and simulation was realized using a custom spring-tail rig that gave us the freedom to control the overall animation whilst keeping a dynamic movement.
Animation and creation of the poinclouds
With the final and cleaned 3D scans we started finalizing the camera moves that were done previously in the animatic. By then we knew from where to cast our points onto the geometry and how dense the pointcloud should be depending on the distance to the camera.
To get the color information of the reference cameras onto the points, we matched the position of the cameras virtually and projected their images onto the points.
Rendering
The rendering of the points was done mostly with a custom pointcloud-renderer, that allowed us to render a massive amount of points very quickly. Also to meet our visual expectations, new functions were developed for us for the renderer, such as depth of field, global illumination (GI) and ambient occlusion (AO).
