Volumetric video creates a massive new narrative paradigm which we are just beginning to scratch the surface. Storytellers now have tools that involve their viewers and allow them to be active participants in those stories. These tools elevate an experience from offering intellectual stimulus to creating a deep emotional impact.

How do we help achieve this? We’ve developed a suite of post-production tools that allow you to enhance volumetrically captured human performances, making them responsive and interactive while viewing.

One of the basic necessities in creating responsive characters is a seamless loop. The loop is critical because it allows for a character to pause and wait for input from the viewer in an unobtrusive and natural looking way.

Much like any production, the first step is planning. While breaking down a standard movie or TV script reveals what is needed to shoot each scene, volumetric video requires more detail. A scripted volumetric performance has to break down further into a series of character actions. These actions, referred to as “clips” in computer animation, represent a small atomic unit of interaction.

Let’s look at a moment from Asad J. Malik’s A Jester’s Tale to see how it’s done:

• Boy delivers a line of dialog
• Boy waits for viewer choice of 2 possible responses
• Boy delivers response to 1st possibility
  or
• Boy delivers response to 2nd possibility

The first action is a “single shot” and is generally used for lines of dialogue. It plays once and then moves on to the next action. In this example, after the first line of dialogue completes, the experience seamlessly transitions into the next action: “Boy waits for viewer choice.”

For the second action to seem natural and unforced, “Boy waits for viewer choice” will need to loop seamlessly until the viewer responds. This “idle” is created by recording a small clip in which the performer starts and stops their activity in as close to the same pose as possible. Our post-production tools are then used to ensure that there is a match between the start and end pose of that clip, thereby creating a seamless loop.

The loop can be used for sequence blending and even more complex actions like branching narratives that can be seen from any angle. How the “Boy delivers response to two possibilities” or “dynamic sequence blending” works will be discussed in our next post.

Production details:
A Jester’s Tale was produced by RYOT Studios and 1RIC. Volumetric capture was done by Metastage. Volumetric post production was done by Arcturus. A Jester’s Tale premiered at Sundance New Frontiers in 2019.