“A Jester’s Tale” by Asad Malik and RYOT challenges you to question reality. In Asad’s words, he wanted the experience “to sink into your dreams.” But it did more than sink into my dreams; it made me question my own existence and place my trust in a simulation. “A Jesters Tale” works by creating a personal relationship between the viewer and the simulation forcing you to make an impossible life or death choice. By design, the software platform behind the lens is invisible. You feel like you are interacting with real people and you are not aware of the technology that is driving the experience.

So, how was I convinced to entrust my own life to the 3D image of a boy in “A Jester’s Tale”?

A foundation of trust is built on physical interaction and emotion. Communicated through subtle details of human motion, gesture, and reaction, a virtual being must feel thoughtful, responsive and interactive. You need “presence” for a grounded sense of reality and you need “responsiveness” to create a sense of character.

Volumetric video provides a sense of physical presence. How? A segment of performance is captured from every angle at once and reconstructed into a volumetric clip (sometimes known as a hologram). Once captured, a clip can be viewed from any angle chosen by the viewer. Because this is a photographic process it captures all the nuances of the performer’s gestures and motion. This reality, combined with the ability to view the performance from any angle creates this sense of physical presence.

But physical presence isn’t enough to create trust, you need the performer to initiate actions in the viewer and to respond to the viewer’s choices. It is this pattern of call and response that makes the experience feel alive. HoloSuite, a technology platform built by Arcturus for the post-production and playback of volumetric video, is the core component that enables this.

HoloSuite breathes life into volumetric captures by making them responsive. There are several steps to this process. Effective responsive sequences start in pre-production where you break your script down into a set of actions that consist of primary actions and idle actions.

A primary action is a self-contained section of performance that plays back without interruption. Primary actions typically have a distinct starting & ending points and do not loop. Idle actions are the connective tissue of an experience. They are relatively short sections of performance that loop seamlessly from the beginning to the end and back to the start again.

For example, in “A Jester’s Tale” the little boy sitting on the bed is an idle action and is used when the boy is not talking with you. Idle actions are easy to create in computer animation but require special tools for creating with volumetric video.

Creating a seamless loop starts on the capture stage. The performer starts each take in a specific pose, performs a short action and at the end of the take returns as close as possible to the initial pose. Recorded as a volumetric clip, each take then moves into post-production. In post, each clip is brought into HoloSuite and processed so that start and end of the clip match exactly. After this step, the clip can play endlessly without the viewer being aware that the clip is looping.

The next step is fitting the HoloSuite control system. The control system enables two key features: Dynamic blending & retargeting.

Dynamic blending allows a clip to blend into another clip dependent on the viewer’s action. For example, after the viewer acts in a manner that would elicit a response we transition from the idle action into a primary action. At the end of the primary action, we transition back into one of the idle clips to continue looping until the next reaction. This seamless shift of action from one clip to another creates the sensation that the volumetric capture is responding to you and is what kept me involved in “A Jester’s Tale”. While the reactive nature of the boy and the other characters makes them feel alive, it still isn’t enough to create trust. You need one more thing, you need to feel that the volumetric performers are focused on you.

This is where retargeting comes into play. Using the HoloSuite control system, the head and the neck of the captured clip are set to track the viewer of the experience and their orientation is continuously adjusted so that their gaze is always on the viewer. It is this focus combined with seamless performances providing “responsiveness” and the grounded reality of volumetric capture creating “presence” that makes the performance feel completely trustworthy.

It was a gratifying experience working with Asad Malik, his 1RIC team, RYOT and the team at Metastage in creating this experience. “A Jester’s Tale” stays with you well after it ends and we are proud to say that it is powered by HoloSuite.