HoloSuite Player Unity Reference¶

Note

This is the reference documentation for the HoloSuite Player Unity Package.

For a step by step guide to setting up a HoloSuite Player in Unity, refer to the HoloSuite Player Unity Configuration Example document.

Overview¶

HoloSuite Player is a Unity package used to play compressed Volumetric Video clips created using HoloEdit. This package contains the scripts and supporting assets necessary to play embedded or streaming OMS clips in your Unity project. With further customization users can integrate the HoloSuite Player with their unity scene logic and other scripts, configure dynamic content, create new VFX shaders and more.

This page describes the contents of the HoloSuite Player Package, including in depth descriptions of the HoloSuite Player and SwingTwistLookIK scripts. If you need to quickly get up and running, refer to the HoloSuite Player Unity Configuration Example for a step-by-step guide.

Installing the HoloSuite Player Unity Package¶

The HoloSuite Player Package supports Unity version 2019.4(LTS). To create a new project using the HoloSuite Player Unity package, first create a new unity project. If you will be working with Unity’s Universal Render Pipeline or HDR Pipeline, create a new project using the appropriate template.

After creating your new project, open the Package Manager window. Click the + button and select Add package from tarball. Navigate to the HoloSuite/Tools folder and select the HoloSuite Player Unity Plugin.tgz file, then import the package.

HoloSuite Player Unity Sample Content¶

After the HoloSuite Player Unity package has been added to a project you can select it in the Package Manager to find a number of sample asset packs:

None of these packs are necessary for using the HoloSuite Player, but each does provide helpful starting places and example UI. To add a sample to your project simply click the Import To Project button beside it. Here’s what you can expect from these sample packs:

HoloSuite Player: Includes two example scenes that will demonstrate standard playback and playback with dynamic retargeting.
HoloSuite Player HDRP: Includes a demo scene that make use of Unity’s High Definition Render Pipeline and shader graph.
- This sample will require you to have also imported the HDRP package and configured your project accordingly. Please see Unity’s HDR Pipeline documentation for more information.
HoloSuite Player URP: Includes a demo scene that make use of Unity’s Universal Render Pipeline and shader graph.
- This sample will require you to have also imported the Universal RP package and configured your project accordingly. Please see Unity’s documentation for more information.
HoloSuite Player Streaming: Includes two example scenes that will demonstrate streaming playback. The Minimal scene has no playback controls while the standard streaming scene includes UI and a clip library to change between a number of clips on the fly.

Getting Started¶

Using the HoloSuite Player requires at least one OMS and corresponding MP4 file are imported into your project, and a game object is configured using the Holo Suite Player script. To quickly get started with OMS playback, add the HoloSuite Player sample to your project and modify the Ewan scene to use your OMS and MP4 files. Below you’ll find additional details about the contents of the package and general use of the HoloSuite Player script.

Note

HoloSuite Player Unity offers playback using StreamingAssets and native asset paths, but mp4 files must be exported with frame encoding to be used as native assets. A telltale sign that an mp4 requires frame encoding is that the texture will update on the mesh, but the mesh does not animate. Refer to the mp4 export documentation for more information.

Build Targets¶

HoloSuite Player can be built for Android, IOS, Windows 10/11, and Mac OS X.

Upgrading Existing OMS Player Projects¶

Firstly, it is not necessary to update an existing project to the new package. While the new package offers numerous advantages, there is not a seamless migration process to quickly update your project: any custom scripts will need to be updated to reference new package assets, and all OMS players will need to be re-configured with the HoloSuite Player script.

If you want to upgrade a project, first make sure there are no assets you rely upon saved anywhere in the Assets/Arcturus/ subfolders. If there are any assets there, you’ll want to move them out into another folder. Next you can import the HoloSuite Player package per the instructions above. Once the package has been imported you’ll need to delete the Arcturus folder within Assets. It’s likely Unity won’t like doing this while it’s open, so we recommend closing the program and deleting the folder via File Explorer instead. Once you have deleted the folder, reopen your project. For each saved scene you’ll need to add a HoloSuite Player component to all former OMS players, and then fill in its fields to match the OMS Player script. Finally, delete or disable the OMS Player script and Playback Speed components on the game object.

If you’re also planning on using one of Unity’s advances render pipelines (URP or HDRP) you’ll need to refer to Unity’s documentation on updating an existing project.

The Holo Suite Player Script¶

The Holo Suite Player Script can be applied to any GameObject and configured support OMS playback.

../_images/holosuite-player-inspector.PNG

The OMS Player script contains the following inspector settings:

Source Type¶

The Source Type dropdown enables you to indicate where the OMS and MP4 assets are within your Unity project. Assets should be used when your clips have been imported as native Unity assets, and can be populated using click-and-drag or the target button. Streaming Assets will be used when your clips have been imported into the StreamingAssets sub-folder.

Warning

As a reminder, if you are using MP4 files that have not been exported using frame encoding, you must use the Streaming Assets import type. Failure to do so will result in clips that do not play correctly. Please re-export your MP4 file(s) with frame encoding enabled if you want to import your files as native assets.

No matter the chosen source type, you will need to provide your clip information for the following fields:

OMS Source/Filename: The OMS asset to be played. When using Streaming Assets, include the relative file path and the .oms extension (Eg: MySubfolder/MyClip.oms).
Video Clip/Filename: The MP4 asset to be used. As above, when using Streaming Assets you must include the relative file path and the .mp4 extension.
Material: Specify the material to be used by the renderer.
- Shader Graph Material: If the specified material uses shader graph, this checkbox must be enabled.

Playback Settings¶

These settings can be used to modify the playback behaviour of your clip.

Play On Awake: Toggles whether or not the clip plays automatically when the scene is run.
Loop: Toggles whether or not the clip loops when it reaches the end of playback. When looping is disabled, the clip will hold on the final frame once it has reached the end.

Playback Events¶

These events are emitted by HoloSuite Player as various internal events happen. Create Listeners to update custom components.

On Load: The clip with the given index has been loaded.
On State Changed: Player state has changed to the given state.
On Playtime Change: The current playback time has changed to the given time (seconds).
On Buffertime Change: The amount of buffered-ahead content has changed to the given time (seconds). Streaming mode only.
On Error: Playback has failed with the given error message.

Retarget¶

Enable this checkbox for an OMS which is intended to use dynamic retargeting. See the Retargeting Overview for more information on preparing a clip for retargeting.

When Retarget is enabled, additional configuration options become available:

Skeleton Root: Select the skeleton generated using the clip’s OMS file after adding it to the scene hierarchy.
Skeleton Base Movement: This field should typically be left empty and will be evaluated at run-time. However if you have a modified animation clip that uses the same base skeleton pose as your clip, it can be set here to adjust the animation of the clip.

Retarget Events¶

On IK Update: HoloSuite Player fires this event when it is time for IK solvers to modify the pose of the retargeting skeleton.

HoloSuite Player Materials and Shaders¶

For the HoloSuite Player to correctly play the animated mesh content of an OMS file it must use one of the Arcturus custom shaders.

Unity Built-in Render Pipeline Materials¶

Generate a new HoloSuite Player material by right-clicking in the project window and selecting Create > Arcturus > HoloSuite Lit/Unlit Material.

../_images/holosuite-player-materials.PNG

HoloSuite Lit Material¶

The HoloSuite Lit Material uses a SkinnedSurface surface shader that exposes the Albedo texture (driven by the HoloSuite Player), smoothness, and metallic values, in addition to a color tint (driven by the HoloSuite Player per-track).

HoloSuite Unlit Material¶

The HoloSuite Unlit Material is an unlit SkinnedMesh shader that exposes the Albedo texture (driven by the HoloSuite player) in addition to a color tint (driven by the OMS player per-track).

Unity URP Materials¶

The four HoloSuite Player shaders for the Unity URP are: HoloSuitePlayerURPLit: The HoloSuitePlayerURPLit: Exposes the Workflow Mode field which allows you to set the specular or metallic value for your material allowing for more complex light interactions with your mesh.

HoloSuitePlayerURPSimpleLit: SimpleLit Material uses a SkinnedSurface surface shader that exposes the Albedo texture and color tint. The Base Map textures will be provided by the HoloSuite Player. This Material will provide diffuse lighting for your mesh.

HoloSuitePlayerURPUnLit: An unlit SkinnedMesh shader that exposes the Albedo texture (driven by the HoloSuite player) in addition to a color tint.

HoloSuitePlayerURPShaderGraph: A prebuilt ShaderGraph material for playback within the Universal Render Pipeline. HoloSuite Player URP ShaderGraph playback is powered by the ArcturusOMS_URP node. Instructions for ArcturusOMS_URP setup can be found within the node in the ShaderGraph.

Unity HDRP Materials¶

All HDRP HoloSuite Player Shaders are ShaderGraph Materials. The HDRP HoloSuite Player ShaderGraph materials are driven by the ArcturusOMS node.

HoloSuitePlayerHDRPLit: Is a standard lit HoloSuite player material, changes to the default of this material type can be done using the ShaderGraph.

HoloSuitePlayerHDRPUnLit: Is an Unlit Master material, changes to the default of this material type can be done using the ShaderGraph.

Retargeting Overview¶

One key feature of OMS playback is Retargeting. Retargeting allows you to dynamically re-pose your volumetric capture like any other Skinned Mesh in Unity without disrupting playback. Using procedural animation code such as IK look-at scripts, you may re-animate any bones embedded in the OMS via the Generate Skeleton stage.

If Skin Weights have been provided, either manually or via the Generate Skin Weights for Head Retargeting stage in HoloEdit, the weighted vertices will be transformed accordingly.

One common usage of Retargeting is “Head Retargeting,” using the head bone to make the volumetric actor look at an animated object or camera in the scene during playback. Clips can be prepared for Head Retargeting in HoloEdit using the Generate Skeleton and Generate Skin Weights For Head Retargeting stages without any manual editing.

See the Retargeting Setup guide for a step by step walkthrough for setting up head retargeting in Unity.

Generating OMS Skeleton Assets¶

You can quickly generate the skeleton asset required for retargeting by right-clicking on an OMS asset and using the Create Retarget Skeleton function.

../_images/unity-create-retarget-skeleton.PNG

When selected, an <OMS filename> Skeleton prefab will be created adjacent to the target OMS file. A console error will be thrown if you try using the function on a non-OMS asset, or if the targeted OMS file does not contain retargeting data.

Note

.oms files will not be recognized as OMS-type assets while they are imported into the StreamingAssets folder. You can move a .oms file from StreamingAssets into the main asset tree to use this operation and generate a skeleton prefab, then move it back into the StreamingAssets folder afterwards.

The SwingTwistLookIK Script¶

The SwingTwistLookIK script is provided with the HoloSuite Player package and can be used to orient a transform to track another at runtime. Typically this script is used for the Head Retargeting workflow, to enable a volumetric actor to track the scene camera with their eyes.

../_images/swing-twist-look-ik-script.PNG

The SwingTwistLookIk script contains the following parameters:

Weight: Control the strength of the swing twist motion. At 1.0, the swing twist motion will be applied completely. At lower values, it will be combined with existing animation through spherical linear interpolation according to the weight value.
Bones: Expand the ‘Bones’ drawer to configure the Bones array:
- Size: Specify the number of bones to control
- Element: Element fields will be displayed according to the number provided in size. Each Element supplies a transform that will be effected by the script.
Effector: Provide a transform to use as the source for the look-at calculation. This can be the same as your target bone, or any other object. Typically this is a child of your target bone, which can be slightly rotated or translated to provide an offset for eye position and angle.
ForwardVector: Modify the ‘forward’ vector if necessary depending on the orientation of your bones.
Target: Specify the target transform. This is the object the selected bones will look at.
TurnSpeedActive: Specify the turn speed for tracking an object in degrees/sec.
TurnSpeedPassive: Specify the turn speed used when returning to idle in degrees/sec.
TurnAccel: Rate at which the turn speed increases when locking on to an object.
TwistAxis: Specify the twist (left/right turn, for head retargeting) axis in your bone’s local space.
SwingAxis: Specify the swing (up down tilt, for head retargeting) axis in your bone’s local space.
- TwistMax: Max Twist rotation in degrees. Will not rotate further.
SwingMax: Max swing rotation in degrees. Will not rotate further.
SwingMaxBias: Bias max swing in one direction or another, in degrees. Negative values bias left, positive values bias right.
SwingPlusTiltMax: Set a maximum degrees for tilt (ear to shoulder tilt, for head retargeting) when turning the head. Too low of values may produce strange rotations when used on the head.
OutOfRangeTimeout: Set a timer, in seconds, after which the target will return to its idle position when the target is outside of swing/twist range.

To apply the SwingTwistLookIk to your object, you must call the DoUpdate() function. To configure a HoloSuite Player to automatically call SwingTwistLookIK.DoUpdate() every frame as needed, configure the On IK Update () event on your HoloSuite Player. Assign the object containing your swing twist IK script, and select the “SwingTwistLookIK -> DoUpdate()”.

Appendix: Unity Package Contents¶

Importing the HoloSuite Player Package will add its contents to the Unity Packages folder tree. Contents of the Packages folder are both read and write enabled. This means that all assets are visible to you, even if you have not imported the sample packages they belong to, and you are free to edit any of the contents. When you import a sample using the Unity Package Manager it will create a local copy of the assets within your project folder.