Unity: OMS Player Reference

Note

This is the reference documentation for the Unity OMS Player Package.

For a step by step guide to setting up an OMS Player in Unity, refer to the OMS Player Configuration example document.

OMS Player Install

The OMS Player Package supports Unity version 2018.4(LTS). To create a new project using the Unity OMS Player, first create a new unity project. You can house multiple projects and their OMS/MP4 files in a single OMS Player scene.

With your new Unity project open, you can select Import Package > Custom Package and import the OMS Player 2021.2.unitypackage located in your HoloSuite 2021.2 install directory, and import all assets when prompted.

Getting Started

An OMS player requires oms and mp4 files located in the Assets/StreamingAssets folder, and a configured object using the OMSPlayer.cs component. To quickly get started with OMS playback, add an instance of the PlaybackOMS prefab, located under Assets/Arcturus/Arcturus.Volumetric/Prefabs. To configure that prefab for playback, you can expand the “Clips” and “Element 0” drawers, and enter the filenames (including extension) of your OMS and MP4 files to the “OMS Filename” and “Video Filename” fields. For example, play the example content by entering ewan.oms and ewan_with_audio.mp4

Read on below to learn about the contents of the Unity OMS Player package. To get started on configuring an OMS Player right away, jump to the OMS Player Configuration example

Build Targets

The Unity OMS Player can be built for Android, IOS, Windows 10, and Mac OS X.

Overview

The Unity OMS Player is a Unity Package used to play back compressed Volumetric Video created in HoloEdit. This Unity package contains scripts and prefabs necessary to play back OMS files in any unity project. Using the included prefabs, a user can quickly assemble a player to play back high quality Volumetric Video from local or streaming assets. With further customization, users can integrate the OMS player with their unity scene logic and other scripts, or create new shaders that include OMS playback.

This documentation page describes the contents of the OMS Player Package, including in depth descriptions of the (OMSPlayer)[unity-player] and (SwingTwistLookIK)[unity-swingtwist] scripts, along with a brief example of configuring an OMS Player. If you need to quickly get up and running, refer to the Unity Player Configuration Example for a step-by-step.

The Unity OMS Player supports all OMS Playback features. These include compressed OMS playback, Retargeting, and Streaming, in addition to driving Unity Events with OMS playback, and basic built in Unlit and Unity Surface shaders.

Overview: Retargeting

One key feature of OMS Playback is Retargeting. Retargeting allows you to dynamically re-pose your volumetric capture like any other Skinned Mesh in Unity without disrupting playback. Using procedural animation code such as IK look-at scripts, you may re-animate any bones embedded in the OMS via the Generate Skeleton stage.

If Skin Weights have been provided, either manually or via the Generate Skin Weights for Head Retargeting stage in HoloEdit, the weighted vertices will be transformed accordingly.

One common usage of Retargeting is “Head Retargeting”, using the head bone to make the volumetric actor look at an animated object or camera in the scene during playback. Clips can be prepared for Head Retargeting in HoloEdit using the Generate Skeleton and Generate Skin Weights For Head Retargeting stages without any manual editing.

See the Retargeting Setup section below for a step by step guide on setting up head retargeting in Unity.

The OMSPlayer Script

The OMS Player Script can be applied to any GameObject and configured to turn that object into an OMS Player.

The OMS Player script contains the following inspector settings:

Default Render Config

The Default Render Config drawer sets the default render options for all clips in the selected OMS Player. It contains the following options:

  • Material: Set the Material for use in the OMS Player. You can choose any OMS Player material, such as SkinnedMeshMaterial or SkinnedSurfaceMaterial.

  • Tint: Specifies a tint color for this material. Default white results in no tint.

  • Shadow Casting Mode: Set Shadow Casting to On, Off, Two Sided, or Shadows only.

Playback Mode

The Playback Mode determines how the underlying OMS Playback functions, mostly switching how different data streams (geometry, texture, audio) are loaded. For most projects Automatic is recommended to choose the best playback mode for your target platform.

Playback mode contains the following options:

  • Video Player: Read textures and audio from a video file using Unity’s built-in VideoPlayer component.

  • Native Video: Read textures and audio from a video file using a platform-specific (“native”) plugin. The OMS Player package includes plugins for several platforms, including OSX, iOS, and Android

  • Automatic: Automatically selects between “Video Player” and “Native Video” depending on the targeted platform per build.

  • Texture: Reads textures from a series of .png files, does not read audio.

  • None: Does not read textures or audio.

  • Stream: Reads geometry and textures from a HoloStream URI.

Hide Video Player

Hides any components spawned at runtime to handle video playback, if you find them distracting.

Clock

Determines the source of the clock used to advance through the frames of an OMS. The default option, Video Time, is recommended for most projects.

Playback mode contains the following options:

  • Video Time: Syncs geometry playback with the current texture frame from the corresponding video file (see “Video Filename” in Clips). The Playback Mode field must be set to VideoPlayer, NativeVideo, or Automatic, and a video file must be provided.

  • Unity Time: Displays the frame corresponding to how much time has elapsed since playback was started, as measured by Unity’s Time.time property.

  • Mecanim Progress: Displays the frame corresponding to the time elapsed in the linked Mecanim animation (see “Mecanim State Name” in Clips). Note: Mecanim playback is EXPERIMENTAL.

  • None: Does not automatically advance the frame. The frame must be controlled manually through the OMSPlayer API (FrameUpdate, StepForward, StepBackward).

Clip Directory

Determines the default directory for clip files. If left empty, the clip directory used will be “Assets/StreamingAssets”. The OMS Filename, Video Filename, and Textures Path values are relative to the configured Clip Directory.

Clips

The Clips drawer contains an array of per Track configuration for each clip included in the OMS Player. Setting the “Size” Value greater than zero generates one or more Clip drawers for configuration.

  • Element: Clip Drawers labeled “Element [Clip Number]” are generated for each Clip specified by the Size value. This drawer’s name will update to reflect the name of the clip when the OMS Filename is configured (see below). Each Clip Drawer contains the following fields:

    • Oms Filename: The path and filename (including extension) to the OMS file used for playback. This path is relative to the Clip Directory field, and by default relative to the “StreamingAssets” folder in your Unity Project. For example, If the Clip Directory is left blank and your OMS file called “file.oms’ is located in the “Assets/StreamingAssets” directory, you would enter “file.oms” in this field.

    • Video Filename: The path and filename (including extension) to the video file used for playback. Like the OMS Filename, this path is relative to the Clip Directory field, and by default relative to the “StreamingAssets” folder in your Unity Project. For example, If the Clip Directory is left blank and your video file called “file.mp4” is located in the “Assets/StreamingAssets” directory, you would enter “file.mp4” in this field.

    • Textures Path: Used only with “Textures” playback mode. Path to a directory in Resources, relative to the Resources directory, containing the textures to use. For example, if your textures are located in “Assets/Resources/myTextures”, this should be set to “myTextures”. Textures are associated with frames by sorting by name in a platform-specific manner. For the most predictable results, textures should be named numerically and padded with leading zeroes (e.g. “0000.png”, “0001.png”, “0002.png”)

    • Mecanim State Name: If specified, the OMSPlayer should automatically switch to this clip when the matching AnimationController changes to the state with the same name. Note: Mecanim playback is EXPERIMENTAL.

    • Track Configs: This provides configuration for MultiTrack OMS playback. MultiTrack OMS files can contain multiple actors, each on a separate “Track” that can be configured individually. Setting the “Size” Value greater than zero generates one or more Track Drawers for configuration labeled “Element [Track Number]”. Track Drawers contain the following fields:

      • Material Override: Allows you to specify any OMS player material for this Track only, separate of the overall OMS Player material specified in the Default Render Config.

      • Tint: Allows you to specify a per-Track Tint separate from the Default Render Config Tint.

      • Retarget: Enable or disable Retargeting for this Track.

      • Skeleton Root: Specify the Skeleton Root for use with Retargeting. If Retarget is disabled this setting has no effect.

      • Skeletal Base Movement: Specify the base animation file for use with Retargeting. If Retarget is disabled this setting has no effect.

    • Experimental: Experimental clip level settings

      • Enable Fast Loading: Speeds up loading of OMS files by skipping a step where the maximum vertex and index count are determined from the contents. If enabled you must manually configure Max Vertex Count and Max Index count to be at least as high as the maximum value in any mesh in the clip. If the value is too low, the OMS will fail to load and the application may crash.

      • Max Vertex Count: Size of pre-allocated vertex table.

      • Max Index Count: Size of pre-allocated index table. Each triangle needs 3 indices.

Loop

Toggles looping playback on all clips and Tracks in this OMS Player.

Ping Pong

Toggles Ping Pong (backwards and forwards loop) playback on all clips and Tracks in this OMS Player.

Draw Editor Label

Toggles display of an OMS Player label in the Unity Scene View. The label appears at the position of the OMSPlayer object, and contains two values. The “F” value shows which overall frame is currently being displayed. The “S” value shows which segment the current frame belongs to.

Timeline and Sync

Fields related to syncing unity animations with OMS Playback. Contains the following fields:

  • Drive Timeline: If enabled, the specified Timeline will have its time set to the OMSPlayer’s current time every frame while the OMSPlayer is playing.

  • Timeline: The timeline selected for Drive Timeline.

  • Video Carries Audio: Unimplemented in HoloEdit 2021.2.

  • Video OMS Offset Start: Subtracted from the video frame when determining what geometry frame to use. Typically Used to allow a video file to contain an audio-only prelude before texture/geometry playback begins. Only functions when PlaybackMode is set to NativeVideo.

UI Events

Emitted by OMSPlayer as various internal events happen. Create Listeners to update custom components.

  • OnClipLoaded: The clip with the given index has been loaded.

  • OnPlayStateChanged: Player state has changed to the given state.

  • OnPlayTimeChanged: The current playback time has changed to the given time (seconds).

  • OnBuffertimeChanged: The amount of buffered-ahead content has changed to the given time (seconds). Streaming mode only.

  • OnPlayError: Playback has failed with the given error message.

Other Events

  • OnIKNeeded: The OMSPlayer fires this event when it is time for IK solvers to modify the pose of the retargeting skeleton.

Animation Sync

Unity Animations in your project can be synchronized with OMS playback using a shared Timeline. To drive a Timeline with an OMS player, configure the Timeline and Sync settings to drive a timeline containing your animations.

OMS Player Materials

The OMS Player Package contains two Materials for use with the OMS Player. These materials should be used with the Default Render Config and Material Override settings on an OMS player.

SkinnedSurfaceMaterial

The SkinnedSurface material is a surface shader that exposes the Albedo texture (driven by the OMS player), smoothness, and metallic values, in addition to a color tint (driven by the OMS player per-track).

SkinnedMeshMaterial

The SkinnedSurface material is an unlit shader that exposes the Albedo texture (driven by the OMS player) in addition to a color tint (driven by the OMS player per-track).

The SwingTwistLookIK Script

The SwingTwistLookIK script can be used to orient a transform to track another at runtime. Typically this script is used for the Head Retargeting workflow, to enable a volumetric actor to track the scene camera with their eyes.

The SwingTwistLookIk script contains the following parameters:

  • Weight: Control the strength of the swing twist motion. At 1.0, the swing twist motion will be applied completely. At lower values, it will be combined with existing animation through spherical linear interpolation according to the weight value.

  • Bones: Expand the ‘Bones’ drawer to configure the Bones array:

    • Size: Specify the number of bones to control

    • Element: Element fields will be displayed according to the number provided in size. Each Element supplies a transform that will be effected by the script.

  • Effector: Provide a transform to use as the source for the look-at calculation. This can be the same as your target bone, or any other object. Typically this is a child of your target bone, which can be slightly rotated or translated to provide an offset for eye position and angle.

  • ForwardVector: Modify the ‘forward’ vector if necessary depending on the orientation of your bones.

  • Target: Specify the target transform. This is the object the selected bones will look at.

  • TurnSpeedActive: Specify the turn speed for tracking an object in degrees/sec.

  • TurnSpeedPassive: Specify the turn speed used when returning to idle in degrees/sec.

  • TurnAccel: Rate at which the turn speed increases when locking on to an object.

  • TwistAxis: Specify the twist (left/right turn, for head retargeting) axis in your bone’s local space.

  • SwingAxis: Specify the swing (up down tilt, for head retargeting) axis in your bone’s local space.

    • TwistMax: Max Twist rotation in degrees. Will not rotate further.

  • SwingMax: Max swing rotation in degrees. Will not rotate further.

  • SwingMaxBias: Bias max swing in one direction or another, in degrees. Negative values bias left, positive values bias right.

  • SwingPlusTiltMax: Set a maximum degrees for tilt (ear to shoulder tilt, for head retargeting) when turning the head. Too low of values may produce strange rotations when used on the head.

  • OutOfRangeTimeout: Set a timer, in seconds, after which the target will return to its idle position when the target is outside of swing/twist range.

To apply the SwingTwistLookIk to your object, you must call the DoUpdate() function. To configure an OMS Player to automatically call SwingTwistLookIK.DoUpdate() every frame as needed, configure the On Ik Needed () event on your OMS Player. Assign the object containing your swing twist IK script, and select the “SwingTwistLookIK -> DoUpdate()”.

Appendix: Unity Package Contents

The OMS Player Package will add several new folders to your project, including a StreamingAssets folder with sample data for use with the OMS Player, and an Arcturus.Volumetric folder containing the volumetric plugin and associated content and scripts. The following content is included:

Arcturus.Compression

Required scripts related to OMS compression and playback.

Arcturus.Dash

Required scripts related to streaming compression and playback.

Arcturus.Volumetric

  • Examples: Data required for the bundled Example Scenes.

    • Animations: Animations required for example scenes.

    • Prefabs: Prefabs used in example scenes.

    • Scenes: Example scenes for OMS setup and playback.

    • Streaming: Assets and scripts used in the Streaming examples.

    • UI: Assets and scripts used for UI in the example scenes

  • Materials: Materials used in examples or in OMS Playback.

  • NativeVideo: Required scripts related to OMS compression and playback.

  • OMSPlugin: Required scripts related to OMS compression and playback

  • Prefabs: OMS Player Prefabs for use in user created scenes. See example scenes for examples of prefab usage.

  • Scripts: Various scripts related to OMS Playback and OMS Player configuration.

  • Shaders: Shaders used for OMS Playback.

  • Textures: Data textures used for some OMS Playback features.

  • Timeline: Scripts related to scheduling and playback.