Mesh Capture Tool¶
Overview¶
The Mesh Capture Tool is a Unity extension that serves as an entry point to data compression for some external formats. It includes a Unity scene to capture mesh and texture data so it may be properly ingested by HoloEdit, as well as tools for subtracting geometry from portions of a clip.
The Mesh Capture Tool is currently capable of capturing .4DS files and mesh and texture sequences using OBJ or PLY mesh formats and PNG or JPEG texture formats.
Version and Layout¶
The Mesh Capture Tool supports Unity version 2018.4(LTS). Ensure your Unity layout has both the Scene and Game windows visible when working with the Mesh Capture tool.
Data Import and Set Up¶
The Mesh Capture Tool can be opened by adding a new project in Unity Hub and selecting the Mesh Capture Tool folder from your HoloSuite 2020.4 install directory.
Source data, like .4DS files, intended for capture must be placed inside of the StreamingAssets folder in the Unity Project.
The Scenes subfolder (nested inside Arcturus.Volumetric) includes ready-made scenes to extract mesh and texture information that can be used with HoloEdit. Select either the Capture4DS or CaptureFromMeshes scene.
The above screenshot displays an open Capture4DS scene with the Mesh4DS object configured to display a .4Ds file in Unity’s Scene and Game windows.
Capture Scenes and GameObjects¶
Depending on your input data, you should open either the Capture4DS scene (if your data is in .4DS format) or the CaptureFromMeshes scene (if your data is in mesh + texture sequence format.)
Capture4DS Scene¶
The Capture4DS scene contains the following GameObjects: a Sequence4DS object with a Mesh 4DS child object, and a Capture 4DS object. Each has specific components to configure in the Unity Inspector to properly capture mesh and texture data from a clip.
The Mesh 4DS game object’s PlugIn 4DS component has a Sequence Name field where you can drag in source data from the StreamingAssets folder.
The Preview Frame slider allows you to scrub through the clip to be captured and record the frame length–the frame length is added to GameObject configurations multiple times during set up.
You will want to ensure that Play on Start is enabled in the Plugin 4DS (Script) component.
The Sequence 4DS GameObject holds Playable Director and Animator components. Ensure that Play on Awake is enabled in the Playable Director component, and that the Playable field is set to Capture4DSTimeline (TimelineAsset).
Opening the Unity Timeline and selecting the Sequence 4DS object exposes an existing FDV Track. Clicking on the shaded FDV Track in the Timeline window will allow you to configure important clip parameters, including start and frame information, assign the correct clip in the Sequence Clip drop down menu, and view the total frame count.
Additionally, the Timeline Settings dropdown allows you to specify a frame rate for your clip. Currently only 30 frames/second is supported.
Note: If you do not see an existing FDV Track in the Timeline, ensure the FDV Plugin and Capture Tool boxes are populated properly with the Mesh4DS and Capture4DS GameObjects.
The Capture4DS game object should be the last item you update when working in a Capture4DS scene.
Selecting the object in the Hierarchy will expose a Capture Tool 4DS (Script) component in the Inspector panel.
Start and end frame information is added to the Timeline Start Frame and Timeline End Frame fields, respectively. You can either capture your entire clip by specifying the corresponding beginning and end frame values, or set a custom start and end point to capture a smaller portion.
Mesh Capture Settings provides options to include mesh and UV information in the capture process.
Typically, you will need both of these checked to ensure you have a complete capture with data that is ready for editing in HoloEdit.
Selecting the correct Texture Capture Mode depends on your capture requirements. The three options available in the Texture Capture Mode drop down are:
Disabled: Do not capture textures
Capture Main Texture: This should be the default choice for capturing UV’d textures for each mesh
Multi View Environment: Generates MVE capture data for the selected clip. Select if you are intending to use a Texture Reconstruction Stage inside HoloEdit. Here you can configure Points Per Axis for how many cameras are used in constructing the camera array (typically, a value of 3 will suffice), Resolution, and Export Alpha Channel and Mirror Horizontal which should both be enabled.
Specify a starting frame number for your output in the Output Start Frame field. To start at the beginning of your clip, enter the number ‘0’ as a value.
Click ‘Play’ in Unity to initiate the capture process. You will see your sequence start. If the Timeline doesn’t advance, ensure that your Unity layout has both the Scene and Game windows visible.
CaptureFromMeshes Scene¶
Before starting the capture process in a CaptureFromMeshes scene, it’s recommended that you create a new folder in StreamingAssets and copy all source data (meshes and textures) into it. Name the new folder appropriately to easily identify it later.
The CaptureFromMeshes scene contains the following GameObjects: a CaptureFromMeshes object with a MeshPlayer and CaptureMesh child objects. Each has specific fields to configure to properly capture mesh and texture data from an asset.
Selecting the CaptureMesh GameObject will expose the Capture Tool Meshes component in the Inspector window.
Start and end frame information is added to the Timeline Start Frame and Timeline End Frame fields, respectively. You can either capture your entire clip by specifying the corresponding beginning and end frame values, or set a custom start and end point to capture a smaller portion.
The Mesh Settings drawer includes fields that must be filled out to specify your source data for the capture. Ensure that In Streaming Assets is enabled, your source data sub folder name is entered correctly to the Input Folder field, and that the Material is set to Unlit.
Inside the Mesh Capture Settings, Capture Meshes should be disabled unless:
The meshes need to be re-scaled,
Volumetric subtraction is going to be performed on the meshes
Enable Write Mesh UVs if you are intending to capture meshes with subtraction. Follow Capture Object should be enabled if you are intending on using an MVE texture capture and wish to center the capture.
Selecting the correct Texture Capture Mode depends on your capture requirements. The three options available in the Texture Capture Mode drop down are:
Disabled: Do not capture textures
Capture Main Texture: This should be the default choice for capturing UV’d textures for each mesh
Multi View Environment: Generates MVE capture data for the selected clip. Select if you are intending to use a Texture Reconstruction Stage inside HoloEdit. Here you can configure Points Per Axis for how many cameras are used in constructing the camera array (typically, a value of 3 will suffice), Resolution, and Export Alpha Channel and Mirror Horizontal which should both be enabled.
The CaptureFromMeshes GameObject holds Playable Director and Animator components. To edit the included animation, open the Unity Timeline with the CaptureFromMeshes object selected.
An existing OBJ Track will be present. By clicking on the shaded OBJClip bar, you will see the Inspector window update with Clip Timing and Mesh Clip components to configure. Here, you will need to:
Set the Duration (frames) to the total number of OBJs - 1 (i.e. 199 if 200 OBJs),
Ensure the OBJ Player and Capture Tool boxes are populated with the OBJPlayer and CaptureMesh game objects, respectively,
Ensure the “In Streaming Assets” option is enabled,
Select the appropriate Sequence Clip from the drop-down,
Set the Start Frame and Frame Count to match the values in Clip Timing,
Set the Frame Rate to 30
With the MeshPlayer selected, perform the following:
Ensure that scale on the X axis is negative,
Verify that the Mesh Player (Script)’s Capture Tool field is set to CaptureMesh (CaptureToolMesh),
Set the Frame Rate to 30
You should be able to confirm that your clip is visible in the Scene Window. If it isn’t visible, hitting ‘F’ will frame the object–this may reveal the meshes are much larger than expected. If this is the case, change the scale of the MeshPlayer until it fits in the CaptureMesh box. Values between 0.01 and 0.001 are a good starting point for typical .OBJ files. Note that the CaptureMesh box will only be displayed if the Capture Mode is set to Multi View Environment.
Click ‘Play’ in Unity to initiate the capture process. You will see your sequence start. If the Timeline doesn’t advance, ensure that your Unity layout has both the Scene and Game windows visible.
Output¶
Once the capture process is initiated, the Mesh Capture Tool will export your captured clip data. Inside the Mesh Capture Tool directory, the Export folder will begin to be populated with the following:
A new output folder for the capture scene you are using,
A subfolder, timestamped to when you initiated the capture process for easy tracking,
Mesh and/or texture data folders from the designated capture
Subtraction Workflow¶
You can use the Mesh Capture Tool to add subtractive volumes to remove unwanted geometry from a volumetric capture. Subtractive volume will remove all vertices that are completely within their volume from the capture. The best use cases for using subtractive volumes are removing small props or consistent noises from a clip.
You should create a duplicate scene before beginning any subtraction work. To do this, it is recommended to:
Create a new folder in Assets for your current project or clip,
From either your Capture4DS or CaptureFromMeshes scene, use Save As to create a scene file in your new folder. Next, from the Project Window, right click in your folder and create a new Timeline object. Name this something appropriate for your clip.
In your new scene, select either the Sequence4DS or CaptureFromMeshes game object and update your Playable Director’s “Playable” field by opening its drop down menu and selecting your new Timeline object. You’ll need to create either a new FDV Track (for 4DS) or a new OBJ Track (for OBJ) depending on the capture scene you are using, as well as a new FDVClip or OBJClip within the Timeline.
Adding Subtractive Volumes¶
To add a Subtractive volume, Create a new GameObject and add a Subtractive Volume component to it. Make sure that the Capture Tool and Mesh Object fields are populated appropriately for your scene (e.g. Capture4DS (CaptureTool4DS) and Mesh4DS when in a Capture4DS scene).
Subtractive Volumes can be animated, using Unity’s keyframing tools. You can create a Track in the same Timeline that will drive your playback, and these volumes can be previewed using the Preview Mode drop-down in the component. Preview Mode items are described as:
Single: Will preview only the current volume’s impact
Hierarchy: Will display the impact of the current volume and all its children
All: Will display the impact of all non-chroma subtractive volumes
Once subtractive volumes are configured and animated in the Timeline, the normal mesh capture process can be initiated and the volumetric subtraction process will be included in the output data.
Adding Custom Cameras¶
When using a MVE workflow, custom cameras allow for unique, manually positioned views that you can add to cover difficult to capture areas in a clip, or simply to attempt to provide a high resolution image of an important area. All custom cameras will be captured for every frame just like those of the MVE, and will appear in the “views” list for each frame of the Textures folder.
To add a custom camera to an initial capture:
Create a new Camera in the scene Hierarchy and place it as desired (the camera can be animated if needed)
Select the CaptureMesh or Capture4DS GameObject and:
Expand the Custom Capture Cameras section and increase the Size by 1
Drag your new camera into the new Element box