Processing & Jobs¶
Overview¶
Most computations in HoloEdit Stages are performed on the Holographic Processing Service (HPS), typically referred to as ‘HPS’, or the ‘Job Server’.
HoloEdit 2020.4 provides two options for your Job Server – the Remote Job Server, a cloud based server that performs fast parallel computations on Arcturus Cloud Servers without tying up your local machine, and a single threaded Local Job Server which runs on the same computer as HoloEdit.
When Stages are processed through the “Execute Intervals” commands, the selected intervals and Stages are broken up into Work Units and those work units are dispatched to the HPS as Jobs.
Jobs in processing can be inspected using the Job Viewer window, available in the “Tools” Menu.
Sending and Receiving Jobs¶
When an interval is Executed inside of HoloEdit, each individual asset required by the job server will be exported and uploaded to the job server. A progress bar will appear in your HoloEdit window to track the upload status.
Once upload is complete, you can continue editing your Composition in HoloEdit, and even save and exit your Composition. As long as you the job isn’t cancelled, as soon as the job server has completed processing a Work Item its results will be imported back into an appropriate HoloEdit Composition. When HoloEdit detects a completed work item, it will pause briefly to Fetch the job results and apply them to the corresponding Interval.
If a Work Item is completed while a Composition is closed, the appropriate files will be fetched as soon as you open the Composition.
Completed Work Items stay on the job server for up to …. If you haven’t fetched your results before they expire, you’ll have to re-submit your job to see results.
Choosing a Job Server¶
HoloEdit defaults to using the Remote Job Server, and the Stages available in HoloEdit are designed with the power of cloud computing in mind.
The Local Job Server saves time on upload and download, and can be sufficient for small jobs and quick processing of larger meshes.
You can change your job server at any time by selecting either “Local Job Server” or “Remote Job Server” from the Tools menu. Any jobs in progress will not transfer to a new job server, and will need to either be re-submitted or fetched from their original job server after completion.
Job Storage¶
Samples produced by completed Jobs are stored on the Job Server and can be retrieved by HoloEdit any time a Composition with a matching job in progress is opened. This process is called “Fetching” jobs, and applies whether the Composition was left open while the job completed, or closed and then reopened to fetch the results at a later date. Remote Job Server results are re-downloaded every fetch.
When a Composition is saved with results that have already been fetched, those results are stored locally to that Composition and will be accessed without repeatedly fetching the results.
Local Job Server results behave the same as Remote Job Server results, but are stored in the JobAPI/Temp directory located in your HoloEdit 2020.4 installation directory and fetched from local storage instead of being downloaded.
Local Job Server¶
The Local Job Server will launch a powershell and npm based server on your local computer while it is in use in HoloEdit. Both of these programs should remain running until you close HoloEdit or change Job Server.
First Time Setup¶
The Local Job Server should install and initialize the required components when it’s activated, but this installation can be run manually by choosing the “Run First Time Setup” option in the Tools Menu.
Remote Job Server¶
The Remote Job Server executes jobs on Cloud Servers provided by Arcturus. When a Job is dispatched to the Remote Job Server, it will be uploaded after export and downloaded before Fetch.
Compute Hours¶
Using the Remote Job Server for Cloud Processing requires a HoloEdit License and uses up Compute Hours associated with your account. Your organization’s remaining Compute Hours are listed in the User Profile window, accessible from the User -> Manage Profile option on the Menu Bar.
The amount of Compute Hours consumed per job can be previewed in the Compute Time column of the Job Viewer window.
Different Stages and files consume different amounts of Compute Time per process, so you should be sure to keep track of the most costly operations in your projects, and try to minimize Compute Time by using the Clean Mesh Stage to reduce polygon count before processing your Clips, if applicable.
If your account has no remaining Compute Hours, you will no longer be able to submit jobs to the Remote Job Server. Jobs in Progress will finish currently processing work units but will not start any new work units. To purchase more compute hours, contact sales@arcturus.studio
Queuing¶
Jobs are currently placed in a first come first serve queue. If all active compute nodes are in use, as a Job finishes, the next Job in the queue will be started.
Job Viewer¶
The Job Viewer window, available in the “Tools” Menu, displays the status of all Jobs under your organization currently running on the HPS, along with their status, owner, and details. The Job Viewer can be used to review or cancel jobs, measure Compute Time, and view or export detailed job output.
Cancelling Jobs¶
To cancel a job, press the “Cancel” button in the Action column of the Job List display. Cancelled jobs will remain in the job view for review, and can be removed by pressing the “Dismiss” button in the Action Column.
Farm Status¶
Farm Status is displayed at the top left of the Job Viewer. This displays the number of jobs running and the number of currently available farms.
The Job Viewer is broken up into 3 panels:
The Job List: The top half of the window, labeled “All Jobs”, is the Job List. This contains each job, with a description and debug information. Jobs can be selected by clicking anywhere in a given job’s row.
Console Output: The Console Output is the bottom left window. With a Work Unit selected, this displays status information for debugging purposes. When a work unit is selected, the “Save Logs” button can be used to store the current logs to disk.
Work Unit List: The Work Unit List is the bottom right portion of the window. It shows the status of each individual work unit within the current selected job. Work Units can be selected with left click to display their output in the Job Console.
The Job list¶
The Job List contains the following columns for each job:
Description: Description of the job containing interval information
Job ID: Unique identifier of the job
Status: See “Job Status” below
Compute Time: The amount of time (in compute hours) spent processing the job. Jobs run in parallel on the Holographic Processing Service and this is the aggregate time on all processors
Elapsed Time: The amount of time (in realtime hours) since the job was executed
Nodes: N/A in HoloEdit 2020.4
Priority: N/A in HoloEdit 2020.4
Server: The name of the Job Server instance in use
Owner: The HoloEdit Instance that created the Job
Action: Cancel or remove the job from the Job list
Work Unit List¶
The Job List contains the following columns for each work unit of the currently selected job:
Status: See “Work Unit Status” below
Frame: The Frame or Frame Range being operated on by this Work Unit
Compute Time: The amount of time (in compute hours) spent processing the Work Unit
Stage Type: The name of the current process being executed by the Work Unit. Varies per Stage
Work Unit ID: Unique identifier of the Work Unit
Job Status¶
In the Job List, each job will have a Status displayed in the Status Column.
Queued: Jobs that are Queued are being prepared for processing. If your organization is running many jobs simultaneously, new jobs may queue until previous jobs have completed In Progress: Jobs in Progress display a %, indicating the percentage of the job’s total work items that have been completed
Failed: Jobs that have Failed will display the “Failed” status. Failed Jobs are usually the result of a server error or malformed data being sent to the job server. Review your input streams and the Job Console Output. If the issue persists, contact HoloSuite Support
Done: Completed jobs are labeled “Done”
Abandoned: When a job’s results are no longer needed by HoloEdit, they will be marked “abandoned”. Abandoned jobs’ results will not be fetched
Work Unit Status¶
In the Work Unit List, each Work Unit will have a Status displayed in the Status Column.
N/A: Status of this Unit Item has not been retrieved from the Job Server yet
Starting: Work Unit is starting processing
Active: Work items in progress are labeled “active”, indicating the work item is currently in progress
Failed: Work Units that have Failed will display the “Failed” status. Failed Jobs are usually the result of a server error or malformed data being sent to the job server. Review your input streams and the Job Console Output. If the issue persists, contact HoloSuite Support
Finished: Completed work units are labeled “Finished”