The Future of...

Immersive Training Environments for Virtual Reality using Google Seurat – Part 2

Matt Coppinger

September 19, 2019

At VMware we’re working on technology to support Spatial Computing in the enterprise. If you’re not familiar with Spatial Computing, please check out my blog here. Spatial Computing is the convergence of emerging technologies such as Augmented Reality (AR), Virtual Reality (VR), computer vision, depth sensing and more.

My role, as Director of AR/VR at VMware, is to lead our product strategy around spatial computing with a focus on research and development of these technologies. In the build-up to VMworld 2019 we wanted to put together a few demos to showcase what our team had been working on (Project VXR). Given the short timelines, it was all hands to the pump and with a (short!) background in software engineering I was happy to get hands-on and contribute. It’s not often I get to stretch my technical legs, so this was certainly a fun and challenging project for me.

In this blog I’m going to cover our experiences around building immersive training scenarios for mobile or standalone virtual reality headsets. Specifically, I’m going to dive into using a scene simplification technology by Google called Seurat. Unfortunately, Seurat is not officially supported by Google, but it is an open source project that can be used and updated by anyone. The first part of this blog introduced immersive training and scene simplification. The second part of this blog is a technical how-to on Google Seurat.

Using Google Seurat in your immersive training scenario

For this blog I’m going to show two examples using two game engines – Unity and Unreal. Unity from Unity Technologies is one of the leading platforms for developing AR and VR applications. Unreal Engine from Epic Games is another leading engine to use for AR/VR application development. At VMware we’re working to accelerate your development of enterprise applications using Unity and Unreal.

Example immersive environment created with Google Seurat with full parallax when a user moves position

Before we start with Seurat, you should be aware that Google has not updated the code for Seurat since May 4^th 2018. This project is likely no longer maintained by Google. However, using this guide you will be able to effectively use Google Seurat and where necessary update it to support the latest versions of Unity and Unreal.

Google Seurat can be used with a variety of render/game engines including Unity and Unreal as well as common modelling software such as Autodesk Maya. Typically, Seurat “captures” are done in a render or game engine while Seurat output is imported into a game engine such as Unity or Unreal.

For this example, I’m going to use both Unity and Unreal to capture environments with Seurat and then show you how to import those captures back into Unity and Unreal. You need basic Unity and Unreal experience to follow along.

Google Seurat process

There is a three-part process to using Google Seurat – Capture, Process, Import.

To start you will need a source model or scene that you are going to simplify. This might be a building CAD model imported into Unity or Unreal, an existing high poly scene created in Unity or Unreal or prefabricated building/scene available via a 3D model marketplace.

Firstly, Seurat is used to capture images and depth information from a scene within a defined “head box”. This is usually done using a Seurat plugin for your chosen scene renderer (this could be Unity, Unreal or Autodesk Maya). The headbox represents the total area the user may move their head/body in VR, this is usually a 1-4m² area that covers from the floor to say 2m in height.

Secondly, those images and depth information are processed using Seurat to generate a new 3D model of the environment with associated textures. For this, Seurat has two applications that can be used to process the images and output the 3D model.

Finally, there is a process to import the Seurat model and texture into Unity (or Unreal) and render it using custom Seurat shaders.

If you want to know how Google Seurat works under the hood, there is a wonderful research paper by the authors here.

Getting Started with Google Seurat

Before we start, I want to highlight some great resources online for using Google Seurat, certainly I wouldn’t have been able to use Seurat without them.

Let’s start with the Google Seurat GitHub which can be found here – https://github.com/googlevr/seurat

The github pages offer some information on how to use Seurat but are by no means complete or can be confusing, which prompted this blog! The main Seurat GitHub project is focused on processing Seurat captures and includes source code for two applications:

Butterfly – a viewer for Seurat captures.
Pipeline – generates the 3D model(s) and textures that can be imported into your game engine

In order to process Seurat output you will need to either compile these applications from source code (out of scope for this blog) or more conveniently download the binaries from here. However, before you can do any processing using these applications, you will need to create some Seurat captures. In order to do that (and render the Seurat output) in Unity or Unreal, you will need to use the Seurat plugins for those engines. These plugins can be found here:

For Unity, the Seurat GitHub page is here – https://github.com/googlevr/seurat-unity-plugin

For Unreal, the Seurat GitHub page is here – https://github.com/googlevr/seurat-unreal-plugin

WARNING! For Unreal users, the current Seurat Unreal plugin does not work with Unreal Engine 4.16.3 and above. Fortunately, as part of this blog, I will detail how you can build the Seurat plugin for Unreal so that it’s compatible with the latest Unreal version.

If you are looking for guidance on Google Seurat with Unreal Engine, skip down to the Unreal section.

Let’s get started using Google Seurat with Unity!

If you want a good video tutorial on using Google Seurat in Unity then this video provides a step by step guide. There are some updates to this procedure that I will include in this blog to help you get the best output from Seurat.

I am going to assume that you have Unity installed. I am using Unity 2019.1.8f1. I also assume that you already have a project setup in Unity with an environment or CAD model that you want to capture. Let’s get started!

Download the Seurat Plugin for Unity by going to the GitHub page and cloning or downloading the repository.
Copy the SeuratCapture and third_party folders from the Asset folder in the ZIP/repo into your project Asset
Launch Unity and open your project, it should register the Seurat assets with your project
Open the Seurat scene, located in your Asset folder as shown
Click on the Seurat Headbox Capture entity and copy it to the clipboard (Ctrl+C)
Open your original scene and paste (Ctrl+V) the Seurat Headbox Capture entity
Optional: My original scene doesn’t have any models, so I will import a few high poly models. Do the same if you are starting with a blank project. I imported a factory CAD model with over 20m tri’s and a rocket CAD model with over 30m tri’s. I placed several of the rocket models in the scene.
Move the Seurat Headbox Capture entity to the same position where your user/player is located.
My current camera view (when selecting the Seurat Headbox Capture entity) shows over 200m tri’s in the frame.
Place the Seurat Headbox Capture entity at a height of 1.7m above the floor so the center of the headbox is at a typical user head height.
Before configuring the Capture Headbox (Script) component and capturing you must ensure that the headbox area you are using has all objects within it either removed or hidden. The critical thing to know here is that later you will need to copy and paste those objects back into a new scene that includes your Seurat 3D mesh.

Capture Process with Unity

Ok, now you are ready to start the Seurat capture process.

Click on the Seurat Headbox Capture entity within your Unity scene.
Under the Inspector panel, configure the Capture Headbox (Script) component with the following settings:
A quick overview of the settings:
- Output Folder is the folder where the color and depth images will be stored, I highly recommend this is not within your Unity project folder structure.
- Headbox size is the area that your player will be able to move while maintaining the correct view of the environment. This is in meters. Increase the box size to increase the playing space. You may want to adjust the Y value to 3 so that the box reaches the floor.
- Sample Count is the number of cameras that will be generated within the headbox (at equal spacings) where for each camera 6 color images and 6 depth images are taken (front, back, top, bottom, left, right). So, 384 images will be generated with a sample count of 32. Increase the sample count based on the headbox size. I’ve successfully used 32 when using a 2x2x2 or 3x2x3 box.
- Center Capture Resolution is the resolution used for the sample at the center of the head box.
- Default Capture Resolution is for the resolution used all samples outside of the center.
- Dynamic Range – I’ve used HDR with success.
Click on the Capture button and you should see the Capture Window appear:
Once the capture process is complete and the the Capture Window button has changed from Cancel to Done, check your Output Folder for the existence of a manifest.json file.

Process the Seurat output to create the 3D mesh and texture

The next step is to process the images captured by Seurat in order to output a 3D model and associated texture.

Copy the aforementioned binaries that either you compiled from source code or more conveniently downloaded from here to your Output Folder.
Open up a command prompt and navigate to the Output Folder and enter the following to create your 3D mesh and textures:
```
seurat-pipeline-msvc2017-x64.exe -input_path=manifest.json -output_path=test -premultiply_alpha=false
```
I highly recommend reviewing the command line parameters for the pipeline tool. Here are some key options to consider:
- gamma [default=1.0] : Gamma-correction exponent.
- triangle_count [default=72000] : The maximum number of triangles to generate.
- premultiply_alpha [default=true] : Determines whether output textures use premultiplied alpha.
- skybox_radius [default=200.0] : Half the side-length of the origin-centered skybox to clamp distant geometry. 0.0 indicates no skybox clamping should be performed.
- fast_preview [default=false] : Determines whether to prefer speed over quality.
Before hitting Enter be aware that the capture process can take some time, usually anywhere from 15 to 60 minutes.
Once the process is completed you should have a number of new files in your Output Folder, namely test.ice, test.png, test.obj and test.exr.

Check your Seurat 3D mesh with Butterfly

Seurat comes with the Butterfly application to preview your capture before importing into Unity or Unreal.

At the command prompt type the following

seurat-butterfly-mvsc2017-x64.exe test.ice

You will now see the result of your Seurat capture – using Seurat’s custom .ice format.
If you used the -premultiply_alpha=false option then you will see an output like this:
Now this output doesn’t look great, but its perfectly normal, that’s because by default Seurat Butterfly expects pre-multiplied alphas. As an example, if you did not change the premultiply_alpha setting you will see an output like this:
Its important to note, that in my testing I needed to use –premultiply_alpha=false for Unity to eliminate unwanted texture cracking.
You can use the left mouse button to look around, the right mouse button to zoom in and out and Q,Z,W,A,S,D keys to move your position.
Close the Butterfly application.

Importing Seurat 3D model and textures into Unity

Now you are ready to import your Seurat model and texture into Unity. There are a number of approaches here, you could create a new scene and import your Seurat model into that. For simplicity sake I’m going to import the model and texture into my existing scene and deactivate the existing models.

Open your Unity project.
Copy test.obj and test.png into your project:
Click on test.png and apply the test Import Settings as shown in the picture above. Make sure to turn Generate Mip Maps off, use Clamp as the Wrap Mode and select Bilinear for the Filter Mode. Finally, set the Max Size to 4096.
Its very important Unity does not resize the texture, if necessary override the texture Max Size settings in the platform settings next to the Default tab.
In my project I now disable the existing models in the scene as well as any lighting.
Drag and drop test.obj onto the Seurat Headbox Capture object in the Hierarchy tab. This ensures the model is in the correct viewing position.
Change the test object’s Rotation Y value to 180 as shown in the image above so that the model is oriented correctly.
Note that the number of tri’s is now only 73k!
Now the scene still doesn’t look right as we need to apply the texture and shader to the object. Drag and drop the test entity in the Hierarchy tab so that it is a direct child of the scene.
Click on the (right facing) arrow to the left of the test entity to see the default child object of the entity. This default entity represents the Seurat 3D mesh object.
Drag and drop test.png onto the default entity as shown below:
Within the Inspector tab configure the Shader component. Click on the dropdown with the value Standard and select GoogleVR > Seurat > AlphaBlended
Now you can test your environment by moving the around within the editor, note how the scene looks normal inside the headbox but quickly deteriorates the further you get from the headbox.

Bringing your objects back into the scene

Now that you have the environment set up there’s a few things you need to do in order for objects and props to appear in your scene.

Within the Inspector tab configure the Shader component. Click on the (right facing) arrow to open up the texture options.
Change the Render Queue value to 1999, as shown in the image above.
Now select your camera, either the Seurat Headbox Capture, your VR Camera or the Main Camera.
Within the Inspector tab configure the Camera component. Change the Clear Flags to Solid Color as shown in the image above.
Now both your Seurat model and other objects are visible from within the camera view.

That’s it you should be done! We’ve gone from over 200m tri’s to just 73k! For more information on importing Seurat meshes, Google offers details here.

Cracking, lines and gaps appearing in your scene

In certain circumstances you may see cracks or gaps appear in your scene. Google also has some guidance on this topic, however, here’s my thoughts:

If when you or the user moves in VR and the scene deteriorates then you need a larger headbox when capturing.
If you are seeing “cracks” / “lines” on the texture of your 3D model when imported into Unity or Unreal, you need to set -premultiply_alpha=false.
If your scene is dark, you can use gamma setting in the Seurat pipeline command line.
If you are seeing gaps in the 3D model then you have a number of options:
- Set the triangle_count=100000 in the Seurat pipeline command line.
- Increase the Samples Count in the Capture Headbox (Script) component

I have found that results vary with Seurat depending on the scene complexity, where the headbox is placed and things like lighting. Increasing Samples Count is really only useful as the headbox gets bigger. Triangle count is also not that effective at improving the result when it comes to gaps. I’d love to hear back on your experiences with using Seurat in Unity (@mcopping on Twitter).

Using Google Seurat with Unreal

Google has not updated the Seurat source code since May 2018. Since both Unreal and Unity have rapid and frequent engine changes, its likely Seurat will not work, at some point, with the latest versions of those engines. So far, for Unity, the plugin works well albeit with an adjusted command line when processing the Seurat captures (see details above).

For Unreal, Google Seurat no longer works with versions above Unreal Engine 4.16. The reason for this is that the Unreal plugin definition changed in 4.16.3. Also, in 4.21 an Unreal Engine function used by Seurat has been removed, causing further issues.

In the following section I will detail how to update the Unreal Plugin for Seurat and use Seurat with Unreal.

Before I start though I want to highlight a blog that helped me get started with fixing the Unreal plugin and using it with Unreal Engine. This blog details the process used to get Seurat working on Unreal versions 4.16-4.20. You don’t need to follow this blog, as I will detail how to do all this below.

First Let’s Fix Google Seurat Unreal Plugin!

I am going to assume that you have Unreal Engine installed – version 4.21 or above. I am using Unreal Engine 4.22.3. I also assume that you already have a project setup in Unreal Engine with an environment or CAD model that you want to capture.

You will also need to have Microsoft Visual Studio 2017 installed. If you don’t then skip this section and go to the next section on installing the Google Seurat Plugin for Unreal Engine.

Let’s get started!

Download the Seurat Plugin for Unreal by going to the GitHub page and cloning or downloading the repository.
Open Unreal Engine and create a new blank C++ project called SeuratPlugin
Once the project is open, close Unreal.
From the GitHub repo/Zip copy the folder Plugins within the Engine folder to your project folder
In your project folder open Plugins\Seurat\Source\Seurat\Seurat.Build.cs

Add the following code within the public Seurat class:

PCHUsage = ModuleRules.PCHUsageMode.UseExplicitOrSharedPCHs;

You code should look like this
Add the following code within the public Seurat class PrivateDependencyModuleNames.AddRange section
```
"ImageWriteQueue",
```
Your code should look like this
Save Seurat.Build.cs
In your project folder open Plugins\Seurat\Source\Seurat\Private\Seurat.cpp

Within the #include section add the following code

#include "Json.h"
#include "ImageWriteQueue.h"
#include "ImageWriteTask.h"

Your code should look like this

Find void FSeuratModule::WriteImage method and add the following code at the following point:

//HighResScreenshotConfig.SaveImage(Filename, OutBMP, DestSize); -- DEPRECATED 4.21 so using ImageWriteTask instead

// New Code by vMattC

// Log a few things
 UE_LOG(Seurat, Display, TEXT("Filename: %s"), *Filename);

// Check to see if ImageWriteQueue has been initialised
 if (!ensureMsgf(HighResScreenshotConfig.ImageWriteQueue, TEXT("Unable to write images unless FHighResScreenshotConfig::Init has been called.")))
 {
      UE_LOG(Seurat, Error, TEXT("FHighResScreenshotConfig has not been initialised"));
 }

// Create ImageTask
 TUniquePtr<FImageWriteTask> ImageTask = MakeUnique<FImageWriteTask>();

// Pass bitmap to pixeldata
 ImageTask->PixelData = MakeUnique<TImagePixelData<FLinearColor>>(DestSize, MoveTemp(OutBMP));

// Populate Task with config data
 HighResScreenshotConfig.PopulateImageTaskParams(*ImageTask);
 ImageTask->Filename = Filename;

// Specify HDR as output format
 ImageTask->Format = EImageFormat::EXR;

// Save the bitmap to disc
 TFuture<bool> CompletionFuture = HighResScreenshotConfig.ImageWriteQueue->Enqueue(MoveTemp(ImageTask));
 if (CompletionFuture.IsValid())
 {
     CompletionFuture.Wait();
 }

// End New Code

Your code should look like this
Save Seurat.cpp
Now we are ready to re-compile the plugin. Open Unreal Editor and open your SeuratPlugin project.
In the Unreal Editor select the Edit menu dropdown and select Plugins
Within the Plugins window scroll down the list of plugins until you find Project > Other
Select Other and click on the Enabled checkbox as above
A warning will appear to restart Unreal Editor, click on Restart Now
You should now get a window pop up with the title Missing SeuratPlugin Modules asking you if you would like to rebuild the modules now. Click Yes.
You should now see a progress bar pop up showing you that the plugin is being recompiled.
If the recompile is successful your project will launch in the Unreal Editor
- If the recompile was not successful an error message will pop up. You can check to see what the issue was using the logs within your project folder. Open Saved\Logs\SeuratPlugin.log or SeuratPlugin_2.log. Either of these logs should indicate where the issue in your code is. Fix as necessary.
In the Unreal Editor, check that the plugin was successfully compiled by clicking on All Classes and scrolling through the list of classes until you get to Scene Capture Seurat class.
You can now close this project and exit Unreal Editor.

Congratulations you have recompiled the Google Seurat Plugin for Unreal! Our next step is to now install the plugin into Unreal Engine so that all projects can use the plugin.

I can appreciate that this is a fair amount of effort and it would be easier for me to just update the open source repository on GitHub. As part of the Open Source efforts at VMware I will be contributing back to the Google Seurat source code in the near future, stay tuned.

Installing the Google Seurat Plugin for Unreal Engine

If you fail to build the Unreal Plugin for Seurat, you can download a version here.

Copy the Seurat folder from your project Plugins folder (or from the ZIP file provided as a download above) to your Plugins folder in your Unreal Engine installation path i.e. D:\Program Files\Epic Games\UE_4.22\Engine\Plugins
Launch Unreal Engine
Open a project where you have your high poly scene or CAD model or create a new project as I have below.
Once the project is open you will need to enable the Seurat plugin. Click on the Edit menu dropdown and select Plugins
Scroll down through the Built-in plugins until you get to Other. Click on Other, scroll down through the plugins until you find Seurat and click on the Enabled check box.
You will now get a warning to restart, click on Restart Now.
Once the project has restarted, close the Plugins dialog. You are now ready to start using Seurat with Unreal Engine!

Using the Google Seurat Plugin with Unreal Engine

Click on All Classes and type Seurat into the search text input title Search Classes
You can now drag and drop the Scene Capture Seurat class into your scene
For my example I’m going to use a free environment (Paragon) from Epic Games available from their marketplace and add that to the project.
Place the Scene Capture Seurat camera in the exact location your user will be positioned. Place the camera at about head height (170cm)
Before configuring the Scene Capture Seurat component and capturing you must ensure that the headbox area you are using has all objects within it either removed or hidden. The critical thing to know here is that later you will need to copy and paste those objects back into a new scene that includes your Seurat 3D mesh.
You should also be aware that animations, particle systems or any effects that change over time should be disabled. The scene should be static to avoid any inconsistencies between camera captures.
You should also turn off any post processing effects such as depth of field, motion blur etc. From the Google documentation the following is stated: “Each pixel must correspond to the color and depth value of exactly one camera ray. This means that antialiasing needs to be turned off and features such as depth of field and motion blur have to be disabled. The same applies to most screen space effects, e.g. bloom and tone mapping.”

Capture process with Unreal

You are now ready to setup and capture within Unreal.

Under the Details tab locate the Seurat Settings box. Note that the Headbox Size is in cm, configure your headbox to according to the area available to your user (usually 100-400cm², with a height of 200-300cm). Set the Samples Per Face to K32 and the Resolution to K2048 or K4096.
Unreal unlike Unity doesn’t provide an option on where to write the output. The capture output will be located in your project folder under \Intermediate\SeuratCapture
Click on the down arrow at the bottom of Seurat Settings to reveal the Capture Button
Press the Capture button to start the Seurat capture process.
Unreal unlike Unity doesn’t provide a progress bar for capture, but it will tell you when its done
Once the capture process is complete, check your \Intermediate\SeuratCapturefolder for the existence of a manifest.json file.
Note: In Unreal, Seurat only creates a single image combining color and depth information per capture, rather than individual color and depth images like in Unity.

Process the Unreal Seurat output to create the 3D mesh and texture

The next step is to process the images captured by Seurat in order to output a 3D model and associated texture. This is very similar to the process used with captures created with Unity, but with some important changes.

Copy the aforementioned binaries that either you compiled from source code or more conveniently downloaded from here to your project folder under \Intermediate\SeuratCapture
Open up a command prompt and navigate to your project folder under \Intermediate\SeuratCaptureand enter the following to create your 3D mesh and textures:
```
seurat-pipeline-msvc2017-x64.exe -input_path=manifest.json -output_path=test -premultiply_alpha=false -skybox_radius=20000
```
Note: With Unreal you have to set a large skybox_radius (recommend 20000) in order to capture your scene properly.
I highly recommend reviewing the command line parameters for the pipeline tool. Here are some key options to consider:
- gamma [default=1.0] : Gamma-correction exponent.
- triangle_count [default=72000] : The maximum number of triangles to generate.
- premultiply_alpha [default=true] : Determines whether output textures use premultiplied alpha.
- skybox_radius [default=200.0] : Half the side-length of the origin-centered skybox to clamp distant geometry. 0.0 indicates no skybox clamping should be performed.
- fast_preview [default=false] : Determines whether to prefer speed over quality.
Before hitting Enter be aware that the capture process can take some time, usually anywhere from 15 to 60 minutes.

You have now created a 3D mesh and texture that you can import into Unreal. Before you do that it’s worth checking the capture by using the Butterfly application. Please refer back to the section above entitled Check your Seurat 3D mesh with Butterfly

Importing Seurat 3D model and textures into Unreal Editor

Now you are ready to import your Seurat model and texture into Unreal. There are a number of approaches here, you could create a new scene and import your Seurat model into that or import the models into your existing scene.

The Google Seurat Plugin for Unreal GitHub page does a fantastic job with instructions on how to import your model and texture into Unreal. I’m not going to reprint that work here.

Please follow the instructions here

Once you have completed those instructions, you can test your environment by moving the around within the editor, note how the scene looks normal inside the headbox but quickly deteriorates the further you get from the headbox.

And that’s it! I’d love to hear feedback or suggestions for improving either the code written here or the instructions. Ping me on Twitter @mcopping or fire us an email at vxr@vmware.com.

Conclusion

We found Google Seurat to be a useful tool to allow us to put users in immersive environments when using mobile or standalone VR headsets. Our immersive training environments utilise Google Seurat, where that technology fits, and helps improve immersion whilst maintaining performance. I really hope this blog helped you in using Google Seurat to create realistic environments in VR. If you’d like more information on immersive training or are a customer looking to adopt spatial computing, please contact us at vxr@vmware.com.