MICROSOFT HOLOGRAMS

Contract Duration: August 2017 - February 2019
Tools: Proprietary systems, Autodesk Maya, Adobe Photoshop, Unity3D
Languages: C#
Desc.: At Microsoft's Mixed Reality Capture Studios, I was a contract Processing Technician and 3D Generalist on holograms (moving photogrammetry) for internal and external clients.

I performed the following:

Generated point clouds, UV maps, and textures from RGB and infrared photos using proprietary systems
Detailed and adjusted meshes using scripts and/or hand-fixes on subdivided meshes
Enhanced textures via feedback from our in-house art director using automated Photoshop actions
Tested mesh quality using lighting tests

Demo Reel

Please Note:

I can only show published footage in this reel due to Microsoft's policy. Any wireframes, in-progress shots, etc. displayed have been shown publicly by Microsoft and/or their clients.

Hologram Creation Process

Please Note: This was written with my NDA in mind.

I cannot talk about the algorithms or anything else related to the design of the systems discussed here
I cannot show in-progress shots of any of my demo reel content that have not already been published publicly
I did not work as an engineer on this team; I worked as a 3D generalist

All work showcased in my demo reel was produced through a combination of Microsoft's proprietary photo-to-model generation software and post-processing edits through Autodesk Maya and Adobe Photoshop.

I split my time per project into thirds:

Test Renders: working live on set to produce test renders and assist in advising the client

Mesh Generation: generating meshes from photos taken on the greenscreen stage and testing mesh quality against lighting standards

Test Renders: additional editing work based on feedback from my art director and the client

Test renders were offered to clients to ensure a mixed reality capture was suitable for their project. A representative would visit us at our green screen stage with examples of their subject matter. My art director, the stage crew, and I would process small groups of frames as proof of concept for the client.

My role during this phase was providing solutions and advice on the spot. Because I had accumulated experience across different projects, I was able to quickly determine key pain points in capturing different subject matter. This allowed my art director and our clients to save time through accurate project decisions.

Mesh Generation

Mesh generation was produced through Microsoft's proprietary greenscreen stage software utilizing photos taken from 106 RGB and infrared cameras each. I would receive these photos from our stage team and process them through another proprietary application. This allowed me to break up mesh generation into different stages:

Point clouds: representations of a 3D objects plotted in the XYZ coordinate system

Retropology: reconstruction of high-poly meshes into simplified versions that retain their original shape

UV mapping & texturing: placing 2D texture images on the 3D model at specific coordinates

I manually controlled each individual stage through an interface, allowing me to continuously test iterative edits on small groups of frames. This was especially useful in discovering solutions for edge cases, such as thin objects like high-heeled shoes and complicated patterns on a dancer's clothing.

Test Renders

Post Processing

Once the general output was approved, I worked on polishing the look of the capture. This was comprised of two main actions:

Cleaning up meshes: Using a combination of PyMel and hand-fixes, I adjusted vertices and faces that disrupted the subject matter's silhouette at specific frames.

Improving textures: My art director would color balance a sample texture from the capture using Photoshop and pass me the values he wanted. I took his settings and applied them across thousands of frames using Adobe’s automation features. I also learned how to create my own Powershell scripts to batch rename these frames as necessary.

At this point, I would continue to iterate, or the project was considered completed upon approval from the client.