Fragments Found My Couch, But How?

UPDATE: 12/14/2016 – We now know that Asobo Studios used an unannounced library to implement the features described in the article below. That library would eventually be released into the toolkit as Spatial Understanding. The article below still serves as a good description of how to use the simpler (and faster) ‘Plane’ functions in the toolkit, but if you’re interested in Spatial Understanding I highly recommend this excellent article by Mike Taulty.


I finally took the time yesterday to really play Fragments, and I gotta say, it’s my favorite demo so far. There’s just so much about it that I find fascinating. The way my living room was mapped to a crime scene, the way the holograms break apart when you come to close to them, the way you hunt for and find evidence, and the way you interact with the characters.

The AI assistant is interesting enough, but when I had my first “Holo Call” and I looked over to see Director Kirkland sitting on my own couch… I mean wow. And that got me started thinking: “How’d they do that?” So I did some digging.

As it seems is the case with most juicy bits right now, that code can be found in the HoloToolkit for Unity. Specifically the two files involved are PlaneFinding.cs and SurfaceMeshesToPlanes.cs. PlaneFinding is the lower-level of the two and its job is to interface with the SDK and hardware. SurfaceMeshToPlanes is what most applications will actually interact with to get work done.

SurfaceMeshToPlanes offers some really useful features. For example it has a two properties FloorYPosition and CeilingYPosition. They tell you the lowest and highest points scanned in the room, respectively. It also has a method called GetActivePlanes that takes in a PlaneTypes enum and returns a list of Unity GameObjects that match.

The PlaneTypes enum is defined in SurfacePlane.cs as:

/// <summary>
/// All possible plane types that a SurfacePlane can be.
/// </summary>
public enum PlaneTypes
	Wall = 0x1,
	Floor = 0x2,
	Ceiling = 0x4,
	Table = 0x8,
	Unknown = 0x10

So you can probably see where this is going.

I don’t have access to the source code for Fragments, but I’d bet money it works something like this:

  1. Call SurfaceMeshToPlanes.GetActivePlanes, pass in PlaneTypes.Table
  2. Look for “tables” that are somewhere between 16″ and 24″ tall
  3. We already know where the floor is, so we put the Directors feet there
  4. Next, we plant her butt squarely on the “low table” (which is the couch)
  5. Finally, we use Inverse Kinematics to calculate where her knees should go

That’s actually pretty simple, and some really cool stuff!

You May Also Like

One thought on “Fragments Found My Couch, But How?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.