3D Body Tracking
The 3D Body Templates allows you to track the position and rotation of joints on the body in 3D. You can track one or more bodies at the same times, as well as partial bodies.
This template comes with four different examples of how you can use 3D Body Tracking.
The first three example, has a 3D model rigged to a skeleton that matches what the tracker sees. This allows the Lens to map the movement of the body, to the movement of the model. In the last example, we have individual objects attached to individual joints of the body. This guide will go through each of these examples, as well as how they are made.
Let's take a look at how our rigged model is set up. The
Stick Figure example shows the most straightforward use case.
To see how it works, unfold the
Stick Figure object in the
Objects panel by pressing the triangle next to it, and select
3D Body Tracking 0 object.
Inspector panel, you'll see that this object has an
Object Tracking component which has a mapping of the model's joint, to the trackers'.
The component requires that the character you use have a certain skeleton structure so that it knows how to correctly drive the model.
Tip: You can drive separate pieces by attaching them to the corresponding slots of the component and it will work just fine too! But we will see this a bit later.
The character you want to manipulate as a whole should have the following hierarchy and naming convention:
Having proper hierarchy and naming allows you to use the automatic way to assign each of the joints to its corresponding slot in the tracking component.
Note: The tail joints with
_end suffix at their name are auxiliary and are not used by the component.
An important step of the workflow with body tracking is that the character should be in T-pose. Like in the image above. The object tracking component takes into account the initial rotations of the character’s joints and adds the person’s movements into those initial values. So, for the best results, even if your character initially is not in perfect T-pose, you can rotate its joints a little bit to make it match T-pose.
Finally, for the best result, you should have the character itself should face the camera. In other words, it should have zero rotations of its parent transform.
Adding a Character
Once you are sure your character satisfies the requirements, it is time to bring it into Lens Studio.
First remove the
Stick Figure [REPLACE_ME] scene object, and parent your character to the
3D Body Tracking 0 object. Then, drag your 3D model from your computer to the
3D Body Tracking 0 object to make it its child.
Tip: If you don't see your 3D model in the
Objects panel, you might have brought them into your
Resources panel. Find the prefab (asset with the letter P on a box) that was made from your 3D Model, and drag it into the
Objects panel to instantiate it.
Note: To be able to use automatic mapping of the character’s joints to the corresponding slots of the
Object Tracking 3D component you must place your character under the root of the body tracking scene object.
With your model placed under the
Object Tracking 3D object, you can select the object and link the joints of our character to the joints of the body. To do this, in the
Inspector panel, press the
Match Hierarchy button in the
Then select the
Hips scene object of your character. This will assign joints of the character into the inputs of the component.
And with that you should be able to see your character is copying movements of the person on the preview video!
Tip: If you don’t see any changes in the preview or some misplacements of the character try refreshing the Lens by hitting the
Refresh button in the top right corner of the
Warning: Make sure that your character does not have any animation by removing or disabling any Animation Mixer components on it. Affecting the joints of the character by scripts or constraints can lead to unexpected results as well.
Animating a Character
Next let's take a look at the
Mark Example. In the
DancingMark example, you'll notice that we have our character detached from the Snapchatter. There are two settings in the
Object Tracking 3D Component found in the
Object Tracking 3D object to look at when wanting to do this:
Track Position, and
In this example,
Mark is freely movable and not attached to the user--so we disable the
Track Position option.
Tracking Mode is set to
Pose only--since we don't want the shape of Mark's character to be modified by the Snapchatter.
Tip: Here we use a
Manipulate Component to allow the user to move
Mark around. Notice that next to
Mark there is a
Touch Collision box which determines where a user can touch the screen to move
Mark. Don't forget to resize this box when replacing
Mark with your own character. Lastly, checkout the
Behavior script next to it which allows the user to tap on
Mark to enable or disable his shadow--this is useful, if the user does not want the character to be on the floor.
Attaching a 3D Suit to the Snapchatter
You can also use 3D Body Tracking to apply 3D models as a suit on a person rather than as a separate character. Let's take a look at the
Space Suit example to see an example of this.
Here, unlike before we enable the
Track Position checkbox, so that our 3D model moves with the Snapchatter's body.
Moreover, sometimes your 3D model could look a bit over- or undersized compared to the person in camera. The shoulders might be wider or the character itself can be taller. To solve this problem we can use the
Proportions And Pose
Preview panel, you can see how the 3D model's proportion is modified based on the two Snapchatters in the camera.
This option works best with human sized characters. That is why it will not change the proportions a lot. If you go back to the Marks example and play around with the parameter you will not see much difference in the proportions of Mark's body.
The best way to understand this option is to treat it as fine tuning for the proportions of human sized characters.
Tip: To ensure that your character is placed properly over the person on the video, you should parent the parent
3D Body Tracking object is under the
Camera hierarchy and its scale to
Note that any child scene object of the root scene object of your character will also affect placement and scale of the skeleton even if Proportions And Pose mode is active. In other words, in
Proportions And Pose, only the scene object with the Object Tracking component on it (3D Body Tracking scene object in our case) does not affect placement of the character. Any other transforms, except for Hips, upper and lower in the hierarchy that affect placement of the skeleton will keep affecting it.
Spacesuit [REPLACE_ME] and
Skeleton objects affect placement and scale of the
Spacesuit even though it is set to track position and proportions. It is advisable to check that the scale of these auxiliary scene objects is set to
1 and positions to
Warning: To have your character stretched properly you need to assign ALL the joints to the Object Tracking 3D component.
Of course one of the coolest features of this technique is that you can track more than one person at a time!
To see this all in action, make sure that you still have the
Two People Dancing 1 preview in the
Preview panel, select the body tracking scene objects and see that one of the components has the Object Index property set to 0, another to 1. 0 being the first person detected, and 1 being the second person detected, and so forth.
Although we are working only with two people on the previews, the body tracking feature is not limited to two people.
Tip: This is similar to the Index options that can be found in Head Attachment!
Sometimes, when the people in the camera overlap, it is impossible to know the position of the invisible parts of the body. In this case, the
Object Tracking 3D component will disable (hide) the joints that are not being tracked automatically!
If the pieces of the mesh of your character are parented to its joints you don’t need to do anything. If the component cannot track a joint it will hide it, and as the mesh is part of its hierarchy, it will become invisible as well.
However, if you have the mesh in its separate group and not referenced by the
Object Tracking 3D Component, you will need a simple script that will disable the pieces of the mesh if the corresponding joint is disabled. You can see an example of that script attached to the body tracking scene objects in the Spacesuit scene object hierarchy.
Tip: In the Space Suits example, we use Head Attachments to attach a visor to the face. Although the 3D Body Tracking system callows you to attach things to the head, you will get more accurate tracking with the dedicated Head Attachment component.
Attaching Separate Pieces
Alternatively, it is also possible to attach individual objects, rather than a skeleton rig to the body. You can simply attach separate objects for the user’s joints and that will work just fine.
You can activate the
Abstract Gear example, to see how it is done.
Unfold it and select
3D Body Tracking 0 scene object. In the
Inspector panel, you will see that only three objects which are connected to the related joints field in this case.
Tip: This is a great way to attach objects to parts of your body (like a tie, or shoulder pads), as the attachment works even when the user is close up.
Previewing Your Lens
You're now ready to preview your world Lens experience. To preview your Lens in Snapchat, follow the Pairing to Snapchat guide.
Please refer to the guides below for additional information:
Still Looking for help?Visit Support