The Hand Segmentation Template let’s you easily segment your hand and replace it with an image or a custom material.
This template shows how you can use the hand segmentation to occlude an object as it goes behind the hand. Additionally, there are other examples for placing an image on a hand and placing images with a parallax effect.
This template also shows the other features of the Lens Studio like hand gestures, hand orientation as well as behavior to create a unique interaction.
Scene Set up
Hand segmentation uses 2D space (Screen Space) to apply the segmentation effect to users' hands. This template comes with a scene set up that utilizes the Hand Segmentation mask to create different types of effects related to the hand:
- Segment an image within the hand
- Segment images within the hand with a parallax effect
- Use the hand as an occluder for 3D objects
Feel free to skip to the relevant sections to find what you need!
Tip: Learn more about how Hand Segmentation works on its guide page.
Objects panel, you will see two cameras in the template.
Camerawhich is responsible for rendering all the visuals in the scene.
Hand Segmentation Mask Camerawhich is responsible for rendering the hand segmentation texture.
The Hand Segmentation Mask Camera generates a texture that shows what is the hand and not the hand (
Mask Camera Render Target in the
Resources panel) that is used by objects visible in the Camera to determine where some visuals should be shown or not shown.
In this template, for the most part we only have to deal with the visual aspects of the Lens since the scene is already set up. Notice how the examples and visuals in this template are visible in the blue (
Default) layer--which is the same layer as the
Tip: If you don’t see the layer indicator, press the Eye icon in the top right of the
Objects panel and enable it.
Hand Segmentation Mask Camera
However, feel free to open the
Hand Segmentation Mask Camera which contains objects used in the yellow layer to see how the hand segmentation mask is created. This camera also contains the
Hand Tracking object which we use to get the position and rotation of the hand in the screen space so we can convert that into 3D space for use in the world space
This means that the camera uses a render target to render the hand segmentation to it. You can find the
Render Target in the
Resources panel, under the
Textures folder, called
Mask Camera Render Target.
Note: To learn more about this set up, check out the Camera and Layers guide to learn more about render targets and camera.
With our understanding of the scene, let's take a look at how the visuals in this template is set up!
Attach To Hand In 3D
Attach To Hand helper allows you to attach any objects, regardless of whether it’s 2D, 3D, Text and more to your hand.
To make sure objects appear on the hand, just place any content as a child of the
Attach To Hand In 3D [OBJECTS_HERE] in the
To customize, select the
Attach To Hand In 3D [OBJECTS_HERE] object in the
Objects panel and adjust it’s setting in the
On Tracking Lost
This option allows you to either keep or hide all the child objects as soon as hand tracking is lost.
This option will apply rotation data from the hand tracking object to all the child objects.
This script can send triggers to the behavior script as soon as hand tracking starts tracking or lost tracking.
To use this trigger and make a custom action without coding, you need to import the behavior script into your project. To learn more about how to import and use behavior script please refer to the Behavior guide.
To make sure the script is working properly, please assign a perspective camera (Make sure it's NOT an orthographic camera). Also make sure to assign an
Object Tracking. This script can be used for other Object Tracking for example cats or dog tracking.
This template comes with 3 examples. Take a look below for a description of each example and information about their settings
2D Texture Example
These two examples are showing how you can assign an image to get displayed only on hand.
To see the example make sure to enable the object in the
Simple Hand Texture
This example uses a simple custom graph to apply a texture to hand segmentation.
To see how it works, select the object and then in the
right click on the material and press
This will automatically select the material which is applied to the
Tip: To see how the custom material works on this example, just double click on the material itself and it will open the material in the
Material Editor panel.
Parallax Hand Texture
Parallax Hand Texture example, uses the position data from the
Hand Object Tracking to create a parallax effect look applied to the hand.
This example uses a helper script called
ScreenTransformToMaterialProperty to get a position data from
Hand Object Tracking and then convert that data into a screen point.
After converting the data to screen point, this will pass the data to a material pass which is defined by the user. In this case it passes the data to the center property.
To customize the material, select the
Parallax Hand Texture [ENABLE_TO_TRY] object in the
Objects panel, and then like before, in the
Inspector panel, right click on the material and press
Select. This will select the material, allowing you to customize its properties.
Similarly. to see how the custom material works on this example, just double click on the material itself and it will open the material in the
Material Editor panel.
3D Object Example
The main example in the Hand Segmentation template is the
3D Object example which shows how you can use an occluder in 3D space so you can have an object to move around a hand and as soon as it goes behind the hand it will get occluded.
Hand Occluder [DO_NOT_MODIFY] has a material which contains
Render Target from the mask camera. You can use this object to occlude any objects that go behind the object. You can use a gizmo provided by the template to see where the object gets occluded in the
Note: The gizmo is just an image of the hand and can be used as a reference in the
Scene panel. It is not actually displayed in the scene.
Tip: The Airplane object uses a Tween script to rotate around the hand.
Effects (Helper Scripts)
This example comes with some helper scripts to help create a more immersive experience through interactivity.
You can find these effects in the
Object panel, under the
Attach To Hand In
3D [OBJECTS_HERE] -> 3D Object Example -> Airplane -> Effects Controllers.
Let’s take a look at all the effects as well as helper scripts listed below.
Hand Gesture Controller
The Hand Segmentation template uses hand gestures to create unique effects. This is the same script which is used in the Hand Gesture Template. To learn more about how the Hand Gesture Controller is working please refer to Hand Gestures Template guide.
World Particle Controller
The template comes with world particles. These particles are set to simulate in world space.
With the help of the world particles we can create a trail of smoke in the scene.
The way world particles work is that we used the same particle system which comes with the Lens Studio. Then we have a helper script in the template called
WorldParticleController. With this helper script you can assign your particles effect and then this script creates copies of the same particle and places it in the scene.
This helper script exposes some APIs so you can update the particles in run time. You can use Behavior to call these APIs with no coding at all.
Take a look at the following examples below to see how they work:
Play or Stop Particles
To start or stop the particle you can call script API using behavior script and call the function to play or stop.
Update Particles Color on Gesture
To change the color of the particles you can use a behavior script and call the
updateParticlePasses function. When you call this function make sure to pass a pass name as well as a value related for the pass you want to change. In our example, we modify the
colorRampTexture of the particle material.
Gesture Speed Control Effects
Another example provided by the template is using the rotation of the hand and applying an effect based on the rotation value. In this template we use rotation angle to change the speed of the plane, as well as changing the hand color as value increases.
This effect starts working as soon as the user does the
One Finger gesture and then can change the speed.
For example, in the
Objects Rotation to Material Parameter example, we get data from the
Hand Tracking Center Object and then we convert that into RGBA data. With use of this script we can change the color on the user's hand to turn to white on the lowest value and turn red on the max value.
There are a couple more examples using Object Rotation which you can find under the
Gesture Speed Control Effects object in the
Objects panel. Feel free to play around with these as you did above.
Exporting the template to use in another project
You can also use this template in an existing project as well!
As mentioned before, the project contains the hand segmentation under the
Hand Segmentation Mask Camera, as well as the
Attach To Hand In 3D [OBJECTS_HERE] object which contains the actual visuals. You’ll need to export both pieces in order to export the template. In addition the animations are run through the Tween system found under
Helper Scripts object.
To export the effects in this template to use in another project, First create an
Empty Object by pressing the
+ button in the
Objects panel and selecting
Then place the
Attach To Hand In 3D [OBJECTS_HERE],
Hand Segmentation Mask Camera as well as
Helper Scripts as a child of newly created objects.
Finally, right click on the new scene object and select
This will create a
.lso file which you can import to your own project.
Tip: Learn more about exporting and importing object in this guide.
Note: After importing into a new project, make sure to place the helper scripts on top of the hierarchy in the Objects panel.
Previewing Your Lens
You’re now ready to preview your Lens. To preview your lens in Snapchat, follow the Pairing to Snapchat guide.
Please refer to the guides below for additional information:
Still Looking for help?Visit Support