The Custom Segmentation template demonstrates an example of how you can apply an effect to a certain part of the screen based on a mask. The template comes with an example model for creating a mask on Pizzas, as well as a Graph Material which makes the pizza look hot.
Unlike the Segmentation template, which utilizes the built in Segmentation Texture to mask things like the sky, hair, and background, the Custom Segmentation template gets its segmentation mask from a machine learning model through the ML Component.
Note: If you have your own segmentation model, you can skip down to the Importing Your Model section below. You can also skip to the Customizing Your Lens Experience section if you want to use pizza segmentation.
Creating a Model
While the template comes with a pizza segmentation model for the ML Component, you can make any kind of segmentation mask by importing your own machine learning model. We’ll go through an example of what this might look like below.
Note: To learn more about Machine Learning and Lens Studio, take a look at the ML Overview page.
To create a model, you’ll need a
- Machine learning training code: code that describes how the model is trained (this is sometimes referred to as a notebook--download our example notebook).
- Data set: collection of data that our code will use to learn from (in this case we will use the COCO data set).
Tip: This dataset comes with multiple labels that you can modify in order to change what kind of mask, try creating different ones.
Training Your Model
Head over to Google Colaboratory, select the
Upload tab, and drag the python notebook into the upload area.
Tip: The notebook is well documented with information about what each section of the code is doing. Take a look at the notebook to learn more about the training process itself!
Once your notebook has been opened, you can choose which labels you want the model to segment out. Then, you can run the code by choosing
Runtime > Run All in the menu bar. This process may take a while to run, as creating a model is computationally intensive.
Downloading your Model
You can scroll to the Train Loop section of the notebook to see how your machine learning model is coming along.
Once you are happy with the result, you can download your
Note: When using a data set to train your model, make sure that you adhere to the usage license of that dataset.
Importing your Model
Now that we have our model, we’ll import it into Lens Studio.
You can drag and drop your
.onnx file into the Resources panel to bring it into Lens Studio.
Next, we’ll tell the template to use this model. In the
Objects panel, select
ML Component. Then, in the
Inspector panel, click on the field next to
Model, and then in the pop up window, choose your newly imported model.
You should now see the
Preview panel showing the heat effect on the area that is segmented by the model.
Customizing your Lens Experience
You can see in the ML Component, that it has an
Output Texture of
Segmentation Texture. This Texture can be used in any way that textures are used.
Using the output texture in material editor
In the case of this template, the texture is being used as input to the Material Editor to create the heat effect.
blur_material takes in the segmentation texture and applies a gaussian blur to it using the gaussian blur subgraph.
The segmentation texture is passed in to the material using the
Texture 2D Object Parameter node to allow your material to access inputs:
Tip: Take a look at the Material Editor guides for more information.
Then, it applies this post effect on the
Blur Camera, which has the
Render Target of
Blur Target (resulting in a blurred segmentation texture).
Next, to create the heat effect, the blurred segmentation texture, as well as the original segmentation texture is used as input to the
head_distortion material. Play around with some of the parameters of heat_distortion and check out the graph to see how it works!
Tip: the heat_distortion material uses the segmentation mask to determine where the area should be grayscale (pixel is not pizza), and then uses the blurred segmentation mask so that the smoke noise can appear on top of the pizza.
Using Segmentation Texture to cut things out
You can also use the
Segmentation Texture directly on a camera to mask what it renders.
For example, let's say we added a color correction object to make our image be sepia toned.
Then, you can select the
Camera object, and use the
Segmentation Texture from the ML Component as the Camera’s
Tip: Take a look at the Segmentation guide to learn about how you can use segmentation textures to cut out parts of an object or image.
Previewing Your Lens
You’re now ready to preview your Lens! To preview your Lens in Snapchat, follow the Pairing to Snapchat guide.
Please refer to the guides below for additional information:
Still Looking for help?Visit Support