Multi Object Detection
Multi object detection template is a great example of a powerful feature of Lens Studio such as SnapML. It helps you to create Lenses which can detect and track: cat, dog, potted plant, TV, car, cup, bottle. This guide leads you through its key features and explains how to use it.
Tip: This template is great to use in conjunction with Scan Triggers. Scan Triggers allow your Lens to be discovered by Snapchatters scanning certain objects. For example: your dog tracking Lens can be found when a Snapchatter scans a dog!
There are two major scene objects in the template:
ML Controller and
Frame Builder. The
ML Controller does all the detection job itself: it detects objects, calculates bounding boxes, adds the confidence number to it, etc. It also passes this information to the
Frame Builder for creating the photo frames.
The Frame Builder object is responsible for creating the photo frames and attaching scene objects to the detected objects on the screen.
In addition, this script allows you to choose which classes of objects you want to detect and which you don’t.
Tip: This script also provides some additional filters to smooth out the results of the ML model. Take a look at the advanced section below to learn more about this filtering.
Enabling and disabling classes
As mentioned, the machine learning model which is used in the template can recognize 7 classes of objects: cat, dog, potted plant, TV, car, bottle, cup. By checking or unchecking the corresponding checkboxes, you can exclude a certain class of objects from being detected. This exclusion will hide the augmentation and icon in the Lens. For example, in the picture below you can see that by disabling the Cat checkbox, the cat icon is not shown.
It is also possible to disable object detection at runtime, by tapping on the corresponding icon. This allows the Snapchatters to modify the Lens to their needs. In this case, the object detection will be “paused” and the chosen class of objects will not be shown until it is enabled back.
Note: The cat icon is grayed which means that cats will not be detected until it is allowed back.
Note: The detection of all the different object types is provided by one ML model. This means that even if you disable the object at runtime, it would not have an effect on the performance of the ML model.
If you want to manipulate the photo frame material for any of the classes you can change its properties in the
Inspector panel. Or you can completely replace the material with your own.
Note: The material that is used on the carousel icons has the
Depth Test property turned off. This is done to avoid overlapping issues with the underlying photo frames. Make sure you turn it off if you want to replace material for the icons.
Tracking Scene Objects to Detected Objects
It is possible to attach a scene object to the center of the bounding box of the detected object. To do so you can assign one to the corresponding property of a class.
Note: In this case the photo frame will not be shown. That way, each of the classes can have its own scene object to be attached to.
Attaching to anchor
If you unfold the
Photo Source Image scene object, you will see the Anchor scene object. This child object exists on every photo frame you see in the preview. So if you add a scene object, a 3D model, a prefab, or anything like that to its hierarchy, it will also be instantiated. Use it if you want to attach something to all photo frames at once.
Finally, there is the
Anchor Position property which is responsible for adjusting the position of the anchored object.
Tip: Since the
Photo Source Image is in the
Orthographic layer, you will have to ensure that the added anchor object is also in the
Sometimes the result of the machine learning model can be noisy: the bounding boxes can be found and lost in just two frames, or the bounding box of the same object can vary in size between two adjacent frames which gives us undesirable flickering of the boxes. To improve the result, the template comes with some post processing clean up techniques such as
Cross Frame Adjustments and
Small Area Threshold.
The first one toggles the cross frame adjustments in general. When it is checked, the script will apply the following clean up techniques:
- Cross Frame Adjustments Show bounding boxes only for those objects that have been tracked more than three frames in a row. In other words, do not show bounding boxes if it was just a one or two frame “blink”.
- Small Area Threshold Show bounding boxes only if their area is greater than the Small Area Threshold value.
Publishing Your Lens
Consider using the Scan Trigger feature when publishing your Lens. This is a great combination with SnapML as it enhance user experience by allowing them to immediately start detecting objects in the camera right after they activated the Scanned those objects.
Still Looking for help?Visit Support