Creating a Face Filter

#MixedRealities

For the first week of my Designing Mixed Realities course we learnt about Virtual Reality (VR) and Augmented Reality (AR). For our technical experiment we were tasked with creating a face filter using the software Spark AR. A face filter is a form of AR as it tracks your face and actions that you do and overlays a piece of virtual reality onto the real world, it has enhanced the real world so to say. When you use a face filter on your phone it is a form of handheld projector and is attached to your body as you hold it in your hand.

Now I have never attempted to make a filter before, so you could say that I really challenged myself for this first week. I wanted to create an animated background as well as some face features - this was a big task, and it took a lot of Youtube tutorials. I chose to create an animated background though as I’d never created an animation before and I also hadn’t seen many filters that had animated backgrounds.

The Background

To create a background, you have to separate the person from the background, which is called person segmentation. The AR software is able to track and detect a person and have the virtual background appear in a fixed position. I made the background from 6 different frames, this was pretty simple to do. I created the first image and for each subsequent frame, I changed the rotation and size of the stars and moon and moved the spaceship across the plane. The most important thing is to make the spaceship look like it is moving across the screen in a smooth motion by only moving the spaceship along in each frame by a small amount so that when you view the frames in sequence, your eye pieces the still frames together and it becomes a video.

The Face

Once the person has been separated from the background, the face can also be tracked and what’s called a face mesh can be applied which looks a bit like a mask. This is a virtual application which allows you to add materials and textures to a face. Filters which change your features such as your face shape or adding glasses or makeup would be added to the face mesh. It is a virtual mask applied on top of your real face.

I added a particle system to my face tracker which flows stars out of your eyes and mouth when you open it. This interaction needs to be coded through a patch editor. This is through a mouth tracking system, which can track when a mouth is open and even how far open it is. I edited the patches so that when the mouth is open, the particle system is applied at a birthrate of 15 which is how fast the particles are produced.

The interaction adds another layer of AR, it’s fun to be able to produce changes when you do something with your face. I think this is also an important distinction between AR and VR as in a virtual reality, everything is completely made up, animated and not real, whereas using the filter, you can still see your real face, it’s adding a piece of magic to the real world which I think is why it’s so fun.

Adding sound

An exciting aspect of creating the filter was adding in sound effects. I created a patch code much like with the particle systems but replaced it with sound. So when the mouth is detected to be open, the sound is played. I loved the bubble sound effect and thought it was funny as an illogical sound in relation to the space background.

Conclusion

As I don’t have much experience working with software like this, I think it was relatively easy to pick up and be able to create amazing filters. The hardest part was learning the different terms such as person segmentation and particle system and being able to locate them, however after watching tutorials and understanding how to find these different additions, it became a lot easier to work with and you have a lot of creative freedom with what you can do.