MediaPipe 3D Face Transform

In the blog post, Kanstantsin Sokal from the MediaPipe team introduces a new face transform estimation module that enhances the existing Face Mesh solution, capable of estimating a 3D face shape in real-time on mobile devices through 468 landmarks. The module offers a semantic API for determining 3D face poses and attaching virtual objects, such as glasses or masks, by transitioning from screen coordinate space to a metric 3D space. This transition is facilitated by Procrustes Analysis, which is lightweight and operates on a CPU with minimal resource use. The module enables the alignment of virtual 3D objects with detected faces by using a canonical face model and a face pose transformation matrix. It includes a face effect renderer that supports real-time rendering of 3D objects and face mesh effects via OpenGL ES 2.0, which can be utilized on mobile applications. The module is part of the MediaPipe Face Mesh solution and includes examples and resources for developers to implement custom use cases.