JavaScript/WebGL lightweight and robust face tracking library designed for augmented reality face filters

This JavaScript library detects and tracks the face in real time from the camera video feed captured with WebRTC. Then it is possible to overlay 3D content for augmented reality applications. We provide various demonstrations using the main WebGL 3D engines. We have included in this repository the release versions of the 3D engines to work with a determined version (they are in /libs/<name of the engine>/).

This library is lightweight and it does not include any 3D engine or third party library. We want to keep it framework agnostic so the outputs of the library are raw: if the face is detected or not, the position and the scale of the detected face and the rotation Euler angles. But thanks to the featured helpers, examples and boilerplates, you can quickly deal with a higher level context (for motion head tracking, for face filter or face replacement...). We continuously add new demonstrations, so stay tuned!

Table of contents

• Features
• Architecture
• Demonstrations and apps
  • Included in this repository
  • Third party
• Specifications
  • Get started
  • Optional init arguments
  • Error codes
  • The returned objects
  • Miscellaneous methods
  • Multiple faces
  • Multiple videos
  • Optimization
  • Changing the 3D engine
  • Changing the neural network
  • Using module
• Integration
  • With a bundler
  • With JavaScript frontend frameworks
    • With REACT and THREE Fiber
    • See also
  • Native
• Hosting
  • The development server
  • Hosting optimization
• About the tech
  • Under the hood
  • Compatibility
• Articles and tutorials

• License
• References

Features

Here are the main features of the library:

• face detection,
• face tracking,
• face rotation detection,
• mouth opening detection,
• multiple faces detection and tracking,
• very robust for all lighting conditions,
• video acquisition with HD video ability,
• mobile friendly,
• interfaced with 3D engines like THREE.JS, BABYLON.JS, A-FRAME,
• interfaced with more accessible APIs like CANVAS, CSS3D.

Architecture

• /demos/: source code of the demonstrations, sorted by 2D/3D engine used,
• /dist/: core scripts of the library:
  • jeelizFaceFilter.js: main minified script,
  • jeelizFaceFilter.module.js: main minified script for use as a module (with import or require),
• /neuralNets: trained neural network models:
  • NN_DEFAULT.json: file storing the neural network parameters, loaded by the main script,
  • NN_<xxx>.json: alternative neural network models,
• /helpers/: scripts which can help you to use this library in some specific use cases,
• /libs/: 3rd party libraries and 3D engines used in the demos,
• /reactThreeFiberDemo/: NPM/React/Vite/Three-Fiber boilerplate.

Demonstrations and apps

Included in this repository

These demonstration are included in this repository. So they are released under the FaceFilter licence. You will probably find among them the perfect starting point to build your own face based augmented reality application:

• REACT/THREE FIBER boilerplate: /reactThreeFiberDemo

• BABYLON.JS based demos:

  • Boilerplate (displays a cube on the user's head): live demo, source code

• THREE.JS based demos - specific README about THREE.js based demo problems:

  • Boilerplates:

    • Boilerplate (displays a cube on the user's head): live demo, source code
    • Boilerplate with only 1 <canvas> element: live demo, source code
    • Same boilerplate but using neuralNets/NN_4EXPR_1.json as neural net, and displays 4 expressions: live demo, source code
    • Multiple face tracking: live demo, source code
    • GLTF fullscreen demo with HD video: live demo, source code

  • AR 3D demos:

    • Werewolf (turn yourself into a werewolf): live demo, source code
    • Angel/Demon (discover who of the angel or demon will win in this animated scene): live demo, source code
    • Anonymous mask and video effect: ,