Loading repository data…
WebAR-rocks / repository
Lightweight WebGL and JavaScript library for real time object detection, tracking and 6DoF pose estimation in the browser 📦. Included in WebAR.rocks.train for custom neural network training.
🚀 NEW: Since March 2025, you can train your own neural network using your 3D model(s) with WebAR.rocks.train.
Here are the main features of the library:
The target object must have an aspect ratio between 1/2.5 and 2.5. An object with an aspect ratio of 1 fits into a square (equal width and height). For example, the standard Red Bull can has an aspect ratio of 2.5 (height/diameter).
Elongated objects, such as a fork, a pen, or a knife, do not meet this requirement. In such cases, it may be easier to target only a specific part of the object (e.g., the end of the fork). We only detect objects that fully fit within the camera's field of view (i.e., objects that are not partially visible).
We can train a neural network to detect up to three different objects simultaneously. The first detected object is then tracked (we currently do not support simultaneous multi-object tracking). The recognized objects should have approximately the same aspect ratio.
Highly reflective objects, such as shiny metallic items, are harder to detect. Similarly, refractive materials are more challenging due to their high variability.
We don’t need any picture of the object but a 3D model. The 3D model should be in one of the following file formats: .OBJ, .GLTF, or .GLB. The textures should have power-of-two dimensions, and their highest dimension (width or height) must be 2048 pixels or less.
If necessary, the 3D model should embed the PBR textures (typically the metallic-roughness texture).
We provide 3D modelling support.
/demos/: source code of the demonstrations,/dist/: heart of the library:
WebARRocksObject.js: main minified script,/helpers/: scripts which can help you to use this library in some specific use cases (like WebXR),/libs/: 3rd party libraries and 3D engines used in the demos,/neuralNets/: neural network models,/reactViteThreeFiberDemos: Demos with Vite/NPM/React/Three Fiber.These demonstrations work in a standard web browser. They only require camera access. They are written in static JavaScript
These demonstrations have been written in a modern front-end environment using:
You can browse adn try them in the /reactViteThreeFiberDemos directory.
To run these demonstrations, you need a web browser implementing WebXR. We hope it will be implemented soon in all web browsers!
Then you can run these demos:
The most basic integration example of this library is the first demo, the debug detection demo.
In index.html, we include in the <head> section the main library script, /dist/WebARRocksObject.js, the MediaStramAPI (formerly called getUserMedia API) helper, /helpers/WebARRocksMediaStreamAPIHelper.js and the demo script, demo.js:
<script src = "../../dist/WebARRocksObject.js"></script>
<script src = "../../helpers/WebARRocksMediaStreamAPIHelper.js"></script>
<script src = "demo.js"></script>
In the <body> section of index.html, we put a <canvas> element which will be used to initialize the WebGL context used by the library for deep learning computation, and to possibly display a debug rendering:
<canvas id = 'debugWebARRocksObjectCanvas'></canvas>
Then, in demo.js, we get the camera video feed after the loading of the page using the MediaStream API helper:
WebARRocksMediaStreamAPIHelper.get(DOMVIDEO, init, function(){
alert('Cannot get video bro :(');
}, {
video: true //mediaConstraints
audio: false
})
You can replace this part by a static video, and you can also provide Media Contraints to specify the video resolution.
When the video feed is captured, the callback function init is launched. It initializes this library:
function init(){
WEBARROCKSOBJECT.init({
canvasId: 'debugWebARRocksObjectCanvas',
video: DOMVIDEO,
callbackReady: function(errLabel){
if (errLabel){
alert('An error happens bro: ',errLabel);
} else {
load_neuralNet();
}
}
});
}
The function load_neuralNet loads the neural network model:
function load_neuralNet(){
WEBARROCKSOBJECT.set_NN('../../neuralNets/NN_OBJ4_0.json', function(errLabel){
if (errLabel){
console.log('ERROR: cannot load the neural net', errLabel);
} else {
iterate();
}
}, options);
}
Instead of giving the URL of the neural network, you can also give the parsed JSON object.
The function iterate starts the iteration loop:
function iterate(){
const detectState = WEBARROCKSOBJECT.detect(3);
if (detectState.label){
console.log(detectState.label, 'IS DETECTED YEAH !!!');
}
window.requestAnimationFrame(iterate);
}
The WEBARROCKSOBJECT.init takes a dictionary as argument with these properties:
<video> video: HTML5 video element (can come from the MediaStream API helper). If false, update the source texture from a videoFrameBuffer object provided when calling WEBARROCKSOBJECT.detect(...) (like in WebXR demos),<dict> videoCrop: see Video cropping section for more details<function> callbackReady: callback function launched when ready or if there was an error. Called with the error label or false,<string> canvasId: id of the canvas from which the WebGL context used for deep learning processing will be created,<canvas> canvas: if canvasId is not provided, you can also provide directly the <canvas> element<dict> scanSettings: see Scan settings section for more details<boolean> isDebugRender: Boolean. If true, a debug rendering will be displayed on the <canvas> element. Useful for debugging, but it should be set to false for production because it wastes GPU computing resources,<int> canvasSize: size of the detection canvas in pixels (should be square). Special value -1 keep the canvas size. Default: 512.<boolean> followZRot: only works with neural network models outputing pitch, roll and yaw angles. Crop the input window using the roll of the current detection during the tracking stage,[<float>, <float>] ZRotRange: only works if followZRot = true. Randomize initial rotation angle. Values are in radians. Default: [0,0].The function which triggers the detection is WEBARROCKSOBJECT.detect(<int>nDetectionsPerLoop, <videoFrame>frame, <dictionary>options).
<int> nDetectionPerLoop is the number of consecutive detections proceeded. The higher it is, the faster the detection will be. But it may slow down the whole application if it is too high because the function call will consume too much GPU resources. A value between 3 and 6 is advised. If the value is 0, the number of detection per loop is adaptative between 1 and 6 with an initial value of 3,<videoFrame> frame is used only with WebXR demos (see WebXR integration section). Otherwise set it to null,<dictionary> options is an optional dictionary which can have these properties:
<float> thresholdDetectFactor: a factor applied on the detection thresholds for the detected object. The default value is 1. For example if it equals 0.5, the detection will be 2 times easier.<string> cutShader: Tweak the default shader used to crop the video area. The possible values are:
null: default value, does not apply a filter and keep RGBA channels,IOS: value optimized of IOS devices for WebXR usage only. Copy the red channel into the other color channels and apply a 5 pixels median filtermedian: apply a 3x3 median filter on RGB channels separately,null: default value, does not apply a filter and keep RGBA channels<boolean> isSkipConfirmation: makes detection easier (more sensitive) but can trigger more false posiSelected from shared topics, language and repository description—not editorial ratings.
WebAR-rocks /
Lightweight WebGL and JavaScript library for real time, on device face detection, tracking and landmark estimation. Ideal for web based augmented reality face filters and virtual try on 😎