Tracky Mouse
Control your computer by moving your head.
Tracky Mouse is a desktop application providing hands-free universal computer access.
It's also embeddable in web applications as a JavaScript library. See the API docs.
Features include:
By building it as a desktop app and an embeddable web UI, users can try it out right away in their browser, and then install the desktop app for full computer control.
✨👉 Try out the Demo! 👈✨
Install Desktop App
- ⬇️ Download for Windows (.exe installer) and run the installer.
- If you get a SmartScreen warning, click "More info" and then "Run anyway".
- ⬇️ Download for Windows (.msix)
- After downloading the
.msix file, right click on it, select Properties, go to the Digital Signatures tab, select the embedded signature from the list, click Details, click View Certificate, and install the certificate. Then double click the .msix file. Then click "Restart as administrator" to restart the installer (you do not need to restart your computer, don't worry). Say Yes to allow App Installer to make changes to the device. Then click Install.
- ⬇️ Download for Linux (.AppImage)
- You may need to make it executable first by right-clicking the file, selecting "Properties", going to the "Permissions" tab, and checking "Allow this file to run as a program" or "Is executable" or similar (depending on your file manager). Then you can double-click it to run it. See How to run an AppImage for more details.
- ⬇️ Download for Linux (.deb)
- ⬇️ Download for Linux (.rpm)
Pre-built binaries are not yet available for macOS, due to a couple issues: camera permissions, and the more powerful clicking modes not clicking properly.
You can still run the app on macOS, if you follow the Development Setup instructions.
Usage Guide
These instructions apply to using the desktop app or the web UI.
Set up your camera and environment:
- Make sure to have good lighting on your face. Placing a lamp beside your monitor can help a lot!
- Back-lighting can be problematic, especially if your head moves in and out from occluding the light during use, but also due to glare.
- Your webcam should be centered in front of your head, with your head fully visible when sitting comfortably.
Start using Tracky Mouse:
- Press the "Start" button to start moving the mouse and clicking. You can also use the keyboard shortcut F9. When using the desktop app, this shortcut works even when the app is not in focus.
- Dwell in one spot to click. To avoid clicking, you have to keep moving your head, or pause the app with F9.
- (Desktop app only) Try changing Clicking mode to Wink to click or Open mouth to click in the settings.
- With Wink to click:
- Close your right eye to left click
- Close your left eye to right click.
- If this feels backwards, you can enable "Swap mouse buttons"
- With Open mouth to click:
- Close your right eye and open your mouth to middle click
- Close your left eye and open your mouth to right click
- Open your mouth with both eyes open to left click.
General usage tips:
- Adjust the settings until you can comfortably move the mouse to the edges of the screen with some accuracy.
- Advice for point tracking mode:
- If the mouse cursor feels off-center, you can recalibrate by simply moving your head past where the cursor meets the edge of the screen.
- Note that not only rotating your head, but translating your head (moving it left/right, up/down, or forward/backward) moves the mouse.
- One nuance to this is, if the camera is positioned above your head, leaning forward generally moves the pointer down, whereas if the camera is below your head, leaning forward generally moves the pointer up.
- Using the Tilt influence slider: This setting lets you blend between using 2D point tracking and 3D head tilt.
- At 0% it will use only point tracking. This moves the cursor according to visible movement of 2D points on your face, so it responds to both head rotation and translation. It's very accurate and responsive, but can get out of sync with your head orientation over time, requiring you to recenter by pushing the cursor to the edge of the screen.
- Recommended: high acceleration
- At 100% it will use only head tilt. This uses an estimate of your face's rotation in 3D space, and ignores head translation. Note that this signal is smoothed (as it's very jittery otherwise), so it's not as responsive as point tracking. In this mode you never need to recenter by pushing the cursor to the edge of the screen, but you do need to calibrate it in the Head tilt calibration settings section first.
- Acceleration does not apply, as movement is absolute based on head tilt.
- In between it will behave like an automatic calibration, subtly adjusting the point tracking to match the head tilt. This works by slowing down mouse movement that is moving away from the position targeted based on the head tilt, and (only past 80% on the slider) actively moving towards it.
- Recommended: medium point tracking acceleration; point tracking sensitivity should roughly match head tilt sensitivity.
- Head tilt calibration settings: You can adjust the horizontal and vertical tilt range and offset. This allows the head tilt feature to be used with different camera placements (above or below the screen) and postures, and lets you balance comfort+speed and precision.
Troubleshooting:
- If you have multiple cameras, make sure to select the correct one under Video > Camera source.
- If the camera feed appears black:
- Make sure there is no privacy/dust cover on the camera.
- Ensure there's enough light.
- Check the camera in another application to make sure it's working.
- Resuming from sleep/hibernate can also cause this (see issue #77). Try restarting the app.
- If the camera can't be accessed at all, make sure it's not being used by another application, then click "Allow Camera Access" in the app. Also try unplugging the camera and plugging it in again (if it's an external camera), or restarting your computer.
- On Linux: Installing (and maybe running?)
guvcview can magically fix a webcam not showing up. (source)
- On Windows 11, it's possible to allow multiple apps to access the camera at once.
- Go to Settings > Bluetooth & devices > Camera, select your camera, and in Advanced camera options, enable Allow multiple apps to use camera at the same time. (You'll need to stop any apps currently using the camera first.)
- Auto-focus and auto-brightness can cause head tracking disruptions. Consider disabling auto-focus on your camera, and adjusting focus manually. If you disable auto-brightness, you will have to adjust the brightness regularly as the lighting changes, at least assuming you have any natural light in the room.
- Advanced camera settings can be accessed with Video > Open Camera Settings in the desktop app on Windows.
Integrating with external software
Tracky Mouse comes with a command-line interface (CLI) which can be used to control the desktop app with a voice command system or other external programs. See CLI documentation for usage.
Add to your project
Tracky Mouse is available on npm:
npm install tracky-mouse
Read the API documentation for more information.
License
MIT-licensed, see LICENSE.txt
Changelog
See CHANGELOG.md for project history and API changes.
Why did I make this?
Someone emailed me asking about how they might adjust the UI of JS Paint to work with eye tracking (enlarging the color palette, hiding other UI elements, etc.)
and I decided to do them one better and build it as an official feature, with dwell clicking and everything.
To test these accessibility features properly, I needed a facial mouse, but eye trackers are expensive, so I tried looking for head tracking software, and found eViacam, but... either it didn't work, or at some point it stopped working on my computer.
- eViacam wasn't working on my computer.
- I didn't find there to be very many facial mouse software options out there, especially cross-platform, and I want people to have options.
- I wanted people to be able to try JS Paint's dwell clicking out easily, and an embeddable facial mouse GUI would be great for that.
- I've had some joint pain issues in the past (although also neck pain, which is a bit ironic)
- I think I can push forward the state of the art in facial mouse software.
Software Architecture
This is a monorepo containing packages for the library (core), the desktop app (desktop-app), and the website (website).
I tried npm workspaces, but it doesn't work with Electron Forge packaging. See electron/forge#2306.
Core
The core library uses the following third-party libraries:
- jsfeat for point tracking (using Lucas–Kanade optical flow)
- clmtrackr.js for fast and lightweight but inaccurate face tracking
- facemesh and TensorFlow.js for accurate face tracking (once this loads, it stops using clmtrackr.js)
Some dependencies are versioned with npm and copied into core/lib/ with npm run in-core -- npm run copy-deps
Others are just stored in core/lib/ without npm