VTuber-Python-Unity GitHub Details, Stars and Alternatives | OpenRepoFinder
mmmmmm44 / repository
VTuber-Python-Unity
An Implementation of VTuber (Both 3D and Live2D) using Python and Unity. Providing face movement tracking, eye blinking detection, iris detection and tracking and mouth movement tracking using CPU only.
A transparent discovery signal based on current public GitHub metadata.
Recent activity35% weight
72
Community adoption25% weight
53
Maintenance state20% weight
85
License clarity10% weight
100
Project information10% weight
75
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
README preview
VTuber Python Unity Tutorial
An Implementation of VTuber (Both 3D and Live2D) using Python and Unity. Supporting face movement tracking, eye blinking detection, iris detection and tracking and mouth movement tracking using CPU only.
Usage
Live2D Demo
UnityChan 3D Demo
Features
Facial landmarks detection and movement tracking supported by Facemesh by Mediapipe.
Various facial expressions detection including eye blinking, iris and mouth movements.
Running smooth 30 FPS with CPU only for the aformentioned features.
Simple and clean UI for adjusting the sensibility of detection in Unity.
Saveload mechanism to save and load your preferences in Unity.
Including sample (Unity) projects for both 3D and Live2D models.
Support Window and M1 machines.
Detailed and thorough explanation videos (with EN and ZH subtitles) playlist
File Explanation
File
Description
main.py
The main program
facial_landmark.py
The module which is used to detect your face and generate the facial landmarks.
pose_estimator.py
The module which estimates your pose/ orientation of your head based on the landmarks.
stabilizer.py
Implementation of Kalman Filter to stabilize the values.
facial_features.py
Various facial features detection implementation, including blinking, iris detection and mouth movement.
model.txt
The points of the 3D Canonical model used in Mediapipe.
Whole Unity Projects (in packages) and Scripts (old version) for both 3D (UnityChan) and Live2D (Hiyori) models
Background
Using avatars for streaming, content creation and VR gaming has been gaining increasing popularity, especially the boom of Hololive and other related companies active apperances in social media platforms such as YouTube and Twitter. Curious about the technology behind, I create this project after multiple researches.
Existing projects rely on Dlib, which although providing reliable and accurate facial landmark detection, requires decent graphic cards to run. However, implemented with the recent FaceMesh model in Mediapipe, accurate detection and tracking can be run smoothly using CPU only, making running on computers with mediocre graphic cards or laptops with integrated graphic cards possible.
How To Use
Clone this project into your directory
git clone https://github.com/mmmmmm44/VTuber-Python-Unity.git
cd "VTuber-Python-Unity"
Setup
Create An empty Unity 3D Project
Import either the Live2D or UnityChan3D package to your project. The corresponding SDKs have been included already. (Excpet that I removed the voice files of UnityChan in the UnityChan Unity package due to file size limit in github.)
last edit: 16-05-2022
[NEW] In the Game window of Unity, create a 9:16 portrait ratio display mode and save it. Then the UI will return normal.
(Or you can adjust the ratio in the AspectGridLayoutCellSize.cs attached on each of the panel Game Object, namely "Setting Panel" and "TCP Setting Panel".) Tutorial
Run the Scene. Click the Setting button to switch on the TCP server. There should be a "Waiting for connection..." log message showing in the Console of Unity.
Run the following code in terminal
[content in the bracket is optional]
Download the Cubism SDK For Unity from this website and the sample model used (桃瀬ひより) from this website
Create an empty Unity 3D project, and import the Cubism SDK. Unzip the model and drag the whole folder to the Project window of the Unity Project.
Drag the live2D model's prefab into the scene. Run the scene immediately to allow the model to be shown in Scene and Game window.
Adjust the camera's position, background and projection properties. If there are some werid projection problems of the model, changing the projection of the camera from Perspective and Orthographic works for me.
Drag the HiyoriController.cs to the Hiyori GameObject. Adjust the parameters in the inspector
Run the scene.
Run the following code in terminal
[content in the bracket is optional]
Download the UnityChan model from the website. Go to "Data Download", accept the terms and agreements, and select the first one. Unzip the file.
Create an empty Unity 3D Project. Drag the unzipped folder to the Project Window of the project.
Go to UnityChan\Prefabs and Drag the "unitychan" prefab into the scene.
Adjust the camera's position, background and field of view.
Drag the UnityChanControl.cs script onto the prefab. Change the update mode of the Animator attached to "Animate Physics" and the Controller to UnityChanLocomotions. (Crucial) Adjust the variables in the inspector. Disable other attached scripts except AutoBlink and UnityChanController. You may tick the box "is Auto Blink Active" in UnityChanContoller to enable auto blinking (enable AutoBlink script when ticked).
Run the scene first
Run the following code in terminal
[content in the bracket is optional]
python main.py --connect [--debug] [--port PORT]
Enjoy
Make sure you run the Unity Scene first before running the python script
The complete Unity Project with fancy stuffs such as UI system, Save Load Preferences, and custom port UI can be found after importing the Unity Packages provided.
Options (main.py)
-h, --help show this help message and exit
--connect connect to the unity character
--port PORT specify the port of the connection to
unity. Have to be the same as in Unity.
--cam CAM specify the camera number to use
if you have multiple cameras connected
to the computer.
(for cv2.VideoCapture(CAM))
--debug show the raw values of the detection
in the terminal
Examples
Connecting to the Unity Project with default port, showing the sent data.
python main.py --connect --debug
Connecting to the Unity Project with custom port (e.g. 5077), showing the sent data.
python main.py --connect --debug --port 5077
Development Environment
Python 3.8.5
Numpy 1.19.2
OpenCV 4.5.1
Mediapipe 0.8.9.1
Unity 2020.3.12f1
(Later version should be supported as well)
(For Windows, it is recommended to run this project using Anaconda and create a virtual environment before installing such packages.)
The whole project is run on a laptop with Intel Core i5-8250U, with 16GB RAM and integrated graphic card only.
The same project is tested on a M1 Max device, running in rosetta.
