Loading repository data…
robot-perception-group / repository
This repository contains the code of AirPose, our multi-view fusion network for Human Pose and Shape Estimation method
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
Check the teaser video
This repository contains the code of AirPose a novel markerless 3D human motion capture (MoCap) system for unstructured, outdoor environments that uses a team of autonomous unmanned aerialvehicles (UAVs) with on-board RGB cameras and computation.
Please clone the repository with the following
git clone https://github.com/robot-perception-group/AirPose.git --recursive
Data can be freely accessed here. Please download the data, and untar it whenever necessary. Content details are following:
The code was tested using Python 3.8.
SMPLX submodule in this repo is a modified version of the official SMLX implementation. Download the SMPLX model weights from here and run the following
# from the download location
unzip models_smplx_v1_1.zip -d models_smplx
unzip models_smplx/models/smplx/smplx_npz.zip -d used_models
rm models_smplx -r
Then copy the content of used_models (just created, with SMPLX_{MALE,FEMALE,NEUTRAL}.npz files) folder into .
Selected from shared topics, language and repository description—not editorial ratings.
adriacabeza /
:helicopter: This repository contains a project that combines DJI Tello drone and Deep Learning (Tiny Yolo).
48/100 healthyour_path/AirPose/copenet/src/copenet/data/smplx/models/smplxYou need to register before being able to download the weights.
Now, you may want to create a virtual environment. Please be sure your pip is updated.
Install the necessary requirements with pip install -r requirements.txt. If you don't have a cuda compatible device, change the device to cpu in copenet_real/src/copenet_real/config.py and copenet/src/copenet/config.py.
In those files (copenet_real/src/copenet_real/config.py and copenet/src/copenet/config.py) change LOCAL_DATA_DIR to /global/path/AirPose/copenet/src/copenet/data".
Check out this link to fix the runtime error RuntimeError: Subtraction, the - operator, with a bool tensor is not supported due to the Torchgeometry package.
Install the copenet and copenet_real packages in this repo
pip install -e copenet
pip install -e copenet_real
Download the head and hands indices files form here and place them in AirPose/copenet/src/copenet/data/smplx (MANO_SMPLX_vertex_ids.pkl and SMPL-X__FLAME_vertex_ids.npy).
The data to be used is copenet_synthetic_data.tar.gz (here)
To run the code of this repository you first need to preprocess the data using
# from AirPose folder
python copenet/src/copenet/scripts/prepare_aerialpeople_dataset.py /absolute/path/copenet_synthetic
And code can be run by the following (from AirPose/copenet folder):
python src/copenet/copenet_trainer.py --name=test_name --version=test_version --model=muhmr --datapath=/absolute/path/copenet_synthetic --log_dir=path/location/ --copenet_home=/absolute/path/AirPose/copenet --optional-params...
The datapath is the location of the training data.
--model specify the model type between [hmr, muhmr, copnet_singleview, copenet_twoview] which corresponds to the Baseline, Baseline+multi-view, Baseline+Fullcam and AirPose respectively.
Logs will be saved in $log_dir/$name/$version/
optional-params is to be substituted with the copenet_trainer available params as weights, lr..
For model type [muhmr, copenet_twoview].
cd AirPose/copenet_real
python src/copenet_real/scripts/copenet_synth_res_compile.py "model type" "checkpoint Path" "/path to the dataset"
For model type [hmr, copenet_singleview], the provided checkpoint is trained with an older pytorch lightning version (<=1.2). If you want to use them, install pytorch-lightning<=1.2.
We provide the precalculated outputs on the syntehtic data using these checkpoints.
To generate the metrics, run
cd AirPose/copenet_real
python src/copenet_real/scripts/hmr_synth_res_compile.py "model type" "precalculated results directory Path" "/path to the dataset" "your_path/AirPose/copenet/src/copenet/data/smplx/models/smplx"
The data to be used is copenet_dji_real_data.tar.gz(here).
Install the human body prior from here and download its pretrained weights (version 2) from here. Set the vposer_weights variable in the .../AirPose/copenet_real/src/copenet_real/config.py file to the absolute path of the downloaded weights (e.g. /home/user/Downloads/V02_05). If you do NOT have a GPU please change human_body_prior/tools/model_loader.py line 68 from state_dict = torch.load(trained_weigths_fname)['state_dict'] to state_dict = torch.load(trained_weigths_fname, map_location=torch.device('cpu'))['state_dict']
Note: for the hmr (Baseline) model pytorch-lightning<=1.2 is required. You might have to recheck requirements, or reinstall the requirements you can find in the main folder of this repo.
Code can be run by the following (from AirPose/copenet_real/ folder)
python src/copenet_real/copenet_trainer.py --name=test_name --version=test_version --model=hmr --datapath=path/location --log_dir=path/location/ --resume_from_checkpoint=/path/to/checkpoint --copenet_home=/absolute/path/AirPose/copenet --optional-params...
The datapath is the location of the training data.
--model specify the model type between [hmr, copenet_twoview] which corresponds to the Baseline, AirPose respectively.
The --resume_from_checkpoint is path to the pretrained checkpoint on the synthetic data.
Install graphviz dependency with pip install graphviz in the same virtual environment.
Following code will generate the plots comparing the results of the baseline method, AirPose and AirPose+ on the real data.
This can be run from AirPose folder.
python copenet_real_data/scripts/bundle_adj.py "path_to_the_real_dataset" \\
"path_to_the_SMPLX_neutral_npz_file" \\
"path_to_vposer_folder" \\
"path_to_the_hmr_checkpoint_directory" \\
"path_to_the_airpose_precalculated_res_on_realdata_pkl" \\
"path_to_the_SMPLX_to_j14_mapping_pkl_file" \\
"type_of_data(train/test)"
Note that:
SMPLX_neutral_npz_file should be in your_path/AirPose/copenet/src/copenet/data/smplx/models/smplx.vposer_folder should be in the vposer_weights folder that you downloaded to finetune on the real dataprecalculated_res_on_realdata_pkl can be found within the same archive you downloaded above. More on how to compute them yourself below.SMPLX_to_j14_pkl can be found here.The evaluation code above needs precalculated results on the real data which are provided with the dataset. If you want to calculate them yourself, run the following code and save the variable outputs in a pkl file when a breakpoint is hit. The pkl files provided with the data are generated in the same way.
For AirPose
python copenet_real/src/copenet_real/scripts/copenet_real_res_compile.py "checkpoint Path" "/path to the dataset"
For Baseline
python copenet_real/src/copenet_real/scripts/hmr_real_res_compile.py "checkpoint Path" "/path to the dataset"
To this end you need to install ros-{melodic,noetic} in your pc (Ubuntu 18.04-20.04).
Please follow the instructions that you can find here
After that you need to install the following dependencies:
sudo add-apt-repository ppa:joseluisblancoc/mrpt-stable
Navigate to your catkin_ws folder (e.g. AirPose/catkin_ws) and run:
touch src/aircap/packages/optional/basler_image_capture/Grab/CATKIN_IGNORE
touch src/aircap/packages/optional/ptgrey_image_capture/Grab/CATKIN_IGNORE
Firstly, checkout the AirPose branch ros-melodic.
Be sure to update the submodule (first command).
git submodule update
sudo apt install libmrpt-dev mrpt-apps
cd /your/path/AirPose/catkin_ws
touch src/aircap/packages/3rdparty/mrpt_bridge/CATKIN_IGNORE
touch src/aircap/packages/3rdparty/pose_cov_ops/CATKIN_IGNORE
sudo apt install -y ros-melodic-octomap-msgs ros-melodic-cv-camera ros-melodic-marker-msgs ros-melodic-mrpt-msgs ros-melodic-octomap-ros ros-melodic-mrpt-bridge ros-melodic-mrpt1
sudo apt install libmrpt-poses-dev libmrpt-obs-dev libmrpt-graphs-dev libmrpt-maps-dev libmrpt-slam-dev -y
sudo apt install -y ros-noetic-octomap-msgs ros-noetic-cv-camera ros-noetic-marker-msgs ros-noetic-mrpt-msgs ros-noetic-octomap-ros ros-noetic-mrpt2
Then you can run catkin_make from the catkin_ws folder to build the whole workspace.
To run the client-server architecture you need:
To test the code you can do the following.
In separated terminals (with the workspace sourced) run:
roscorerosparam set use_sim_time trueroslaunch airpose_client one_robot.launch host:=127.0.0.1 port:=9901 feedback_topic:=info img_topic:=camera/image_raw camera_info_topic:=camera/info robotID:=1 reproject:=false groundtruth:=true, with host you can change the server IP address, port must correspond, feedback_topic must contain the ROI and is of type neural_network_detector::NeuralNetworkFeedback, robotID should be either 1 or 2, reproject is used to avoid a reprojection to different intrisics parameters and groundtruth:=true is used to provide {min_x, max_x, min_y, max_y} in the ROI message (description below)