Skip to content

Latest commit

 

History

History
77 lines (64 loc) · 4.66 KB

POSE_README.md

File metadata and controls

77 lines (64 loc) · 4.66 KB

Sapiens-Lite: 2D Human Pose Estimation

Model Zoo

We provide 4 models of varying size. Sapiens-0.3B, Sapiens-0.6B, Sapiens-1B, Sapiens-2B. In general, performance improves with increasing the model size.

BBox Detection

We use an offshelf detector to do top-down pose estimation. Please install, download and set the path appropriately.

  • Install mmdet 
    export SAPIENS_ROOT=/path/to/sapiens
cd $SAPIENS_ROOT/engine; pip install -e .
cd $SAPIENS_ROOT/cv; pip install -e .
cd $SAPIENS_ROOT/det; pip install -e .

You can also skip using a bounding box detector by remove the --det-config and --det-checkpoint from the scripts - in this case the entire image is used as input.

Body: 17 Keypoints

Best for general in-the-wild scenarios with body keypoints only, adhering to the COCO keypoint format.
Please download the models from hugging-face-pose-lite.

Model Checkpoint Path
Sapiens-0.3B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_0.3b/sapiens_0.3b_coco_best_coco_AP_796_$MODE.pt2
Sapiens-0.6B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_0.6b/sapiens_0.6b_coco_best_coco_AP_812_$MODE.pt2
Sapiens-1B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_1b/sapiens_1b_coco_best_coco_AP_821_$MODE.pt2
Sapiens-2B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_2b/sapiens_2b_coco_best_coco_AP_822_$MODE.pt2

Body + Face + Hands + Feet: 133 Keypoints

Offers second-best generalization with body, face, hands, and feet keypoints, following the COCO-WholeBody keypoint format.
Please download the models from hugging-face-pose-lite.

Model Checkpoint Path
Sapiens-0.3B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_0.3b/sapiens_0.3b_coco_wholebody_best_coco_wholebody_AP_620_$MODE.pt2
Sapiens-0.6B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_0.6b/sapiens_0.6b_coco_wholebody_best_coco_wholebody_AP_695_$MODE.pt2
Sapiens-1B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_1b/sapiens_1b_coco_wholebody_best_coco_wholebody_AP_727_$MODE.pt2
Sapiens-2B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_2b/sapiens_2b_coco_wholebody_best_coco_wholebody_AP_745_$MODE.pt2

Body + Dense Face + Hands + Feet: 308 Keypoints

The highest number of keypoints predictor. Detailed 274 face keypoints. Following the Sociopticon keypoint format.
Please download the models from hugging-face.

Model Checkpoint Path
Sapiens-0.3B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_0.3b/sapiens_0.3b_goliath_best_goliath_AP_573_$MODE.pt2
Sapiens-0.6B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_0.6b/sapiens_0.6b_goliath_best_goliath_AP_609_$MODE.pt2
Sapiens-1B $SAPIENS_LITE_CHECKPOINT_ROOT/pose/checkpoints/sapiens_1b/sapiens_1b_goliath_best_goliath_AP_639_$MODE.pt2

Inference Guide

  • Navigate to your script directory: 
      cd $SAPIENS_LITE_ROOT/scripts/demo/[torchscript,bfloat16]
  • For 17 keypoints estimation (uncomment your model config line for inference): 
    ./pose_keypoints17.sh
  • For 133 keypoints estimation (uncomment your model config line for inference): 
    ./pose_keypoints133.sh
  • For 308 keypoints estimation (uncomment your model config line for inference): 
    ./pose_keypoints308.sh

Define INPUT for your image directory and OUTPUT for results. Visualization and keypoints in JSON format are saved to OUTPUT.
Customize LINE_THICKNESS, RADIUS, and KPT_THRES as needed. Adjust BATCH_SIZE, JOBS_PER_GPU, TOTAL_GPUS and VALID_GPU_IDS for multi-GPU configurations.

Keypoints 17 Keypoints 133 Keypoints 308