From Skin to Skeleton:

Towards Biomechanically Accurate 3D Digital Humans

Marilyn Keller, Keenon Werling, Soyong Shin, Scott Delp, Sergi Pujades, C. Karen Liu, Michael J. Black

SIGGRAPH ASIA 2023
[Paper] [Supplementary] [Code]

Abstract

Great progress has been made in estimating 3D human pose and shape from images and video by training neural networks to directly regress the parameters of parametric human models like SMPL. However, existing body models have simplified kinematic structures that do not correspond to the true joint locations and articulations in the human skeletal system, limiting their potential use in biomechanics. On the other hand, methods for estimating biomechanically accurate skeletal motion typically rely on complex motion capture systems and expensive optimization methods. What is needed is a parametric 3D human model with a biomechanically accurate skeletal structure that can be easily posed. To that end, we develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton. To enable this, we need training data of skeletons inside SMPL meshes in diverse poses. 

We build such a dataset by optimizing biomechanically accurate skeletons inside SMPL meshes from AMASS sequences. We then learn a regressor from SMPL mesh vertices to the optimized joint locations and bone rotations. Finally, we re-parametrize the SMPL mesh with the new kinematic parameters. The resulting SKEL model is animatable like SMPL but with fewer, and biomechanically-realistic, degrees of freedom. We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods. By fitting SKEL to SMPL meshes we are able to “upgrade" existing human pose and shape datasets to include biomechanical parameters. SKEL provides a new tool to enable biomechanics in the wild, while also providing vision and graphics researchers with a better constrained and more realistic model of human articulation. The model, code, and data are available for research at https://skel.is.tue.mpg.de.

Video

Code

We provide the following code for this project:

1) SKEL loader

This code will let you load the SKEL model. SKEL is a body model, that given shape parameters β and pose parameters q yields a posed body mesh and skeleton mesh. SKEL's body mesh is the same as SMPL but contrary to SMPL, SKEL has more anatomically sound degrees of freedom, i.e knee flexion, arm supination, etc.

2) SMPL2AddBiomechanics: Fitting an OpenSim model to SMPL sequences

Support code for running AddBiomechanics on SMPL sequences .

3) SKEL Viewer for BSM, BIOAMASS and SKEL :

Viewer based on ait-viewer and nimblephysics that lets you visualize the different models we use in the paper and motion sequences.

Acknowledgments

Acknowledgments and Disclosures. Michael J. Black (MJB) has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB is a consultant for Meshcapade, his research was performed solely at MPI. Soyong Shin performed this work while an intern at the Max Planck Institute for Intelligent Systems. Marilyn Keller was supported by the International Max Planck Research School for Intelligent Systems. Sergi Pujades’ work was funded by the ANR SEMBA project. This work uses aitviewer for visualization [Kaufmann et al. 2022].

Citation

@article{keller2023skel,
  title = {From Skin to Skeleton: Towards Biomechanically Accurate {3D} Digital Humans},
  author = {Keller, Marilyn and Werling, Keenon and Shin, Soyong and Delp, Scott and Pujades, Sergi and Liu, C. Karen and Black, Michael J.},
  journal = {ACM Transaction on Graphics (ToG)},
  volume = {42},
  number = {6},
  pages = {253:1-253:15},
  month = dec,
  year = {2023},
  doi = {https://doi.org/10.1145/3618381},
  month_numeric = {12}
}