Registering clothes from 4D scans with vertex-accurate correspondence is challenging, yet important for dynamic appearance modeling and physics parameter estimation from real-world data. However, previous methods either rely on texture information, which is not always reliable, or achieve only coarse-level alignment. In this work, we present a novel approach to enabling accurate surface registration of texture-less clothes with large deformation. Our key idea is to effectively leverage a shape prior learned from pre-captured clothing using diffusion models. We also propose a multi-stage guidance scheme based on learned functional maps, which stabilizes registration for large-scale deformation even when they vary significantly from training data. Using high-fidelity real captured clothes, our experiments show that the proposed approach based on diffusion models generalizes better than surface registration with VAE or PCA-based priors, outperforming both optimization-based and learning-based non-rigid registration methods for both interpolation and extrapolation tests.
Diffusion model for cloth deformation. In the forward process, we gradually add noise to the UV displacement map to acquire an isotropic Gaussian distribution. To sample from the learned data distribution, we recover the clean UV displacement map by gradually denoising the corrupted UV displacement map.
The manifold guidance maximizes the log-likelihood of every diffusion state given the observation. We use a multi-stage posterior sampling process, where the early stage of the denoising process is guided by a learning-based coarse registration approach, and the later stage is refined with spatial proximity guidance.
@article{guo2023diffusion,
title={Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration},
author={Guo, Jingfan and Prada, Fabian and Xiang, Donglai and Romero, Javier and
Wu, Chenglei and Park, Hyun Soo and Shiratori, Takaaki and Saito, Shunsuke},
year={2023},
journal={arXiv preprint arXiv:2311.05828}
}