Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes

Authors: Chao Sun, Zhedong Zheng, Xiaohan Wang, Mingliang Xu, Yi Yang

Published in IEEE Transactions on Multimedia (TMM), 2022

Recommended citation: Chao Sun, Zhedong Zheng, Xiaohan Wang, Mingliang Xu, Yi Yang, "Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes." IEEE Transactions on Multimedia, 2022. DOI: 10.1109/TMM.2022.3206664
Download PDF: https://zdzheng.xyz/files/2022/TMM_3D_Pre_Training.pdf
中文解读: https://zhuanlan.zhihu.com/p/623476158

Code is available at: https://github.com/cyysc1998/3d-pretraining

Abstract: The manual annotation for large-scale point clouds costs a lot of time and is usually unavailable in harsh real-world scenarios. Inspired by the great success of the pre-training and fine-tuning paradigm in both vision and language tasks, we argue that pre-training is one potential solution for obtaining a scalable model to 3D point cloud downstream tasks as well. In this paper, we, therefore, explore a new self-supervised learning method, called Mixing and Disentangling (MD), for 3D point cloud representation learning. As the name implies, we mix two input shapes and demand the model learning to separate the inputs from the mixed shape. We leverage this reconstruction task as the pretext optimization objective for self-supervised learning. There are two primary advantages: 1) Compared to prevailing image datasets, eg, ImageNet, point cloud datasets are de facto small. The mixing process can provide a much larger online training sample pool. 2) On the other hand, the disentangling process motivates the model to mine the geometric prior knowledge, eg, key points. To verify the effectiveness of the proposed pretext task, we build one baseline network, which is composed of one encoder and one decoder. During pre-training, we mix two original shapes and obtain the geometry-aware embedding from the encoder, then an instance-adaptive decoder is applied to recover the original shapes from the embedding. Albeit simple, the pre-trained encoder can capture the key points of an unseen point cloud and surpasses the encoder trained from scratch on downstream tasks. The proposed method has improved the empirical performance on both ModelNet-40 and ShapeNet-Part datasets in terms of point cloud classification and segmentation tasks. We further conduct ablation studies to explore the effect of each component and verify the generalization of our proposed strategy by harnessing different backbones.

@article{sun2022self,
author = "Sun, Chao and Zheng, Zhedong and Wang, Xiaohan and Xu, Mingliang and Yang, Yi",
title = "Self-supervised Point Cloud Representation Learning via Separating Mixed Shapes",
abstract = "The manual annotation for large-scale point clouds costs a lot of time and is usually unavailable in harsh real-world scenarios. Inspired by the great success of the pre-training and fine-tuning paradigm in both vision and language tasks, we argue that pre-training is one potential solution for obtaining a scalable model to 3D point cloud downstream tasks as well. In this paper, we, therefore, explore a new self-supervised learning method, called Mixing and Disentangling (MD), for 3D point cloud representation learning. As the name implies, we mix two input shapes and demand the model learning to separate the inputs from the mixed shape. We leverage this reconstruction task as the pretext optimization objective for self-supervised learning. There are two primary advantages: 1) Compared to prevailing image datasets, eg, ImageNet, point cloud datasets are de facto small. The mixing process can provide a much larger online training sample pool. 2) On the other hand, the disentangling process motivates the model to mine the geometric prior knowledge, eg, key points. To verify the effectiveness of the proposed pretext task, we build one baseline network, which is composed of one encoder and one decoder. During pre-training, we mix two original shapes and obtain the geometry-aware embedding from the encoder, then an instance-adaptive decoder is applied to recover the original shapes from the embedding. Albeit simple, the pre-trained encoder can capture the key points of an unseen point cloud and surpasses the encoder trained from scratch on downstream tasks. The proposed method has improved the empirical performance on both ModelNet-40 and ShapeNet-Part datasets in terms of point cloud classification and segmentation tasks. We further conduct ablation studies to explore the effect of each component and verify the generalization of our proposed strategy by harnessing different backbones.",
journal = "IEEE Transactions on Multimedia",
url = "https://zdzheng.xyz/files/2022/TMM\_3D\_Pre\_Training.pdf",
code = "https://github.com/cyysc1998/3d-pretraining",
blog = "https://zhuanlan.zhihu.com/p/623476158",
doi = "10.1109/TMM.2022.3206664",
year = "2022" }