3D Point Cloud Data Augmentation via Scene Representation Network

Try novel view synthesis method to do point cloud data augmentation.

We design a 3D point cloud augmentation based on a novel view synthesis method, SRN. The 3D point cloud is a set of data points in 3D space, consisting of coordinates in world space and RGB color information. However, the precision is limited if there are insufficient points to describe the objects. Moreover, the amount of point cloud training samples is also limited, leading to the poor diversity of data. This motivates us to design a data augmentation method for the point cloud. As our base model, given extrinsic and intrinsic matrix, the SRN model can estimate the world coordinate and get the corresponding depth and RGB color information, which can easily be transferred to point cloud form. After the SRN model is trained, we can render new images of unseen views by changing the extrinsic matrix and consequently realize the point cloud augmentation. To verify whether the augmented point cloud is good enough to describe the 3D objects, we use PointNet as our evaluation model to evaluate the quality of the augmentation.

In order to apply our method on unseen objects and do interpolation with other objects, we have replace the instance object id embedding with images features from ResNet encoder. Also, for intra-class and inter-class interpolation, we mix different objects in the latent space to generate augmented objects. This is a way of increasing diversity of our data.

We have done many experiments, and our method is successful in ModelNet10. But we have observe the limitation of novel view synthesis method on non-texture data when we use ModelNet40. To make up the color information that non-texture data is lack of, there must be extra information be included when training. This may be an interesting future work.

More information of our implementation:
Report:
Code:
Slides: