CLA-NeRF

Category-Level Articulated Neural Radiance Field


ICRA 2022
Wei-Cheng Tseng, Hung-Ju Liao, Lin Yen-Chen, Min Sun


Overview

Image

We propose CLA-NeRF -- a Category-Level Articulated Neural Radiance Field that can perform view synthesis, part segmentation, and articulated pose estimation. CLA-NeRF is trained at the object category level using no CAD models and no depth, but a set of RGB images with ground truth camera poses and part segments. During inference, it only takes a few RGB views (i.e., few-shot) of an unseen 3D object instance within the known category to infer the object part segmentation and the neural radiance field. Given an articulated pose as input, CLA-NeRF can perform articulation-aware volume rendering to generate the corresponding RGB image at any camera pose. Moreover, the articulated pose of an object can be estimated via inverse rendering. In our experiments, we evaluate the framework across five categories on both synthetic and real-world data. In all cases, our method shows realistic deformation results and accurate articulated pose estimation. We believe that both few-shot articulated object rendering and articulated pose estimation open doors for robots to perceive and interact with unseen articulated objects.




Method

Image

(a) Our framework retrieves features from two instance as the condition of NeRF model and predicts color c, density σ and segmentation s. The volume rendering is applied to predict rendered results. (b) We calculate the deformation matrix based on the articulated pose. Then, we deform the sampled rays with the deformation matrix. Finally, the deformed visual image is rendered using our learned framework. (c) The articulated pose is estimated via inversely minimizing Lcolor.




Real-World Results

Articulated View Systhesis: We directly test our model on real-world images without finetuning. Here, we visulaize novel view synthesis, part segmentation, and deformarion results with predicted joint attributes.

Image

Articulated Pose Estimation We show overlaid images of the rendered and observed images during the optimization of the articulated pose. These examples show that CLA-NeRF is able to recover real setting.

Image



Synthetic Results

Articulated View Systhesis: We directly test our model on holdout objects generated with CAD models. Here, we visulaize novel view synthesis, part segmentation, and deformarion results with predicted joint attributes.

Image



Resource and Citation

Video   Paper   Code (comming soon)   Dataset (comming soon)  
  @inproceedings{icra_tseng,
      author = {Wei-Cheng Tseng, Hung-Ju Liao, Lin Yen-Chen, Min Sun},
      title = {CLA-NeRF: Category-Level Articulated Neural Radiance Field},
      journal = {ICRA},
      year = {2022}
  }