3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation

Francis Engelmann1,2      Martin Bokeloh2      Alireza Fathi2      Bastian Leibe1      Matthias Nießner3     

1RWTH Aachen University       2Google       3Technical University of Munich

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.




We present 3D-MPA, a method for instance segmentation on 3D point clouds. Given an input point cloud, we propose an object-centric approach where each point votes for its object center. We sample object proposals from the predicted object centers. Then, we learn proposal features from grouped point features that voted for the same object center. A graph convolutional network introduces inter-proposal relations, providing higher-level feature learning in addition to the lower-level point features. Each proposal comprises a semantic label, a set of associated points over which we define a foreground-background mask, an objectness score and aggregation features. Previous works usually perform non-maximum-suppression (NMS) over proposals to obtain the final object detections or semantic instances. However, NMS can discard potentially correct predictions. Instead, our approach keeps all proposals and groups them together based on the learned aggregation features. We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.

Demo

Object semantics: Cabinet Bed Chair Sofa Table Door Window Bookshelf Picture Counter Desk Curtain Refrigerator Bathtub Shower curtain Toilet Sink Other furniture
Object instances: Different colors represent different instances.

Video

Publication

Paper IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.


BibTeX

@inproceedings{Engelmann20CVPR,
  title = {{3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation}},
  author = {Engelmann, Francis and Bokeloh, Martin and Fathi, Alireza and Leibe, Bastian and Nie{\ss}ner, Matthias},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year = {2020}
}