Francis Engelmann

I am a postdoctoral research fellow at ETH Zurich AI Center working with Prof. Dr. Marc Pollefeys. I am also a visiting researcher with Federico Tombari at Google Zurich. Before joining ETH Zurich, I finished my Ph.D. in Computer Vision, Machine Learning and 3D Scene Understanding in the Computer Vision Group of Prof. Dr. Bastian Leibe at RWTH Aachen University and spent some time at Google X in Munich with Martin Bokeloh and Google Research Zurich with Kostas Rematas, as well as Apple in California.

News

Mar 2024 I will give a talk at the CV4AEC Workshop at CVPR'24 in Seattle.
Feb 2024 SceneFun3D is accepted at CVPR'24 in Seattle, great work with Alexandros Delitzas and Ayca Takmaz.
Jan 2024 ICGNet is accepted at ICRA'24 in Yokohama, great work with René Zurbrügg.
Jan 2024 OpenNeRF and AGILE3D are accepted to ICLR'24 in Vienna.
Jan 2024 I gave a talk on High-Fidelity Open-Vocabulary 3D Scene Understanding at the Mila Robot Learning Seminar.
Dec 2023 The second iteration of our Open-Vocabulary 3D Scene Understanding Workshop will be held in conjunction with CVPR'24 in Seattle.
Dec 2023 Our NeurIPS 2023 paper on open-vacbulary 3D instance segmentation method OpenMask3D was featured in the Computer Vision News.
Nov 2023 I was recognized as top reviewer for NeurIPS 2023.
Oct 2023 Our work LabelMaker was accepted at 3DV'24 in Davos, Switzerland.
Sep 2023 Our open-set 3D instance segmentation OpenMask3D was accepted at NeuRIPS'23 in New Orleans.
Sep 2023 I will give a keynote talk at the ICCV'23 CVAAD workshop.
Aug 2023 I will serve as areas chair (AC) for WACV'24.
Jul 2023 Our work on 3D human segmentation Human3D was accepted at ICCV'23 in Paris.
Jun 2023 Check out our latest work on open-set 3D scene segmentation OpenMask3D.

Previous news

Jun 2023 I will serve as senior program committee (SPC) for AAAI 2024.
Mar 2023 Our workshop OpenSun3D on Open-Vocabulary 3D Scene Understanding will be held in conjunction with ICCV'23 in Paris.
Feb 2023 Our paper RoomFormer is accepted at CVPR'23.
Jan 2023 Our paper Mask3D was accepted at ICRA'23. Check out the online demo.
Dec 2022 I received the ETH Zurich Career Seed Award.
Aug 2022 Our Mask3D approach ranks 1^st on ScanNet, ScanNet200 and 2^nd on STPLS3D.
Aug 2022 One paper accepted at the ECCV 2022 AVVision Workshop.
Jul 2022 I will serve as senior program committee (SPC) for AAAI 2023.
Jul 2022 Our paper Box2Mask on weakly supervised 3D instance segmentation using bounding box annotations is accepted as oral (2.7%) to ECCV 2022 in Tel Aviv.
Jun 2022 I gave an invited talk at the CVPR 2022 ScanNet workshop in New Orleans.

Jun 2022 Our winning entry on 2D floorplan reconstruction from large-scale point clouds was presented at CV4AEC with Ekin Celikkan and Theodora Kontogianni.
Jun 2022 Our model Mix3D won first place on the ScanNet 3D Semantic Segmentaion challenge with Alexey Nekrasov and Jonas Schult.
May 2022 I will co-organize the HBHA workshop on human body, hands, and activities from egocentric and multi-view cameras at ECCV 2022.

Publications

SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes
Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, Francis Engelmann
Proc. Computer Vision and Pattern Recognition (CVPR), 2024.
Paper BibTeX Project

@inproceedings{delitzas2024scenefun3d, 
    author = {Alexandros Delitzas and Ayca Takmaz and Federico Tombari and Robert Sumner and Marc Pollefeys and Francis Engelmann}, 
    title = {SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes}, 
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    year = {2024}
}

Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels
Rui Huang, Songyou Peng, Ayça Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, Francis Engelmann
Paper BibTeX Project Demo

  @article{Huang2023Segment3D,
        author    = {Huang, Rui and Peng, Songyou and Takmaz, Ayca and Tombari, Federico and Pollefeys, Marc and Song, Shiji and Huang, Gao and Engelmann, Francis},
        title     = {Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels},
        journal   = {arXiv},
        year      = {2023}
      }

ICGNet: A Unified Approach for Instance-Centric Grasping
René Zurbrügg, Yifan Liu, Francis Engelmann, Suryansh Kumar, Marco Hutter, Vaishakh Patil, Fisher Yu
Proc. International Conference on Robotics and Automation (ICRA), 2024.
Paper BibTeX

  @article{zurbrugg2024icgnet,
      title     = {{ICGNet: A Unified Approach for Instance-Centric Grasping}},
      author={Zurbr{\"u}gg, Ren{\'e} and Liu, Yifan and Engelmann, Francis and Kumar, Suryansh and Hutter, Marco and Patil, Vaishakh and Yu, Fisher},
      journal   = {Internationl Conference on Robotics and Automation (ICRA)},
        year      = {2024}
      }

OpenMask3D: Open-Vocabulary 3D Instance Segmentation
Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Marc Pollefeys, Federico Tombari, Francis Engelmann
Conference on Neural Information Processing Systems (NeurIPS), 2023.
Paper BibTeX Project Code

@inproceedings{Takmaz2023openmask3d,
    title = {{OpenMask3D: Open-Vocabulary 3D Instance Segmentation}},
    author={Takmaz, Ay{\c{c}}a and Fedele, Elisabetta and Sumner, Robert W and Pollefeys, Marc and Tombari, Federico and Engelmann, Francis},
    booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
    year = {2023}
}

LabelMaker: Automatic Semantic Label Generation from RGB-D Trajectories
Silvan Weder, Hermann Blum, Francis Engelmann Marc Pollefeys
International Conference on 3D Vision (3DV), 2024.
Spotlight
Paper BibTeX Project Code

@inproceedings{Weder2024labelmaker3d,
    title = {{LabelMaker: Automatic Semantic Label Generation from RGB-D Trajectories}},
    author={Weder, Silvan and Blum, Hermann and Engelmann, Francis and Pollefeys, Marc},
    booktitle = {International Conference on 3D Vision (3DV)},
    year = {2024}
}

OpenNeRF: Open-Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views
Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, Federico Tombari
International Conference on Learning Representations (ICLR), 2024.
Paper BibTeX Project Code

@inproceedings{engelmann2023opennerf,
    title = {{OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views}},
    author={Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, Federico Tombari},
    booktitle = {{International Conference on Learning Representations (ICLR)}},
    year = {2024}
}

AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation
Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, Theodora Kontogianni
International Conference on Learning Representations (ICLR), 2024.
Paper BibTeX Project Code

@inproceedings{yue2023agile3d,
    title     = {{AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation}},
    author    = {Yue, Yuanwen and Mahadevan, Sabarinath and Schult, Jonas and Engelmann, Francis and Leibe, Bastian and Schindler, Konrad and Kontogianni, Theodora},
    booktitle = {{International Conference on Learning Representations (ICLR)}},
    year      = {2024}
}

3D Segmentation of Humans in Point Clouds with Synthetic Data
Ayca Takmaz, Jonas Schult, Irem Kaftan, Cafer Mertcan Akcay, Bastian Leibe, Robert Sumner, Francis Engelmann, Siyu Tang
Proc. International Conference on Computer Vision (ICCV), 2023.
Paper BibTeX Project Code

@inproceedings{Takmaz2022human3d,
    title = {{3D Segmentation of Humans in Point Clouds with Synthetic Data}},
    author = {Ayca Takmaz, Jonas Schult, Irem Kaftan, Cafer Mertcan Akcay, Bastian Leibe, Robert Sumner, Francis Engelmann and Siyu Tang},
    booktitle = {{arXiv}},
    year = {2022}
}

Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries
Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, Francis Engelmann
Proc. Computer Vision and Pattern Recognition (CVPR), 2023.
Paper BibTeX Project Code

@inproceedings{Yue2022floorformer,
    title = {{Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries}},
    author = {Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, Francis Engelmann},
    booktitle = {IEEE Computer Vision and Pattern Recognition Conference (CVPR)},
    year = {2023}
}

Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe
Proc. International Conference on Robotics and Automation (ICRA), 2023.
1st Place ScanNet Challenge
Paper BibTeX Project Code Demo

@inproceedings{Schult2022mask3d,
    title = {{Mask3D: Mask Transformer for 3D Semantic Instance Segmentation}},
    author = {Schult, Jonas and Engelmann, Francis and Hermans, Alexander and Litany, Or and Tang, Siyu and Leibe, Bastian},
    booktitle = {Internationl Conference on Robotics and Automation (ICRA)},
    year = {2022}
}

Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll
European Conference on Computer Vision (ECCV), 2022.
Oral Presentation
Paper BibTeX Project Video Code

@inproceedings{Chibane2022box2mask,
  title = {Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes},
  author = {Chibane, Julian and Engelmann, Francis and Tran, Tuan Anh and Pons-Moll, Gerard},
  booktitle = {{European Conference on Computer Vision (ECCV)}},
  year = {2022}
}

4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation
Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, Bastian Leibe
European Conference on Computer Vision Workshops (ECCVW), 2022.
Paper BibTeX Code

@inproceedings{Kreuzberg22ECCVW,
  title = {4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation},
  author = {Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, Bastian Leibe},
  booktitle = {{European Conference on Computer Vision Workshops (ECCVW)}},
  year = {2022}
}

Mix3D: Out-of-Context Data Augmentation for 3D Scenes
Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, Francis Engelmann
International Conference on 3D Vision (3DV), 2021.
Oral Presentation, 1st Place ScanNet Challenge
Paper BibTeX Demo Project Code

@inproceedings{Nekrasov213DV,
  title     = {{Mix3D: Out-of-Context Data Augmentation for 3D Scenes}},
  author    = {Nekrasov, Alexey and Schult, Jonas and Litany, Or and Leibe, Bastian and Engelmann, Francis},
  booktitle = {{International Conference on 3D Vision (3DV)}},
  year      = {2021}
}

From Points to Multi-Object 3D Reconstruction
Francis Engelmann, Konstantinos Rematas, Bastian Leibe, Vittorio Ferrari
Proc. Computer Vision and Pattern Recognition (CVPR), 2021.
Paper Poster BibTeX Project Code

@inproceedings{Engelmann21CVPR,
  title     = {{From Points to Multi-Object 3D Reconstruction}},
  author    = {Engelmann, Francis and Rematas, Konstantinos and Leibe, Bastian and Ferrari, Vittorio},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year      = {2021}
}

3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner
Proc. Computer Vision and Pattern Recognition (CVPR), 2020.
Paper BibTeX Project Video

@inproceedings{Engelmann20CVPR,
  title     = {{3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation}},
  author    = {Engelmann, Francis and Bokeloh, Martin and Fathi, Alireza and Leibe, Bastian and Nie{\ss}ner, Matthias},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year      = {2020}
}

DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe
Proc. Computer Vision and Pattern Recognition (CVPR), 2020.
Oral Presentation
Paper BibTeX Project Code

@inproceedings{Schult20CVPR,
  title     = {{DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes}},
  author    = {Jonas Schult and Francis Engelmann and Theodora Kontogianni and Bastian Leibe},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year      = {2020}
}

Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds
Francis Engelmann, Theodora Kontogianni, Bastian Leibe
Proc. International Conference on Robotics and Automation (ICRA), 2020.
Paper BibTeX Poster Project Video

@inproceedings{Engelmann20ICRA,
  author    = {Engelmann, Francis and  Kontogianni, Theodora and Leibe, Bastian},
  title     = {{Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds}},
  booktitle = {{International Conference on Robotics and Automation (ICRA)}},
  year      = {2020}
}

3D-BEVIS: Bird’s-Eye-View Instance Segmentation
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe
Proc. German Conference on Pattern and Recognition (GCPR), 2019.
YRF DAGM Best Master's Thesis Award 2019
Paper BibTeX

@inproceedings{ElichGCPR19,
  title     = {{3D-BEVIS: Birds-Eye-View Instance Segmentation}},
  author    = {Elich, Cathrin and Engelmann, Francis and Schult, Jonas and Kontogianni, Theodora and Leibe, Bastian},
  booktitle = {{German Conference on Pattern Recognition (GCPR)}},
  year      = {2019}
}

Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds
Francis Engelmann, Theodora Kontogianni, Jonas Schult, Bastian Leibe
Proc. European Conference on Computer Vision Workshops (ECCVW), 2018.
Paper BibTeX

@inproceedings{Engelmann18ECCVW,
  author    = {Francis Engelmann and Theodora Kontogianni and Jonas Schult and Bastian Leibe},
  title     = {Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds},
  booktitle = {{European Conference on Computer Vision Workshops (ECCVW)}},
  year      = {2018}
}

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Francis Engelmann*, Theodora Kontogianni*, Alexander Hermans, Bastian Leibe
Proc. Internationcal Conference on Computer Vision Workshops (ICCVW), 2017.
Oral Presentation
Paper BibTeX Poster Project Video Code

@inproceedings{3dsemseg_ICCVW17,
  author    = {Francis Engelmann and Theodora Kontogianni and Alexander Hermans and Bastian Leibe},
  title     = {Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds},
  booktitle = {{International Conference on Computer Vision Workshops (ICCVW)},
  year      = {2017}
}

Keyframe-Based Visual-Inertial Online SLAM with Relocalization
Anton Kasyanov, Francis Engelmann, Jörg Stückler, Bastian Leibe
Proc. International Conference on Intelligent Robots and Systems (IROS), 2017.
Paper BibTeX

@inproceedings{Kasyanov17IROS,
  title     = {{Keyframe-Based Visual-Inertial Online SLAM with Relocalization}},
  author    = {Anton Kasyanov and Francis Engelmann and J\"org St\"uckler and Bastian Leibe},
  booktitle = {{International Conference on Intelligent Robots and Systems (IROS)}},
  year      = {2017}
}

SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction
Francis Engelmann, Jörg Stückler, Bastian Leibe
Proc. Winter Conference on Applications of Computer Vision (WACV), 2017.
Paper BibTeX Poster Supplementary Material Project

@inproceedings{Engelmann17WACV,
  author    = {Francis Engelmann and J{\"{o}}rg St{\"{u}}ckler and Bastian Leibe},
  title     = {{SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction}},
  booktitle = {{Winter Conference on Applications of Computer Vision (WACV)}},
  year      = {2017}
}

Joint Object Pose Estimation and Shape Reconstruction in Urban Street Scenes Using 3D Shape Priors
Francis Engelmann, Jörg Stückler, Bastian Leibe
Proc. German Conference on Pattern Recognition (GCPR), 2016.
Oral Presentation
Paper BibTeX Supplementary Material Project Video Code

@inproceedings{Engelmann16GCPR,
  title     = {{Joint Object Pose Estimation and Shape Reconstruction in Urban Street Scenes Using 3D Shape Priors}},
  author    = {Francis Engelmann and J\"org St\"uckler and Bastian Leibe},
  booktitle = {{German Conference on Pattern Recognition (GCPR)}},
  year      = {2016}
}

Projects

This is the online demo of Mask3D our state-of-the-art 3D semantic instance segmentation method. You can upload your own 3D scan or reconstruction (.ply) and segment the scene into the semantic classes of ScanNet, including ScanNet 200.
Link
PyViz3D is a simple web-based visualizer for 3D point clouds and other primitives such as meshes, polylines, bounding boxes and labels. It is useful for quick protyping and remote development.
Code Documentation

Virtual KITTI 3D: Extension of the Virtual Kitti dataset including 3D cloud representations and refined semantic labels.
Code
Graph Convolutional Nets: Implementation of Kipf et al. graph convolutional networks.
Code
Fast Voxel Traversal: Implementation of the Fast Voxel Traversal algorithm.
Code
Spectrum Analyzer: Some tools to visualize audio input using OpenAL and FFTW.
Code
MCMC-MH: Implementation of MCMC-MH and sudoku solver using MCMC.
Code