Francis Engelmann

I am a postdoc at Stanford University with Prof. Leonidas Guibas and Prof. Jeannette Bohg. Previously, I was a postdoctoral research fellow at ETH Zurich AI Center working with Prof. Dr. Marc Pollefeys, and a visiting researcher with Federico Tombari at Google Zurich. Before joining ETH Zurich, I finished my Ph.D. in Computer Vision, Machine Learning and 3D Scene Understanding in the Computer Vision Group of Prof. Dr. Bastian Leibe at RWTH Aachen University and spent some time at Google X in Munich with Martin Bokeloh and Google Research Zurich with Kostas Rematas, as well as Apple in California.

News

Oct 2024 We have 2 papers accated at WACV'25.
Sept 2024 I will serve as areas chair (AC) for CVPR'25.
July 2024 We have 4 papers accepted at ECCV'24 in Milan, among them Segment3D!
July 2024 I will serve as areas chair (AC) for 3DV'25 and WACV'25.
May 2024 I was recognized as outstanding reviewer for CVPR 2024 (top 2%).
Apr 2024 The third iteration of our Open-Vocabulary 3D Scene Understanding Workshop will be held at ECCV'24 in Milan.
May 2024 Our Spot-Compose received a 🏆 Best Paper Award at the ICRA'24 MOMA.v2 workshop.

Mar 2024 I will give a talk at the CV4AEC Workshop at CVPR'24 in Seattle.
Feb 2024 SceneFun3D is accepted at CVPR'24 as 🏆 Oral in Seattle, great work with Alex Delitzas and Ayca Takmaz.
Jan 2024 ICGNet is accepted at ICRA'24 in Yokohama, great work with René Zurbrügg.
Jan 2024 OpenNeRF and AGILE3D are accepted to ICLR'24 in Vienna.
Jan 2024 I gave a talk on High-Fidelity Open-Vocabulary 3D Scene Understanding at the Mila Robot Learning Seminar.

Previous news

Dec 2023 The second iteration of our Open-Vocabulary 3D Scene Understanding Workshop will be held in conjunction with CVPR'24 in Seattle.
Dec 2023 Our NeurIPS 2023 paper on open-vacbulary 3D instance segmentation method OpenMask3D was featured in the Computer Vision News.
Nov 2023 I was recognized as top reviewer for NeurIPS 2023.
Oct 2023 Our work LabelMaker was accepted at 3DV'24 in Davos, Switzerland.
Sep 2023 Our open-set 3D instance segmentation OpenMask3D was accepted at NeuRIPS'23 in New Orleans.
Sep 2023 I will give a keynote talk at the ICCV'23 CVAAD workshop.
Aug 2023 I will serve as areas chair (AC) for WACV'24.
Jul 2023 Our work on 3D human segmentation Human3D was accepted at ICCV'23 in Paris.
Jun 2023 Check out our latest work on open-set 3D scene segmentation OpenMask3D.
Jun 2023 I will serve as senior program committee (SPC) for AAAI 2024.
Mar 2023 Our workshop OpenSun3D on Open-Vocabulary 3D Scene Understanding will be held in conjunction with ICCV'23 in Paris.
Feb 2023 Our paper RoomFormer is accepted at CVPR'23.
Jan 2023 Our paper Mask3D was accepted at ICRA'23. Check out the online demo.
Dec 2022 I received the ETH Zurich Career Seed Award.
Aug 2022 Our Mask3D approach ranks 1^st on ScanNet, ScanNet200 and 2^nd on STPLS3D.
Aug 2022 One paper accepted at the ECCV 2022 AVVision Workshop.
Jul 2022 I will serve as senior program committee (SPC) for AAAI 2023.
Jul 2022 Our paper Box2Mask on weakly supervised 3D instance segmentation using bounding box annotations is accepted as oral (2.7%) to ECCV 2022 in Tel Aviv.
Jun 2022 I gave an invited talk at the CVPR 2022 ScanNet workshop in New Orleans.

Jun 2022 Our winning entry on 2D floorplan reconstruction from large-scale point clouds was presented at CV4AEC with Ekin Celikkan and Theodora Kontogianni.
Jun 2022 Our model Mix3D won first place on the ScanNet 3D Semantic Segmentaion challenge with Alexey Nekrasov and Jonas Schult.
May 2022 I will co-organize the HBHA workshop on human body, hands, and activities from egocentric and multi-view cameras at ECCV 2022.

Publications

P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoising
Mathias Vogel, Keisuke Tateno, Marc Pollefeys, Federico Tombari, Marie-Julie Rakotosaona, Francis Engelmann
European Conference on Computer Vision (ECCV), 2024.
Paper BibTeX Project Code

@inproceedings{delitzas2024scenefun3d, 
    author = {Vogel, Mathias and Tateno, Keisuke and Pollefeys, Marc and Tombari, Federico and Rakotosaona, Marie-Julie and Engelmann, Francis}, 
    title = {P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoising}, 
    booktitle = {{European Conference on Computer Vision (ECCV)}},
    year = {2024}
}

Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels
Rui Huang, Songyou Peng, Ayça Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, Francis Engelmann
European Conference on Computer Vision (ECCV), 2024.
Paper BibTeX Project Demo

  @inproceedings{Huang2023Segment3D,
        author    = {Huang, Rui and Peng, Songyou and Takmaz, Ayca and Tombari, Federico and Pollefeys, Marc and Song, Shiji and Huang, Gao and Engelmann, Francis},
        title     = {{Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels}},
        booktitle = {{European Conference on Computer Vision (ECCV)}},
        year      = {2024}
      }

SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs
Yang Miao, Francis Engelmann, Olga Vysotska, Federico Tombari, Marc Pollefeys, Dániel Béla Baráth
European Conference on Computer Vision (ECCV), 2024.
Paper BibTeX Project Code

@inproceedings{miao2024scenegraphloc,
  author    = {Miao, Yang and Engelmann, Francis and Vysotska, Olga and Tombari, Federico and Pollefeys, Marc and Baráth, Dániel Béla},
  title     = {{SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs}},
  booktitle = {{European Conference on Computer Vision (ECCV)}},
  year      = {2024}
}

FiT3D: Improving 2D Feature Representations by 3D-Aware Fine-Tuning
Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, Jan Eric Lenssen
European Conference on Computer Vision (ECCV), 2024.
Paper BibTeX Project Code

@inproceedings{miao2024scenegraphloc,
  author    = {Yue, Yuanwen and Das, Anurag and Engelmann, Francis and Tang, Siyu and Lenssen, Jan Eric},
  title     = {{Improving 2D Feature Representations by 3D-Aware Fine-Tuning}},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2024}
}

SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes
Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Marc Pollefeys, Francis Engelmann
Proc. Computer Vision and Pattern Recognition (CVPR), 2024.
🏆 Oral Presentation
Paper BibTeX Project

@inproceedings{delitzas2024scenefun3d, 
    author = {Alexandros Delitzas and Ayca Takmaz and Federico Tombari and Robert Sumner and Marc Pollefeys and Francis Engelmann}, 
    title = {SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes}, 
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    year = {2024}
}

Spot-Compose: A Framework for Open-Vocabulary Object Retrieval and Drawer Manipulation in Point Clouds
Oliver Lemke, Zuria Bauer, René Zurbrügg, Marc Pollefeys, Francis Engelmann^*, Hermann Blum^*
🏆 Best Paper Award (ICRA MOMA.v2 workshop)
Proc. International Conference on Robotics and Automation Workshops (ICRAW), 2024.
Paper BibTeX Project

@article{zurbrugg2024icgnet,
  title     = {{Spot-Compose: A Framework for Open-Vocabulary Object Retrieval and Drawer Manipulation in Point Clouds}},
  author={Oliver Lemke and Zuria Bauer and Ren{\'e} Zurbr{\"u}gg and Marc Pollefeys and Francis Engelmann and Hermann Blum},
  journal   = {Internationl Conference on Robotics and Automation Workshops (ICRAW)},
  year      = {2024}
}

ICGNet: A Unified Approach for Instance-Centric Grasping
René Zurbrügg, Yifan Liu, Francis Engelmann, Suryansh Kumar, Marco Hutter, Vaishakh Patil, Fisher Yu
Proc. International Conference on Robotics and Automation (ICRA), 2024.
Paper BibTeX Project

@inproceedings{zurbrugg2024icgnet,
  title = {{ICGNet: A Unified Approach for Instance-Centric Grasping}},
  author = {Zurbr{\"u}gg, Ren{\'e} and Liu, Yifan and Engelmann, Francis and Kumar, Suryansh and Hutter, Marco and Patil, Vaishakh and Yu, Fisher},
  booktitle = {Internationl Conference on Robotics and Automation (ICRA)},
  year = {2024}
}

OpenMask3D: Open-Vocabulary 3D Instance Segmentation
Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Marc Pollefeys, Federico Tombari, Francis Engelmann
Conference on Neural Information Processing Systems (NeurIPS), 2023.
Paper BibTeX Project Code

@inproceedings{Takmaz2023openmask3d,
    title = {{OpenMask3D: Open-Vocabulary 3D Instance Segmentation}},
    author={Takmaz, Ay{\c{c}}a and Fedele, Elisabetta and Sumner, Robert W and Pollefeys, Marc and Tombari, Federico and Engelmann, Francis},
    booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
    year = {2023}
}

LabelMaker: Automatic Semantic Label Generation from RGB-D Trajectories
Silvan Weder, Hermann Blum, Francis Engelmann Marc Pollefeys
International Conference on 3D Vision (3DV), 2024.
🏆 Spotlight Presentation
Paper BibTeX Project Code

@inproceedings{Weder2024labelmaker3d,
    title = {{LabelMaker: Automatic Semantic Label Generation from RGB-D Trajectories}},
    author={Weder, Silvan and Blum, Hermann and Engelmann, Francis and Pollefeys, Marc},
    booktitle = {International Conference on 3D Vision (3DV)},
    year = {2024}
}

OpenNeRF: Open-Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views
Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, Federico Tombari
International Conference on Learning Representations (ICLR), 2024.
Paper BibTeX Project Code Demo

@inproceedings{engelmann2023opennerf,
    title = {{OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views}},
    author={Francis Engelmann, Fabian Manhardt, Michael Niemeyer, Keisuke Tateno, Marc Pollefeys, Federico Tombari},
    booktitle = {{International Conference on Learning Representations (ICLR)}},
    year = {2024}
}

AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation
Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, Theodora Kontogianni
International Conference on Learning Representations (ICLR), 2024.
Paper BibTeX Project Code

@inproceedings{yue2023agile3d,
    title     = {{AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation}},
    author    = {Yue, Yuanwen and Mahadevan, Sabarinath and Schult, Jonas and Engelmann, Francis and Leibe, Bastian and Schindler, Konrad and Kontogianni, Theodora},
    booktitle = {{International Conference on Learning Representations (ICLR)}},
    year      = {2024}
}

3D Segmentation of Humans in Point Clouds with Synthetic Data
Ayca Takmaz, Jonas Schult, Irem Kaftan, Cafer Mertcan Akcay, Bastian Leibe, Robert Sumner, Francis Engelmann, Siyu Tang
Proc. International Conference on Computer Vision (ICCV), 2023.
Paper BibTeX Project Code

@inproceedings{Takmaz2022human3d,
    title = {{3D Segmentation of Humans in Point Clouds with Synthetic Data}},
    author = {Ayca Takmaz, Jonas Schult, Irem Kaftan, Cafer Mertcan Akcay, Bastian Leibe, Robert Sumner, Francis Engelmann and Siyu Tang},
    booktitle = {{arXiv}},
    year = {2022}
}

Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries
Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, Francis Engelmann
Proc. Computer Vision and Pattern Recognition (CVPR), 2023.
Paper BibTeX Project Code

@inproceedings{Yue2022floorformer,
    title = {{Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries}},
    author = {Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, Francis Engelmann},
    booktitle = {IEEE Computer Vision and Pattern Recognition Conference (CVPR)},
    year = {2023}
}

Mask3D: Mask Transformer for 3D Semantic Instance Segmentation
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe
Proc. International Conference on Robotics and Automation (ICRA), 2023.
1st Place ScanNet Challenge
Paper BibTeX Project Code Demo

@inproceedings{Schult2022mask3d,
    title = {{Mask3D: Mask Transformer for 3D Semantic Instance Segmentation}},
    author = {Schult, Jonas and Engelmann, Francis and Hermans, Alexander and Litany, Or and Tang, Siyu and Leibe, Bastian},
    booktitle = {Internationl Conference on Robotics and Automation (ICRA)},
    year = {2022}
}

Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll
European Conference on Computer Vision (ECCV), 2022.
🏆 Oral Presentation
Paper BibTeX Project Video Code

@inproceedings{Chibane2022box2mask,
  title = {Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes},
  author = {Chibane, Julian and Engelmann, Francis and Tran, Tuan Anh and Pons-Moll, Gerard},
  booktitle = {{European Conference on Computer Vision (ECCV)}},
  year = {2022}
}

4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation
Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, Bastian Leibe
European Conference on Computer Vision Workshops (ECCVW), 2022.
Paper BibTeX Code

@inproceedings{Kreuzberg22ECCVW,
  title = {4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation},
  author = {Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, Bastian Leibe},
  booktitle = {{European Conference on Computer Vision Workshops (ECCVW)}},
  year = {2022}
}

Mix3D: Out-of-Context Data Augmentation for 3D Scenes
Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, Francis Engelmann
International Conference on 3D Vision (3DV), 2021.
🏆 Oral Presentation, 1st Place ScanNet Challenge
Paper BibTeX Demo Project Code

@inproceedings{Nekrasov213DV,
  title     = {{Mix3D: Out-of-Context Data Augmentation for 3D Scenes}},
  author    = {Nekrasov, Alexey and Schult, Jonas and Litany, Or and Leibe, Bastian and Engelmann, Francis},
  booktitle = {{International Conference on 3D Vision (3DV)}},
  year      = {2021}
}

From Points to Multi-Object 3D Reconstruction
Francis Engelmann, Konstantinos Rematas, Bastian Leibe, Vittorio Ferrari
Proc. Computer Vision and Pattern Recognition (CVPR), 2021.
Paper Poster BibTeX Project Code

@inproceedings{Engelmann21CVPR,
  title     = {{From Points to Multi-Object 3D Reconstruction}},
  author    = {Engelmann, Francis and Rematas, Konstantinos and Leibe, Bastian and Ferrari, Vittorio},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year      = {2021}
}

3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner
Proc. Computer Vision and Pattern Recognition (CVPR), 2020.
Paper BibTeX Project Video

@inproceedings{Engelmann20CVPR,
  title     = {{3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation}},
  author    = {Engelmann, Francis and Bokeloh, Martin and Fathi, Alireza and Leibe, Bastian and Nie{\ss}ner, Matthias},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year      = {2020}
}

DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe
Proc. Computer Vision and Pattern Recognition (CVPR), 2020.
🏆 Oral Presentation
Paper BibTeX Project Code

@inproceedings{Schult20CVPR,
  title     = {{DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes}},
  author    = {Jonas Schult and Francis Engelmann and Theodora Kontogianni and Bastian Leibe},
  booktitle = {{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}},
  year      = {2020}
}

Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds
Francis Engelmann, Theodora Kontogianni, Bastian Leibe
Proc. International Conference on Robotics and Automation (ICRA), 2020.
Paper BibTeX Poster Project Video

@inproceedings{Engelmann20ICRA,
  author    = {Engelmann, Francis and  Kontogianni, Theodora and Leibe, Bastian},
  title     = {{Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds}},
  booktitle = {{International Conference on Robotics and Automation (ICRA)}},
  year      = {2020}
}

3D-BEVIS: Bird’s-Eye-View Instance Segmentation
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe
Proc. German Conference on Pattern and Recognition (GCPR), 2019.
🏆 YRF DAGM Best Master's Thesis Award 2019
Paper BibTeX

@inproceedings{ElichGCPR19,
  title     = {{3D-BEVIS: Birds-Eye-View Instance Segmentation}},
  author    = {Elich, Cathrin and Engelmann, Francis and Schult, Jonas and Kontogianni, Theodora and Leibe, Bastian},
  booktitle = {{German Conference on Pattern Recognition (GCPR)}},
  year      = {2019}
}

Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds
Francis Engelmann, Theodora Kontogianni, Jonas Schult, Bastian Leibe
Proc. European Conference on Computer Vision Workshops (ECCVW), 2018.
Paper BibTeX

@inproceedings{Engelmann18ECCVW,
  author    = {Francis Engelmann and Theodora Kontogianni and Jonas Schult and Bastian Leibe},
  title     = {Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds},
  booktitle = {{European Conference on Computer Vision Workshops (ECCVW)}},
  year      = {2018}
}

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Francis Engelmann*, Theodora Kontogianni*, Alexander Hermans, Bastian Leibe
Proc. Internationcal Conference on Computer Vision Workshops (ICCVW), 2017.
🏆 Oral Presentation
Paper BibTeX Poster Project Video Code

@inproceedings{3dsemseg_ICCVW17,
  author    = {Francis Engelmann and Theodora Kontogianni and Alexander Hermans and Bastian Leibe},
  title     = {Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds},
  booktitle = {{International Conference on Computer Vision Workshops (ICCVW)},
  year      = {2017}
}

Keyframe-Based Visual-Inertial Online SLAM with Relocalization
Anton Kasyanov, Francis Engelmann, Jörg Stückler, Bastian Leibe
Proc. International Conference on Intelligent Robots and Systems (IROS), 2017.
Paper BibTeX

@inproceedings{Kasyanov17IROS,
  title     = {{Keyframe-Based Visual-Inertial Online SLAM with Relocalization}},
  author    = {Anton Kasyanov and Francis Engelmann and J\"org St\"uckler and Bastian Leibe},
  booktitle = {{International Conference on Intelligent Robots and Systems (IROS)}},
  year      = {2017}
}

SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction
Francis Engelmann, Jörg Stückler, Bastian Leibe
Proc. Winter Conference on Applications of Computer Vision (WACV), 2017.
Paper BibTeX Poster Supplementary Material Project

@inproceedings{Engelmann17WACV,
  author    = {Francis Engelmann and J{\"{o}}rg St{\"{u}}ckler and Bastian Leibe},
  title     = {{SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction}},
  booktitle = {{Winter Conference on Applications of Computer Vision (WACV)}},
  year      = {2017}
}

Joint Object Pose Estimation and Shape Reconstruction in Urban Street Scenes Using 3D Shape Priors
Francis Engelmann, Jörg Stückler, Bastian Leibe
Proc. German Conference on Pattern Recognition (GCPR), 2016.
🏆 Oral Presentation
Paper BibTeX Supplementary Material Project Video Code

@inproceedings{Engelmann16GCPR,
  title     = {{Joint Object Pose Estimation and Shape Reconstruction in Urban Street Scenes Using 3D Shape Priors}},
  author    = {Francis Engelmann and J\"org St\"uckler and Bastian Leibe},
  booktitle = {{German Conference on Pattern Recognition (GCPR)}},
  year      = {2016}
}

Projects

This is the online demo of Mask3D our state-of-the-art 3D semantic instance segmentation method. You can upload your own 3D scan or reconstruction (.ply) and segment the scene into the semantic classes of ScanNet, including ScanNet 200.
Link
PyViz3D is a simple web-based visualizer for 3D point clouds and other primitives such as meshes, polylines, bounding boxes and labels. It is useful for quick protyping and remote development.
GitHub
FabScan is a do-it-yourself, open-source 3D laser scanner that I developed a long time ago as my Bachelor thesis at the RWTH Aachen FabLab. It consists of a webcam, line-laser, microcontroller, stepper-engines, and a turn-table. The project was featured on Thingiverse and is still maintained by Mario Lukas at fabscan.org.

Virtual KITTI 3D: Extension of the Virtual Kitti dataset including 3D cloud representations and refined semantic labels.
Code
Graph Convolutional Nets: Implementation of Kipf et al. graph convolutional networks.
Code
Fast Voxel Traversal: Implementation of the Fast Voxel Traversal algorithm.
Code
Spectrum Analyzer: Some tools to visualize audio input using OpenAL and FFTW.
Code
MCMC-MH: Implementation of MCMC-MH and sudoku solver using MCMC.
Code