An End-to-End Transformer Model for 3D Object Detection

We propose 3DETR, an end-to-end Transformer based object detection model for 3D point clouds. Compared to existing detection methods that employ a number of 3D specific inductive biases, 3DETR requires minimal modifications to the vanilla Transformer block. Specifically, we find that a standard Transformer with non-parametric queries and Fourier positional embeddings is competitive with specialized architectures that employ libraries of 3D specific operators with hand-tuned hyperparameters. Nevertheless, 3DETR is conceptually simple and easy to implement, enabling further improvements by incorporating 3D domain knowledge. Through extensive experiments, we show 3DETR outperforms the well-established and highly optimized VoteNet baselines on the challenging ScanNetV2 dataset by 9.5%. Furthermore, we show 3DETR is applicable to 3D tasks beyond detection, and can serve as a building block for future research.

People

Ishan Misra

Rohit Girdhar

Armand Joulin

Paper

		I. Misra, R. Girdhar and A. Joulin An End-to-End Transformer Model for 3D Object Detection IEEE/CVF International Conference on Computer Vision (ICCV), 2021 (Oral Presentation) [arXiv] [code/models] [BibTex]

@inproceedings{misra2021-3detr,
  title={{An End-to-End Transformer Model for 3D Object Detection}},
  author={Misra, Ishan and Girdhar, Rohit and Joulin, Armand},
  booktitle={ICCV},
  year={2021},
}

Results

3DETR achieves comparable or better performance to these improved baselines despite having fewer hand-coded 3D or detection specific decisions.

Detection results for scenes from the val set of the SUN RGB-D dataset. 3DETR does not use color information (used only for visualization) and predicts boxes from point clouds. 3DETR can detect objects even with single-view depth scans and predicts amodal boxes e.g., the full extent of the bed (top left) including objects missing in the ground truth (top right).

Acknowledgements

We thank Zaiwei Zhang for helpful discussions and Laurens van der Maaten for feedback on the paper.