Runpei Dong

Xi'an Jiaotong University

Email:  runpei dot dong at gmail dot com

Runpei Dong 董润沛

I am a final-year MS student in Computer Science at Xi’an Jiaotong University, under the supervision of Prof. Andrew Chi-Chih Yao. I collaborate closely with Prof. Kaisheng Ma and Prof. Li Yi at IIIS, Tsinghua University. I am also a research intern of the Foundation Model Group at Megvii Research (Face++), where I work with Prof. Xiangyu Zhang and Dr. Zheng Ge.

My recent research focuses on several areas, including Generative Modeling, Embodied AI (2D/3D/4D), Self-Supervised Learning, and Multi/Cross-Modal Representation Learning. My goal is to develop Vision-Language Foundation Models and embodied robots that can learn world knowledge and effectively interact with humans. I enjoy working on interesting and cool things that fascinate me. I am enthusiastic about contributing to the research community and I support Slow Science.

2024 Fall I’m going to get my MS degree in 06/2024 and I’m currently looking forward to opportunities of PhD positions for 2024 fall.

Recent News

  • New! [2023/11] Honorably selected as NeurIPS 2023 Top Reviewer.
  • New! [2023/09] One paper accepted to NeurIPS 2023.
  • New! [2023/09] Introduce multimodal LLM: DreamLLM
  • [2023/09] Invited talk on multimodal foundation models at Tsinghua University and Xi’an Jiaotong University.
  • [2023/04] I will be attending ICLR 2023 in person, see you in Rwanda!
  • [2023/04] One paper accepted to ICML 2023.
  • [2023/03] Invited talk on 3D representation learning at Tsinghua University.
  • [2023/03] One paper accepted to CVPR 2023.
  • [2023/03] One paper accepted to ICASSP 2023.
  • [2023/01] One paper accepted to ICLR 2023.
  • [2022/09] I join Megvii (Foundation Model Group) as a research intern.
  • [2022/06] One paper accepted to ECCV 2022 as Oral presentation.
  • [2021/12] One paper accepted to ICML 2022 as Spotlight presentation.
  • [2021/10] One paper accepted to BMVC 2021.
  • [2021/06] One paper accepted to ISCA 2021.

Selected Publications

*: equal contribution, the full list can be found here or Google Scholar. Representative papers are highlighted.

DreamLLM: Synergistic Multimodal Comprehension and Creation
Runpei Dong*, Chunrui Han*, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, Li Yi
Project | PDF | Code (coming soon) | BibTex

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning
Liang Zhao*, En Yu*, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, Xiangyu Zhang
PDF | Demo | Code (coming soon) | BibTex

VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation
Zekun Qi*, Muzhou Yu*, Runpei Dong, Kaisheng Ma
NeurIPS 2023 | PDF | Code | BibTex

CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
Junbo Zhang, Runpei Dong, Kaisheng Ma
OpenSUN 3D @ ICCV 2023 | PDF | Code (coming soon) | BibTex

Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining
Zekun Qi*, Runpei Dong*, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, Li Yi
ICML 2023 | PDF | Code | OpenReview | Slides | BibTex

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
Runpei Dong, Zekun Qi, Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, Kaisheng Ma
ICLR 2023 | PDF | Code | OpenReview | Slides | BibTex

PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detection
Linfeng Zhang*, Runpei Dong*, Hung-Shuo Tai, Kaisheng Ma
CVPR 2023 | PDF | Code | BibTex

Contrastive Deep Supervision
Linfeng Zhang, Xin Chen, Junbo Zhang, Runpei Dong, Kaisheng Ma

ECCV 2022 | PDF | Code | Slides | BibTex
Oral Presentation (2.7%)

Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks
Runpei Dong*, Zhanhong Tan*, Mengdi Wu, Linfeng Zhang, Kaisheng Ma
ICML 2022 | PDF | Code | Slides | BibTex
Spotlight Presentation

Professional Services

Programme Committee Members or Conference Reviewers

*: Outstanding Reviewer or Top Reviewer.

  • International Conference on Machine Learning (ICML) 2022
  • International Conference on Learning Representations (ICLR) 2024
  • Advances in Neural Information Processing Systems (NeurIPS) 2022, 2023*
  • Computer Vision and Pattern Recognition (CVPR) 2023, 2024
  • European Conference on Computer Vision (ECCV) 2022
  • International Conference on Computer Vision (ICCV) 2023
  • International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 2023, 2024

Journal Reviewers