报告人：Yonhon Ng 博士 Senior Researcher @ Tencent, Canberra, Australia
报告时间：2024年9月26日（周四）16:00-17:00
报告地点：东南大学九龙湖校区计算机楼513室
报告摘要：3D Content-Generation is at the heart of many computer graphics applications, including video gaming, filmmaking, virtual reality, and augmented reality. This paper proposes a novel approach for automatically generating interactive (i.e., playable) 3D game scenes from users' casual prompts, including hand-drawn sketches and text descriptions. Sketch-based input offers a natural, convenient, and precise way to capture the user's design intention. To circumvent the prohibitive challenge of the lack of large-scale training data for 3D scenes, our method leverages a pre-trained 2D diffusion model to generate images as conceptual guidance. We advocate for the use of isometric projection images to factor out unknown camera poses while simultaneously obtaining the scene layout. Given a generated isometric image, we utilize pre-trained image understanding models to identify scene components and extract the scene layout. Finally, we leverage a procedural generation engine to render the obtained 3D scenes, resulting in a playable 3D game scene that can be seamlessly integrated into a game environment such as Unity or Unreal Engine. Extensive experimental results demonstrate that our method can efficiently generate high-quality 3D game scenes with layouts that closely follow users' intentions.
报告人简介：Dr Yonhon Ng is a Senior Researcher at Tencent XR Vision Labs based in Canberra, Australia. He is the Tech Lead in Generative 3D Scenes for games. He was also the Project Lead for Multi-camera SLAM at XR Vision Labs, where the team won HILTI SLAM Challenge 2023 for both single and multi-session visual-inertial SLAM. Yonhon obtained Outstanding Contributor award in Tencent in 2022 and 2024 (Tech Future). He was a Research Fellow at the Australian National University (ANU) from 2019-2022, and lead multiple industry funded research projects. He graduated from the ANU with PhD in Engineering in 2018, and Bachelor of Engineering with University Medal in 2013. He published in top conferences and journals, such as TPAMI, ICCV, ECCV, RAL, TAC, ICRA, IROS, IFAC, CDC. His research interest covers generative AI, SLAM, non-linear filter and state estimation.
