Nvidia's new AI model turns static images into 3D models that can be used to train self-driving cars

Nvidia has added depth to shallow graphics again.After converting 2D images into 3D scenes, models and videos, the company turned its focus to editing.According to foreign media reports, on June 22, local time, the GPU giant announced a new AI method that can turn static pictures into 3D objects that creators can easily modify.Nvidia researchers have developed an inverse rendering product, Nvidia 3D MoMa, that allows users to reconstruct a series of still photos into a 3D computer model of an object or even a scene.The main benefit of this workflow over more traditional photogrammetry is the ability to output clean 3D models that can also be imported and exported through 3D games and vision engines.

Nvidia’s new AI model (Image credit: Nvidia)

While other photogrammetry programs are reported to be able to turn 3D images into 3D models, Nvidia's 3D MoMa technology goes a step further, by creating mesh, material and lighting information for objects and exporting them to be compatible with existing 3D graphics and models. Die tool-compatible formats, and all can be completed in a relatively short period of time.Nvidia says 3D MoMa can generate triangular mesh models using a single Nvidia Tensor Core GPU in under an hour.

Nvidia's technology could help train robots and self-driving cars, or be used to more easily create 3D scenes for games and animations.

"By treating each part of the inverse rendering problem as a GPU-accelerated differentiable component, the NVIDIA 3D MoMa rendering tool uses modern AI machines and the raw computing power of NVIDIA GPUs to rapidly generate 3D rendering," said David Luebke, vice president of graphics research at NVIDIA. object, and creators can import, edit, and extend that 3D object without limitations within existing tools."

Nvidia says the technology is one of the first models to combine ultrafast neural network training with fast rendering.Nvidia previously mentioned in a blog post that Instant NeRF can learn a high-resolution 3D scene in seconds and render an image of that scene in milliseconds.3D MoMa is more than 1000 times faster than NeRF.

What is NeRF?

According to Nvidia, NeRF uses neural networks to represent and render 3D scenes based on 2D images.Gathering data into NeRF is a bit like a red carpet photographer trying to capture what a star is wearing from every angle, the neural network also needs to take dozens of photos from around the scene and from multiple camera positions.

In scenes with people or other moving elements, the faster these shots are captured, the better.If there is too much movement during 2D image capture, the AI-generated 3D scene will be blurry.So NeRF basically makes up for this, training a small neural network to reconstruct the scene bypredictingthe color of light radiating in any direction from any point in 3D space.The technique even works around occluders, i.e. when objects are obscured by obstacles such as pillars in some images.

The technology can be used to train robots and self-driving cars to understand the size and shape of real-world objects by capturing 2D images or video clips.It can also be used in architecture and entertainment to quickly generate digital representations of real-world events that creators can modify and build upon.In addition to NeRF, Nvidia researchers are exploring how this input encoding technique can be used to accelerate a variety of AI applications, such as reinforcement learning, language translation, and general-purpose deep learning algorithms.