3D Gaussian Splatting

Website Link: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting

3DGS reconstructs environments from many photos and renders views of them; Not generation but rather reconstruction

3D Gaussian Splatting represents a scene with many 3D Gaussians, if the scene contains N Gaussians then the gaussian has:

• center
• covariance
• opacity
• color or appearance parameters

Lot of these clouds arranged properly approximate surfaces and volumetric detail

Pipeline §

1. Structure from Motion
   • Photos of static scenes, SFM estimates camera poses and gives a sparse point cloud
2. Turn each sparse point into a Gaussian
   • Simple blobs and during trained they become stretched rotated, split, cloned or removed
3. Project each 3D Gaussian into the image
   • Every 3D Gaussian becomes a 2D elliptical splat
4. Sort splats by depth and alpha-bend them
   • Group work into 16x16 titles, sort splat instances by tile and depth, then accumulate color front-to-back
5. Backpropagate image error and adapt density
   • Renders training image, compares to ground truth w/ L1 Loss and D-SSIM loss, updates Gaussian Parameters with gradient descent. Sometimes clones useful Gaussians, splits oversized ones, and prunes transparent ones