3DGS · Neural rendering · Vulkan
A structured path from 3D Gaussian Splatting foundations to original contributions in surgical visualization. Four phases — each building toward NVIDIA-relevant skills and novel research territory.
Understand 3DGS from first principles, get first scene rendering on DGX Spark, integrate into VertexNova.
Gaussian ellipsoids, tile-based rasterization, alpha blending, spherical harmonics. How it maps to existing OIT knowledge from surgical robotics visualization work.
Short technical overview of gsplat, render pipeline stages, and practical API usage on DGX Spark.
gsplat 1.5.3 from source on GB10 Blackwell. SM 12.0 arch, --no-build-isolation, PyTorch 2.7 compatibility.
Download pre-trained garden scene. View in nerfstudio. Understand what the .ply file contains and how the rasterizer works.
Create vne3dgs module. Parse .ply Gaussian attributes. First OpenGL render pass — blobs on screen before beauty.
Understand the full neural rendering landscape — NeRF vs 3DGS, 4DGS for dynamic scenes, training from own images.
Speed, quality, editability, storage tradeoffs. When to use each. From a real-time rendering background, not an ML background.
Capture photos → COLMAP → sparse point cloud → 3DGS training. First surgical-style scene reconstruction.
vertexnova/nerfstudio-spark container. Train splatfacto on custom data. Measure training time on GB10 vs reported benchmarks.
4D-GS achieves 82 FPS at 800x800 for temporal sequences. Apply to surgical anatomy — moving instruments, tissue deformation.
Close the key gaps identified for NVIDIA Omniverse and neural graphics roles. Public demos that signal readiness.
Ship the Vulkan renderer. Profile with NVIDIA Nsight Graphics. Write a technical blog post. This is the single biggest NVIDIA signal.
VK_KHR_ray_tracing_pipeline. Simple path tracer. Shows understanding of RTX hardware — core to NVIDIA's stack.
Parallel mesh processing pipeline using CUDA. Connects C++ rendering background with GPU compute fluency.
OpenUSD added official GS support in April 2026. Import a USD scene into VertexNova or write a USD-to-splat pipeline. Direct Omniverse bridge.
Apply 3DGS to surgical visualization — a domain nobody has seriously explored. Potential paper or patent.
Endoscopic footage or CT multi-view → COLMAP → 3DGS training → real-time Vulkan render. Unique combination of OR rendering experience + 3DGS.
Extract meshes from trained Gaussian scenes. Bridge between traditional mesh-based surgical planning and neural rendering.
Document findings. Target a venue like MICCAI or IEEE VR. Patent potential given surgical visualization background and existing co-inventor credits.
NVIDIA team alignment
Which NVIDIA teams each phase maps to.
3DGS + gsplat + DGX Spark work maps directly to NVIDIA's Neural Graphics team hiring.
nerfstudio + OpenUSD + simulation background from robotic surgery platforms.
Vulkan ray tracing + VertexNova multi-backend engine + CUDA compute pipeline.
Medical 3DGS is novel territory. Surgical visualization + neural rendering is a unique combination nobody else has.