Tokenizing the World into Gaussians
Encoding raw sensor streams into a compact vocabulary of Gaussian tokens — one representation for reconstruction, registration, and occupancy.
We design algorithms that tokenize dense sensor feeds (like 3D point clouds) into a compact vocabulary of Gaussian primitives — continuous, generative world tokens that both perception and planning can query directly.
Robots typically fragment their understanding of the world across disparate pipelines: one map for reconstruction, another for pose estimation, a third for occupancy inference. We replace them with a single representation built by tokenizing the environment — encoding raw sensor streams into a compact vocabulary of Gaussian primitives rather than voxels or raw points.
Each Gaussian acts as a token of the scene, and a self-organizing principle from information-theoretic learning decides how many tokens a given scene demands — allocating detail where the geometry is complex and merging redundant observations elsewhere. A spatial hash map for rapid submap extraction keeps this tokenization incremental and real-time, improving computational speed by an order of magnitude over prior GMM-based mapping while retaining a favorable accuracy-to-size tradeoff.
Because the tokens are generative, the same set reconstructs high-resolution surfaces, registers new observations, and answers occupancy queries — with no separate pipeline per task. The representation also exposes analytical structure that planners can query directly, from microsecond distance and collision-probability estimates against ellipsoid approximations to gradients for navigation. Our GIRA framework packages these capabilities open-source, with GPU-accelerated learning that fits the Gaussian tokens 10–100x faster than CPU implementations.
Compactness is what carries these tokens into the field. Because a handful of Gaussians summarizes dense sensor data in a fraction of the memory, size-weight-and-power constrained aerial robots can map at high fidelity and share their tokenized models over low-bandwidth links — which we have demonstrated in information-theoretic exploration and in cave surveys conducted in total darkness.
Work in this area
