GaussianVLM

Scene-centric 3D Vision-Language Models using
Language-aligned Gaussian Splats
for Embodied Reasoning and Beyond

Anna-Maria Halacheva1, Jan-Nico Zaech1, Xi Wang1, 2, 3, Danda Pani Paudel 1, Luc Van Gool1
1INSAIT, Sofia University "St. Kliment Ohridski" 2ETH Zurich 3TU Munich
Logo 1 Logo 2
Preprint Data (Coming Soon) Code (Coming soon)

TL;DR

We present GaussianVLM, the first 3D VLM operating on Gaussian splats. Each Gaussian in the scene is enriched with language features, forming a dense, scene-centric representation. A novel dual sparsifier reduces ~40k language-augmented Gaussians to just 132 tokens, retaining task-relevant and location-relevant information. This enables open-vocabulary, detector-free reasoning and yields state-of-the-art performance on both scene- and object-centric embodied benchmarks.

Abstract

As multimodal language models advance, their application to 3D scene understanding is a fast-growing frontier, driving the development of 3D Vision-Language Models (VLMs). Current methods show strong dependence on object detectors, introducing processing bottlenecks and limitations in taxonomic flexibility. To address these limitations, we propose a scene-centric 3D VLM for 3D Gaussian splat scenes that employs language- and task-aware scene representations. Our approach directly embeds rich linguistic features into the 3D scene representation by associating language with each Gaussian primitive, achieving early modality alignment. To process the resulting dense representations, we introduce a dual sparsifier that distills them into compact, task-relevant tokens via task-guided and location-guided pathways, producing sparse, task-aware global and local scene tokens. Notably, we present the first Gaussian splatting-based VLM, leveraging photorealistic 3D representations derived from standard RGB images, demonstrating strong generalization: it improves performance of prior 3D VLM (LL3DA) five folds, in out-of-the-domain settings.

🔍 Project Highlights

🗣️ Built on Visuo-Linguistic 3D Maps

Constructs maps of 3D Gaussian splats enriched with language features for vision-language spatial reasoning.

🔍 Object Detector-Free

GaussianVLM does NOT rely on object detectors, ensuring scene-centric representations and open vocabulary support.

đź’ˇ Novel Dual Sparsification

Sparsifiers reduce ~40k tokens to 132, making maps interpretable for LLMs via two sparsifiers - task-guided, and location-aware.

🌎 OOD Generalization

Outperforms prior methods in real-world RGB-derived 3D datasets due to Gaussian splat utilization.

🚀 State-of-the-Art Performance

Outperforms leading baselines across scene- and object-centric tasks, including SQA3D and 3D-LLM embodied benchmarks.

GaussianVLM Architecture

The GaussianVLM architecture takes as input a user-defined task prompt—consisting of a query and an optional spatial location—and a 3D scene represented using Gaussians. A 3D vision module, the SceneSplat Transformer, first predicts per-Gaussian language features across the scene.

These dense language features are then processed by a dual sparsifier. This sparsifier includes two parallel components: (1) a location-guided pathway that selects Gaussians within a spatial Region of Interest (ROI) to produce ROI tokens; and (2) a task-guided pathway that uses cross-attention with task tokens to extract 128 task-selected tokens based on the decoder’s hidden states and dense scene features.

The resulting sparse representation—composed of both ROI and task-selected tokens—is then combined with the task tokens and passed to a multimodal decoder, enabling precise and grounded reasoning over the 3D scene for downstream language-vision tasks.

Quantitative Results

Our evaluation comprises scene-centric [situated QA & embodied tasks] and object-centric tasks.
To assess real-world robustness, we also test generalization to out-of-domain (OOD) data on scenes reconstructed from RGB images, a more realistic input setting compared to traditional point cloud capture methods.

Scene-centric: require comprehensive understanding of entire 3D environments—including spatial layout, agent context, and multi-turn interaction—covering benchmarks like embodied dialogue, planning, and situated question answering. These tasks demand holistic reasoning beyond individual objects.

Object-centric: emphasize detailed reasoning about specific objects through captioning and question answering using localized queries and object annotations.

While our model is primarily designed to excel at scene-centric understanding without relying on explicit object detectors, it remarkably retains strong performance on object-centric benchmarks, matching or surpassing baselines that use dedicated object detection modules.

OOD generalization: We evaluate on ScanNet++ scenes reconstructed from RGB images using an object counting task (in-house dataset). Given a question targeting an object category (e.g., “How many chairs are in the scene?”), the model predicts the correct count. QA pairs are automatically generated from ScanNet++ instance segmentation annotations.

Situated QA, SQA3D Benchmark
Model EM1 C B-4 M R
GPT341.0----
ClipBERT43.3----
SQA3D46.6----
3D-VisTA48.5----
PQ3D47.1----
LEO*47.0124.79.425.548.4
Ours49.4129.617.126.450.2
3D-LLM Embodied Tasks
Model Embodied Dialogue Embodied Planning Scene Captioning
SimCB-4MR SimCB-4MR SimCB-4MR
OPT-1.3B-0.310.235.624.83-0.160.130.243.56-0.00.848.4011.7
OPT-2.7B-0.380.397.386.28-0.100.263.594.35-0.110.006.6012.32
OPT-6.7B-0.250.436.886.16-0.000.283.653.94-0.061.138.9916.96
LLAMA-7B-0.270.507.816.68-0.040.293.534.71-0.200.927.0012.31
LL3DA*48.2145.922.240.936.750.265.17.120.832.266.40.23.019.418.4
Ours72.3270.131.555.748.659.0220.420.344.548.065.80.86.423.521.1
Evaluation on object-centric LL3DA benchmarks
ScanRefer ScanQA Nr3D
Model SimMR EM1MR SimMR
Scan2Cap-21.443.5------
VoteNet+MCAN---17.311.429.8---
ScanQA----13.1433.3---
3D-LLM-13.133.219.313.834.0---
3D-VLP----13.534.5---
Scene-LLM-21.845.6-15.8----
LL3DA*55.951.654.814.322.834.748.15.89.9
Ours59.152.457.414.422.934.848.220.819.2
Model Accuracy (%) EM CIDEr METEOR ROUGE
LL3DA 4.2 1.5 54.4 25.5 26.8
Ours 24.1 9.3 120.0 35.2 47.3
Improvement % +474.0% +520.0% +120.6% +38.0% +76.5%
Legend:
  • EM1: Top-1 Exact Match
  • EM: Exact Match
  • C: CIDEr
  • B-4: BLEU-4
  • M: METEOR
  • R: ROUGE
  • Sim: Sentence-BERT Similarity

Qualitative Results

BibTeX


    @article{halacheva2025gaussianvlm,
      title={GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond},
      author={Anna-Maria Halacheva and Jan-Nico Zaech and Xi Wang and Danda Pani Paudel and Luc Van Gool},
      year={2025},
      journal={arXiv preprint arXiv:2507.00886},
    }