ConeGaussian

面向任意中心相机的抗锯齿 Gaussian 光线追踪:把每个像素当成一束光线。Anti-aliased Gaussian ray tracing for any central camera: treat each pixel as a bundle of rays.

Deheng Zhang*, Letian Shi*, Runyi Yang, Zhendong Li, Lei Sun, Kanzhi Wu, Ajad Chhatkuli, Danda Pani Paudel, Luc Van Gool
INSAIT, Sofia University · vivo · arXiv 2609.13397

→ / 空格:下一步 ← 上一步 L:中 / EN F:全屏 O:目录→ / Space: next ← back L: 中 / EN F: fullscreen O: contents

为什么通用相机需要 Gaussian 光线追踪Why Gaussian ray tracing for generic cameras

\[\mathbf J=\frac{\partial\pi}{\partial\mathbf t}\Big|_{\mathbf t_\mu}=\begin{pmatrix}f_x/z&0&-f_x x/z^{2}\\0&f_y/z&-f_y y/z^{2}\end{pmatrix}\]
01

到相机坐标To camera coordinates

\(\mathbf t_\mu=\mathbf V(\boldsymbol\mu-\mathbf c)=(x,y,z)\)
02

在中心处一阶展开First-order expansion at the centre

\(\pi(\mathbf t)\approx\pi(\mathbf t_\mu)+\mathbf J(\mathbf t-\mathbf t_\mu)\)
03

得到一个 2D GaussianOne 2D Gaussian

\(\boldsymbol\mu_{\rm 2D}=\pi(\mathbf t_\mu),\ \ \boldsymbol\Sigma_{\rm 2D}=\mathbf J\mathbf V\boldsymbol\Sigma\mathbf V^{\top}\mathbf J^{\top}\)
04

逐像素求值Per-pixel evaluation

\(\alpha(\mathbf p)=\sigma\,e^{-\frac12(\mathbf p-\boldsymbol\mu_{\rm 2D})^{\top}\boldsymbol\Sigma_{\rm 2D}^{-1}(\mathbf p-\boldsymbol\mu_{\rm 2D})}\)

为什么通用相机需要 Gaussian 光线追踪Why Gaussian ray tracing for generic cameras

长轴Long axis
精确投影:Gaussian 的样本点逐个经过真实相机投影exact projection: each sample of the Gaussian through the real camera Jacobian 近似:中心处线性化得到的 2D Gaussian,画 2σ 轮廓Jacobian approximation: the 2D Gaussian from linearizing at the centre, 2σ contour

针孔Pinhole

等距鱼眼Equidistant fisheye

一个像素看到的是一束光线,而不是一条A pixel sees a bundle of rays, not a single ray

中心光线的像素值:随场景移动忽明忽暗Centre-ray pixel value: flickers as the scene moves
光线锥的平均值:稳定,接近真实像素Cone average: steady, close to the true pixel

3DGEER、3DGUT、3DGRT 这类 Gaussian 光线追踪器,每个像素只追踪中心那一条光线。Gaussian ray tracers such as 3DGEER, 3DGUT and 3DGRT trace only the centre ray of each pixel.

真实的像素会积分一整束光线,即一个光线锥。分辨率越低,锥越粗,一条光线漏掉的信息就越多。A real pixel integrates a whole bundle: a cone of rays. The lower the resolution, the wider the cone and the more a single ray misses.

3DGEER bicycle at quarter sampling rate: spokes break into speckle

缩小 → 锯齿、闪烁Zoom-out → aliasing

3DGEER,¼ 采样率:辐条碎成亮点3DGEER at ¼ sampling: spokes break into speckle

3DGEER garden trained at one eighth resolution and magnified: needle artifacts

放大 → 针状伪影Zoom-in → needle artifacts

3DGEER,⅛ 分辨率训练后 ×16 放大3DGEER trained at ⅛, magnified ×16

缩小时的不连续:中心光线碰巧打中谁,就报告谁Zoom-out discontinuity: a centre ray reports whatever it happens to hit

一维示意:一排细 Gaussian(像自行车辐条)。“真值”是像素内的精确平均;ConeGaussian 行用同均值、同方差的 Gaussian 近似像素盒(与论文假设 3 相同)。1D toy: a row of thin Gaussians (like bicycle spokes). "Ground truth" is the exact average over each pixel; the ConeGaussian row replaces the pixel box with a moment-matched Gaussian (the paper's Assumption 3).

放大时的不连续:split 留下比像素还细、没有监督的 GaussianZoom-in discontinuity: splits leave sub-pixel Gaussians no pixel supervises

一维示意,24 个训练像素;split 是脚本化示意(一个 Gaussian 换成两个更细的矩匹配子 Gaussian),不是真实优化;亚像素 = 标准差小于 1/√12 像素,即一个像素盒的标准差。1D sketch, 24 training pixels; the splits are scripted (one Gaussian becomes two finer moment-matched children), not a real optimisation; sub-pixel = standard deviation below 1/√12 pixel, the standard deviation of a pixel box.

现有方法要么支持任意相机,要么做像素滤波,两者没有同时做到Prior work supports generic cameras or pixel filtering, not both

只支持针孔(投影到屏幕平面)Pinhole only (screen-plane projection)
任意中心相机(沿光线求值)Any central camera (evaluated along rays)
每像素一条中心光线One centre ray per pixel
3DGS局部仿射 EWA 投影,绑定透视成像平面Local affine EWA projection, tied to a perspective image plane
3DGEER · 3DGUT · 3DGRT直接在 3D / 沿光线求值,任何相机都能渲染,但像素 = 一条光线Evaluate in 3D or along rays; any camera renders, but a pixel is one ray
近似的像素 footprintApproximate pixel footprint
Mip-Splatting · Analytic-Splatting在针孔屏幕平面上滤波,鱼眼没有对应的平面Filter on the pinhole screen plane; a fisheye has no such plane
VKRayGS光线渲染中的滤波,但用各向同性 z/f 估大小,丢掉剪切,离轴过度膨胀Filters inside ray rendering, but sizes an isotropic z/f blob: no shear, over-dilated off-axis
各向异性 footprint,来自相机自己的光线Anisotropic footprint from the camera's own rays
— 屏幕平面方法无法直接用于强畸变鱼眼— screen-plane methods do not carry over to strong fisheye
ConeGaussian邻居光线 → 光线垂直平面上的各向异性协方差 → 闭式响应。同一份实现用于 3DGEER 和 3DGUT(也移植到了 3DGRT)。Neighbour rays → anisotropic covariance in the ray-orthogonal plane → closed-form response. One implementation on 3DGEER and 3DGUT (also ported to 3DGRT).

缩小:像素 footprint 滤波Zoom out: Pixel-Footprint Filtering

01

取邻居光线Neighbour rays

\(\mathbf d_i=\Pi^{-1}(\mathbf p+\boldsymbol\Delta_i)\)

半像素偏移处的光线,只需相机的反投影:针孔、鱼眼用同一份代码。Rays at half-pixel offsets from the camera's inverse projection: one code path for pinhole and fisheye.

02

搬到最大响应深度 t*Transport to t*

\(\mathbf e_i=t^\ast\,\mathbf M_{\!\perp}\mathbf W(\mathbf d_i-\mathbf d)\)

footprint 随深度线性变大;在 Gaussian 响应最大处取截面。The footprint grows linearly with depth; cut it where the Gaussian responds most.

03

白化 + 矩匹配Whiten + moment-match

\(\boldsymbol\Sigma_{\rm fp}=\tfrac13(\mathbf e_x\mathbf e_x^{\!\top}+\mathbf e_y\mathbf e_y^{\!\top})\)

平行四边形 → 同均值、同协方差的 Gaussian,方向和长宽比都保留。Parallelogram → Gaussian with the same mean and covariance; orientation and aspect ratio survive.

04

闭式响应Closed-form response

\(T=\frac{\sigma}{\sqrt{\det\mathbf A}}e^{-\frac12\mathbf q^{\!\top}\mathbf A^{-1}\mathbf q},\ \mathbf A=\mathbf I+\boldsymbol\Sigma_{\rm fp}\)

卷积变成协方差相加;每个像素–Gaussian 对只多几次 2×2 运算。Convolution becomes a covariance sum; a few 2×2 operations per pixel–Gaussian pair.

放大:约束表示的频率Zoom in: Regularizing Representation Frequencies

  • 同一个 Gaussian 在不同训练视角里被不同大小的像素 footprint 看到:越近越细。One Gaussian is seen through differently sized footprints in different training views: closer means finer.
  • 取最细的那个作为下限 \(\rho^2\)。比它更细的结构,训练数据从来没有监督过。The finest one sets the floor \(\rho^2\). Anything finer was never supervised by the training data.
  • 把 Gaussian 膨胀到下限,并补偿不透明度。放大时就不会露出针状的高频伪影。Dilate the Gaussian to the floor and compensate its opacity, so magnification exposes no needle-like detail.
\[\rho^2=\min_{v\in\mathcal V}\tfrac12\operatorname{tr}\boldsymbol\Sigma_{\rm fp}^{(v)}\qquad \boldsymbol\Sigma_{\rm eff}=\boldsymbol\Sigma+\rho^2\mathbf I\]

和渲染滤波用同一套邻居光线几何,所以鱼眼边缘也准确;Mip-Splatting 的下限用的是 z/f 近似。It reuses the same neighbour-ray geometry as the render filter, so it stays accurate at the fisheye edge; Mip-Splatting's floor uses a z/f approximation.

两种滤波的边际组合Marginal Composition for Two Filters

渲染尺度(相对训练分辨率)Render scale (relative to training)
下限 ρ²I(训练时已加进 Gaussian)floor ρ²I (baked into the Gaussian) 当前像素 footprintcurrent pixel footprint 直接相加 additiveadditive 边际 marginal(逐方向取大)marginal (direction-wise max)
\[\boldsymbol\Sigma_{\rm render}=\big[\boldsymbol\Sigma_{\rm fp}-\rho^2\mathbf I\big]_+\]

footprint 只有秩 2,所以 PSD 截断是逐方向的:已被下限覆盖的方向不再加模糊,更粗的方向只补差值。The footprint has rank 2, so the PSD clamp acts per direction: directions the floor covers get no extra blur; wider ones get only the excess.

ScanNet++ 消融(¼ 分辨率训练):marginal 相对 additive 的 PSNR 差ScanNet++ ablation (trained at ¼): PSNR of marginal minus additive

相机模型决定每个像素 footprint 的大小、形状和朝向The camera sets each footprint's size, shape and orientation

图里的形状是什么What the shapes mean

  • 淡橙色平行四边形:像素光线束截面Pale orange parallelogram: pixel ray-bundle cross-section
  • 红色实线椭圆:ConeGaussian ΣfpRed solid ellipse: ConeGaussian Σfp
  • 绿色虚线圆:z/f 近似(VKRayGS)Green dashed circle: z/f estimate (VKRayGS)

三种协议,分别检验渲染滤波、表示下限,以及两者的联合Three protocols: render filter, representation floor, and both

MTMT

1
½
¼
⅛

四个分辨率一起训练、分别测试。联合检验两个模块。Train on all four scales, test each. Tests both modules together.

缩小 Zoom-out(STMT)Zoom-out (STMT)

1
½
¼
⅛

原分辨率训练,直接渲染到更低分辨率。主要检验渲染时的 footprint 滤波。Train at native, render coarser without retraining. Mainly tests the render-time filter.

放大 Zoom-inZoom-in

×1
×2
×4

最低分辨率训练,渲染到 ×1–×4。主要检验训练频率下限。Train at the lowest scale, render ×1–×4. Mainly tests the training-frequency floor.

训练分辨率trained on测试分辨率tested on
ScanNet++ · 鱼眼,室内fisheye, indoor Zip-NeRF · 鱼眼fisheye Mip-NeRF 360 · 针孔pinhole Blender · 针孔,合成(附录)pinhole, synthetic (appendix) 3DGEER · 精确光线积分exact ray integral 3DGUT · unscented transform

指标 PSNR / SSIM / LPIPS;每 8 张取 1 张做测试;鱼眼指标只在腐蚀 3 像素后的有效区域内计算。缩放只改变采样分辨率(标定同步缩放),不改变焦距或位姿。Metrics PSNR / SSIM / LPIPS; every 8th image held out; fisheye metrics inside a 3-pixel-eroded valid mask. Scaling changes only the sampling resolution (calibration scaled to match), never focal length or pose.

MTMT 结果MTMT result

点击柱子查看各尺度曲线Click a bar for the per-scale curve

缩小:各向异性 footprint 平均 PSNR 最高,z/f 圆在强缩小时崩掉Zoom-out: anisotropic leads average PSNR; z/f collapses

3DGEER, bicycle at quarter sampling rate
ConeGaussian, bicycle at quarter sampling rate
Ground truth
3DGEERConeGaussian真值 GTGround truth
拖动对比。Mip-NeRF 360 bicycle,¼ 采样率;此裁剪区 PSNR 19.05 → 22.74 dB(单个裁剪,不是平均)。Drag to compare. Mip-NeRF 360 bicycle, ¼ sampling; this crop 19.05 → 22.74 dB PSNR (one crop, not an average).

放大时提升较小,但三个数据集上完整模型的平均指标都是最好Zoom-in: smaller gains, but the full model is best on all three

+ Mip-Splatting floor(z/f)+ footprint floor(ours,只有下限)+ footprint floor (ours, floor only)ConeGaussian 完整模型ConeGaussian, full model

鱼眼边缘(ScanNet++,×4,r > 0.9):Mip-Splatting 的 z/f floor 反而 −0.33 dB;footprint floor +0.09;完整模型 +0.15 dB。Fisheye edge (ScanNet++, ×4, r > 0.9): Mip-Splatting's z/f floor loses −0.33 dB; footprint floor +0.09; full model +0.15 dB.

3DGEER garden magnified
ConeGaussian garden magnified
3DGEERConeGaussian

Mip-NeRF 360 garden,⅛ 分辨率训练,×16 放大(视频帧):基线露出亚像素的“针”,ours 保持平滑。Mip-NeRF 360 garden trained at ⅛, magnified ×16 (video frame): the baseline exposes sub-pixel needles; ours stays smooth.

跨相机渲染:不用重训,footprint 按目标相机重新计算Cross-camera: no retraining; footprints follow the new camera

同一个训练好的场景,在不同中心相机模型下渲染(左 3DGEER,右 ConeGaussian)。One trained scene rendered through different central camera models (left 3DGEER, right ConeGaussian).

消融:两个模块分工明确,2D footprint 滤波撑起缩小,3D 下限帮助放大Ablation: 2D footprint filter carries zoom-out; 3D floor helps zoom-in

比完整模型差(越深差越多)worse than the full model (darker = larger drop)比完整模型好better than the full model
−2.9 dB

去掉 2D footprint 滤波,⅛ 渲染从 27.18 掉到 24.30 dB,和 3DGEER 基线几乎一样。Without the 2D footprint filter, ⅛ drops from 27.18 to 24.30 dB, essentially the 3DGEER baseline.

−0.8 dB

去掉 3D 下限,½ 渲染掉 0.80 dB,×4 掉 0.10 dB。下限通过训练也影响了中间尺度。Without the 3D floor, ½ drops 0.80 dB and ×4 drops 0.10 dB. Through training, the floor also shapes intermediate scales.

Marginal 比 additive 在放大侧高约 0.2 dB;⅛ 时 additive 略好 0.06 dB。Marginal beats additive by about 0.2 dB on the zoom-in side; additive is 0.06 dB better at ⅛.

更多关于我More about me

1 / 19