BA-T: An Iterative Transformer
for Two-View Bundle Adjustment

NeurIPS 2026

Ganlin Zhang1,2 Weirong Chen1,2 Daniel Cremers1,2 Xi Wang1,2,3

1TU Munich 2MCML 3ETH Zurich

TL;DR A repeatable compact layer, enabling geometry and camera refinement in latent space.
Input views Input view A: kitchen counter Input view B: kitchen counter with toaster
Geometry & relative camera pose refined across BA-T iterations.
Overview of BA-T: a mono encoder followed by a shared BA-T layer applied for iterations 0 to 3; reconstructions and 3D point error maps improve with each iteration.
Overview of BA-T. Given input images, BA-T performs iterative updates on camera and local geometry tokens using a compact, reusable BA-T layer in latent space. The arrow indicates the error between GT poses and estimated poses (blue and pink), which gradually decreases; red boxes highlight regions progressively refined across iterations. The error maps visualize per-view 3D point errors.
16%of DUSt3R's decoder parameters (38M vs. 227M)
1shared, lightweight layer applied iteratively instead of a deep attention stack
−44%3D correspondence error from iteration 1 to 4 (vs. −27% for a stacked decoder)
3–4iterations to converge on pose and geometry

Abstract

Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency. We address this by drawing inspiration from classical bundle adjustment (BA), which can be viewed as an iterative information propagation process between poses and local geometry. Inspired by BA, we propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space. Instead of relying on deep attention stacks, BA-T refines predictions based on latent residual by a single lightweight layer. Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders, and matches or surpasses substantially larger models while using only 16% of their decoder parameters. BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture.

Method

BA-T pipeline: encoder, shared BA-T layers, heads; detail shows latent residual calculation, camera update and local geometry update.
The BA-T pipeline. BA-T takes camera tokens (learnable initialization) and local geometry tokens (from the image encoder) and refines them iteratively. Each step transforms geometry tokens across camera spaces, matches correspondences, and computes latent residuals; camera tokens and per-view geometry tokens are then refined by the Camera Update and Local Geometry Update.
Query tokens in view a and their attention scores in view b, highlighting correct correspondences.
Token-level correspondence. For a geometry token from one view, attention scores highlight the correct region in the other view, for both ambiguous (left, middle) and distinctive (right) regions.
Latent residual norm maps for four iterations, getting darker (smaller) each iteration.
Latent residuals shrink across iterations. The L2 norm of the latent residual, computed in the space of view b, decreases as the pose and geometry estimates improve.

Every iteration is supervised, with later iterations weighted more heavily (\(\mathcal{L}_{\text{total}} = \sum_{k} \alpha_\lambda^{K-k}\mathcal{L}^{(k)}\)), so the network learns to refine progressively rather than only fitting its final output.

Results

Iterative refinement, two views

Local geometry and the relative camera are refined across iterations.

Input view: whiteboard Input view: office chair
Input view: room corner Input view: room corner, closer
Input view: cluttered desk Input view: cluttered desk, shifted
Input view: bookshelf Input view: bookshelf and desk

Extension to multi-view

All cross-view interactions in BA-T happen in each view's local space, so it extends to more views without architectural changes: each query view attends to all others, and all views are refined together.

Office input 1 Office input 2 Office input 3 Office input 4
Bedroom input 1 Bedroom input 2 Bedroom input 3 Bedroom input 4
Four-view reconstruction across iterations 1 to 3 with trajectory ATE 0.232m, 0.149m and 0.029m.
Four-view reconstruction. Green frustums are GT poses. Trajectory ATE drops from 0.232 m to 0.029 m over three iterations.

Qualitative results

Left: reconstructions and 3D error maps at iteration 1 and 4. Right: estimated relative poses across iterations with decreasing pose error.
Left: reconstructions and local 3D point error maps at iterations 1 and 4 on 7Scenes, BundleFusion and TUM-RGBD. Red boxes mark misalignments at iteration 1 that are corrected by iteration 4 (green). Right: estimated relative poses across iterations. BA-T keeps refining the pose even with very little visual overlap (second row).

BibTeX

@inproceedings{zhang2026bat,
  title     = {{BA-T}: An Iterative Transformer for Two-View Bundle Adjustment},
  author    = {Zhang, Ganlin and Chen, Weirong and Cremers, Daniel and Wang, Xi},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}