GeoDiffuser: Geometry-Based Image Editing with Diffusion Models

IEEE/CVF WACV 2025 · Best Student Paper

Rahul Sajnani^1,2 · Jeroen Vanbaar² · Jie Min² · Kapil Katyal² · Srinath Sridhar^1,2

¹ Brown University ² Amazon Robotics

arXiv Paper Code BibTeX

Edits Overview Framework Method Results Limitations Citation Contact

Real Image Editing with GeoDiffuser

2D and 3D Edits

Object Removal

Overview

GeoDiffuser is a zero-shot, optimization-based method that unifies common 2D and 3D object-level image edits into a single pipeline. The key insight is to view image editing operations as geometric transformations that can be injected directly into the attention layers of a diffusion model — producing the edit implicitly, with no fine-tuning and no 3D supervision.

Given an input image and user guidance, the foreground is segmented with SAM and a corresponding transform is estimated. GeoDiffuser supports translation, 3D rotation, and object removal — and inpaints disoccluded regions that are revealed when objects move.

General Editing Framework

Our editing pipeline first inverts the image to obtain a noise latent, then runs two parallel diffusion processes:

A reference diffusion process that regenerates the input image and preserves untouched regions.
An edit diffusion process that reuses attention from the reference to realize the desired geometric edit.

Shared attention is the key mechanism that ties the two processes together.

Method Overview

GeoDiffuser introduces a novel attention-sharing mechanism that handles both 2D and 3D geometric edits inside a shared-attention layer. The method is entirely zero-shot and training-free, using latent optimization for edit guidance. The shared attention acts as a proxy for the post-edit foreground location, and the loss terms fill disocclusions while penalizing divergence of edit attention from reference attention.

Results

Limitations

Because edits occur in a lower-dimensional latent space, aliasing / interpolation artifacts can occasionally appear (e.g., the ship example). Optimization also sometimes yields sub-optimal foreground reconstructions or background disocclusion.

Citation

@inproceedings{sajnani2024geodiffuser, title={GeoDiffuser: Geometry-Based Image Editing with Diffusion Models}, author={Rahul Sajnani and Jeroen Vanbaar and Jie Min and Kapil Katyal and Srinath Sridhar}, booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, year={2025}, eprint={2404.14403}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Acknowledgements & Contact

Part of this work was done during Rahul's internship at Amazon. Additionally supported by NSF grant CNS #2038897, ONR grant N00014-22-1-259, and an AWS Cloud Credits award.

Questions? Reach out at rahul_sajnani@brown.edu.