Appearance Pointers: MultiModal Region Control of Diffusion Transformers

Under review · 2026

Rahul Sajnani^1,2 · Yulia Gryaditskaya² · Radomír Měch² · Srinath Sridhar¹ · Matheus Gadelha²

¹ Brown University ² Adobe Research

Capabilities Methodology Qualitative Results Dataset Citation Contact

Capabilities

Appearance Pointers enables fine-grained multi-modal region control of a pretrained diffusion transformer: each region of the output can be independently steered by a text prompt, a reference image, or a material texture — all in a single denoising pass.

Methodology

Appearance Pointers performs region-conditioned image generation and editing by injecting regional tokens into a pre-trained FLUX Kontext generation model. Special tokens denoted ⁱAP and ᵀAP tie each region to its conditions, enabling multi-modal region-level control inside a single forward pass.

Overview of the Appearance Pointers architecture.

Region Linking

Each region is processed independently. Its text prompt, reference image, and mask signals are fused to produce region tokens ⁱMᵢ (image stream) and ᵀMᵢ (text stream), which feed the FLUX transformer per region i.

Region Aggregation

Region tokens from the image and text streams across all regions are combined into a single, unified per-region representation. Aggregation happens independently at every spatial patch, which preserves the spatial cues carried by each Appearance Pointer token.

Attention Map for Appearance Pointers

After training, the Appearance Pointer learns to attend to its corresponding region condition. Queries come from the Appearance Pointer token; keys come from the Image Condition tokens. The heatmap below visualizes the attended region at a highlighted query location.

Qualitative Results — In the Wild

We showcase heterogeneous region-condition combinations: each region of a target image is independently driven by text, a reference image, or a material texture. Numbered regions tie back to their corresponding condition.

Dataset

We release a region-control dataset containing, for each image:

Region segments
Region captions
Region pose-variation edits
Region texture on a sphere
Pose and texture consistency scores

Citation

@misc{sajnani2026AppearancePointers, title={Appearance Pointers: MultiModal Region Control of Diffusion Transformers}, author={Rahul Sajnani and Yulia Gryaditskaya and Radomír Měch and Srinath Sridhar and Matheus Gadelha}, year={2026}, eprint={}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Acknowledgements & Contact

Part of this work was done during Rahul's internship at Adobe Research.

Questions? Reach out at rahul_sajnani@brown.edu.