Generative Demos

Across these three projects I build legible, designed structures inside the forward pass of pretrained diffusion models — so a user can control where content goes, what it looks like, and how it moves, without retraining the base model. For full write-ups: GeoDiffuser, AppearancePointers, GenHSI.

WACV 2025 · Best Student PaperGeoDiffuser

Geometry-based image editing on a frozen text-to-image diffusion model — training-free, invited talk at WACV 2025. User specifies a 3D rigid transformation (translate, rotate, scale) on an object; GeoDiffuser realizes it by rewriting the model's attention maps at inference time, preserving physically-consistent shadows and reflections.

Edit gallery — movement

Edit gallery — object removal

Headline results: MD 7.3 vs 10.8 on 3D edits (vs fine-tuned Diffusion Handles); 5.6 vs 20.0 on 2D edits (vs Zero123-XL + LaMa).

WACV 2026Training-freeGenHSI

Training-free controllable generation of long human–scene interaction (HSI) videos. The user supplies three things — an image of a scene, an image of a character / avatar, and a text prompt describing a chain of interactions — and GenHSI synthesizes a long, 3D-aware video of the character performing those interactions inside the scene, with identity preservation and physically plausible affordances. No fine-tuning of the base video-diffusion model.

Three-stage pipeline inspired by movie animation: (1) Script writing — a VLM parses the high-level text prompt into atomic actions grounded in the scene's affordances; (2) Pre-visualization — a pretrained 2D inpainting diffusion model is used to generate plausible 2D human poses via view canonicalization, which are then lifted to 3D keyframes under contact + VLM reasoning; (3) Animation — the 3D keyframes condition a pretrained video-diffusion model to render the final long interaction video.

Authors: Zekun Li, Rui Zhou, Rahul Sajnani, Xiaoyan Cong, Daniel Ritchie, Srinath Sridhar.

Same instinct as GeoDiffuser — use a pretrained generative model as a compositional substrate without retraining — carried from 2D image editing to 3D-aware long-form video generation.

Scene 1 — "A man sits on the bed, then sits on the chair."

Scene 2 — "The girl stands next to the chair, then sits on it. After that, she approaches the table and touches it."

Under review · ICLR 2027AppearancePointers

Fine-grained multi-region controllable image generation on FLUX / MMDiT. Every region in the output can be independently specified by a different modality — text, reference image, or material texture — without region bleed, in a single denoising pass. Introduces per-region appearance-pointer tokens plus a position-ID resampling scheme that routes each conditioning signal to its designated spatial region.

Method overview

Qualitative results

Headline results: Beats DreamRenderer, MSDiffusion, InstanceDiffusion, and Seg2Any — gold on CLIP-I / DINO-I / MIoU for image-conditioned region generation, and gold on CLIP-I / DINO-I for text-conditioned. Also curated and released AP-37K, a 37K-sample multimodal dataset of text + reference-image + texture conditioning triplets.