What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction

Truong, Loc-Phat; Madadi, Meysam; Escalera, Sergio

What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction

Loc-Phat Truong, Meysam Madadi, Sergio Escalera

Computer Vision Center (CVC), Universitat de Barcelona
ICPR 2026

Paper Code arXiv

What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction

Virtual Try-On lays a rich foundation on architectures and techniques to drap-on the garment onto a person image. However, whether they are transferable and which one is effective to the inverse problem, Virtual Try-Off, remain open questions. State-of-the-art single-UNet, TryOffDiff and Try-Off-Anyone, produces artifacts in garment shape, texture, color, and less generalized capabilities. In contrast, our Dual-UNet architecture achieves high-quality and realistic garment generation while precisely preserving fine-grained details across diverse datasets.

Methodology

We investigate and measure the impact on the model performance along three key design axes:

(i) Generation Backbone: Comparative analysis of modern Stable Diffusion variants adapted for high-resolution canvas reconstruction.
(ii) Conditioning: Extensive ablations on clothing masks, comparing masked vs. unmasked inputs for spatial image conditioning, and leveraging high-level semantic features.
(iii) Losses and Training Strategies: Evaluating the impact of auxiliary attention-based losses, perceptual objectives, and a multi-stage curriculum training schedule to steer reconstruction details.

Our final Dual-UNet architecture.

Key Insights

Comparison of garment reconstruction and attention maps without and with Leffa alignment regularization

Attention Alignment Regularization [3] is Essential for Texture Accuracy

L_Leffa: Aligns intermediate layer keys & queries to map worn regions directly to canonical positions, preventing logo distortion and texture warping.

Three-phase curriculum covering CreationNet warm-up, strong alignment, and fine-grained ConditionNet refinement

Careful Curriculum Design Prevents Collapse

Given a large architecture, an end-to-end joint training of both UNets causes training instability and metric degradation. We design a curriculum to progressively enhance both networks.

Backbone and masking ablations favoring SDXL, base diffusion, and dilated masks

Backbone & Masking Choices Matter

Backbone selection dictates CreationNet's generative capacity, while using a base model over inpainting proves crucial. Masking directly impacts low-level fine-grained features, but has negligible effect on global guidance.

DISTS is the primary metric

for VTOFF because SSIM/LPIPS heavily penalize harmless shape length variations under occlusion while ignoring fine texture loss.

Quantitative Results

Quantitatively evaluated on VITON-HD and DressCode datasets, our framework achieves state-of-the-art performance with a drop of 9.5% on the primary metric DISTS and competitive performance on LPIPS, FID, KID, and SSIM, providing both stronger baselines and insights to guide future Virtual Try-Off research.

Method	Resolution	SSIM ↑	LPIPS ↓	DISTS ↓	FID ↓	KID ↓
TryOffDiff [1] (HD)	512 × 384	75.02	28.52	22.32	24.56	9.52
Try-Off-Anyone [2] (HD)	512 × 384	72.35	34.08	22.11	11.57	2.01
Ours (HD)	512 × 384	74.70	28.75	20.10	9.73	1.82
IGR [3] (HD)	1024 × 768	78.95	29.46	20.45	13.14	2.97
Ours (HD)	1024 × 768	76.04	31.41	19.59	10.69	2.31
TryOffDiff [1] (DC)☆	512 × 384	80.8	31.6	21.6	17.1	4.7
Ours (DC)☆	512 × 384	75.53	32.61	20.85	12.30	2.12

☆ DISTS is computed at 341 × 256 resolution following the TryOffDiff evaluation protocol.

Qualitative Result

VITON-HD qualitative comparison sample 1

Studio-condition Testing Samples, which the model was trained with.

VITON-HD qualitative comparison sample 2

Studio-condition Testing Samples, which the model was trained with.

VITON-HD qualitative comparison sample 3

Studio-condition Testing Samples, which the model was trained with.

VITON-HD qualitative comparison sample 4

Studio-condition Testing Samples, which the model was trained with.

VITON-HD qualitative comparison sample 5

Studio-condition Testing Samples, which the model was trained with.

Unsplash real-world generalization sample 1

Unsplash Real-World Samples: Zero-shot generalization in-the-wild (No Ground Truth canonical flat garment exists).

Unsplash real-world generalization sample 2

Unsplash Real-World Samples: Zero-shot generalization in-the-wild (No Ground Truth canonical flat garment exists).

Unsplash real-world generalization sample 3

Unsplash Real-World Samples: Zero-shot generalization in-the-wild (No Ground Truth canonical flat garment exists).

Unsplash real-world generalization sample 4

Unsplash Real-World Samples: Zero-shot generalization in-the-wild (No Ground Truth canonical flat garment exists).

DressCode qualitative comparison sample 1

High Resolution Samples: Model is progressively trained from low resolution to higher resolution.

DressCode qualitative comparison sample 2

High Resolution Sample 2: Model is progressively trained from low resolution to higher resolution.

BibTeX

@inproceedings{truong2026matters,
  title={What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction},
  author={Truong, Loc-Phat and Madadi, Meysam and Escalera, Sergio},
  year={2026},
  eprint={2604.08716},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2604.08716}
}