(a) Pixel-level differencing fails under diffusion-based editing: a low threshold marks unedited regions, and a high threshold drops real but subtle edits. (b) Localizing only from the instruction marks what the prompt names, and misses side-effects the editor actually made. (c) SIGMA compares semantic features and grounds the instruction, so the mask falls on the region that changed.
Text-driven image editing has advanced rapidly, but localizing these manipulations requires image manipulation localization (IML) models trained on large pixel-annotated datasets. There is still no low-cost way to obtain such training data at scale. Public editing datasets already contain millions of (original, edited) pairs that match the structure of IML training samples. They lack only pixel-level masks. Recovering those masks is not a thresholding problem: pixel differencing is overwhelmed by diffusion-induced perturbations, and instruction-only grounding localizes only what the prompt describes.
We propose SIGMA (Semantic-difference Instruction-Grounding Mask Annotator). It differences features in a vision foundation backbone and injects an instruction-derived spatial prior through bidirectional cross-modal refinement. Stage I supervises on inpainting masks. Stage II closes the diffusion-domain shift with VAE-roundtrip noise calibration, EMA self-training, and an edit-noise disentanglement loss. On five benchmarks, SIGMA outperforms existing automatic mask generators by +12.20 F1 and +11.16 IoU. Applied to public editing corpora, it produces a ~1.1M IML training set that improves four detectors by +20.98 F1 across five datasets.
Public editing corpora have reached the million-sample scale. ByteMorph contains about 6.0M pairs and AnyEdit about 2.5M. The largest IML resources that already ship pixel masks are much smaller: SID-Set is about 300K and GRE about 228.7K. Each editing pair is already an (original, edited, instruction) triplet. The missing piece is a pixel mask that says where the edit landed.
Public editing datasets and masked IML datasets, drawn on a shared linear scale. The editing corpora are about an order of magnitude larger.
SIGMA uses two branches that check each other. The semantic branch differences multi-scale features from a frozen DINOv2 encoder, which answers where the content of the two images actually differs. The instruction branch parses the edit instruction into an original concept, an edited concept, and an action (add, remove, replace, attribute change, or global), then turns that tuple into a spatial prior. Bidirectional cross-modal refinement lets the semantic stream attend to the instruction first, and then lets the instruction re-anchor itself to the updated visual evidence.
Training has two stages. Stage I learns localized edits from inpainting pairs that already have exact masks. Stage II adapts the model to unlabeled public editing pairs. A VAE round-trip supplies a zero-edit noise reference, an EMA teacher (μ = 0.999) provides high-confidence pseudo-labels, and an edit-noise disentanglement loss separates edit features from reconstruction noise.
SIGMA localizes the edited region with a semantic-difference branch and an instruction-grounding branch.
We compare SIGMA with pixel-difference thresholds and with learning-based change detectors on CoCoGlide, AutoSplice, MagicBrush, DEAL-300K, and OpenSDI. Fixed thresholds do not transfer: PixDiff-Fixed at its best setting, τ = 10, reaches 71.10 average F1. DDPM-CD is strong on CoCoGlide (97.80 F1) and weak on DEAL-300K (20.90 F1). After Stage II, SIGMA reaches 89.83 average F1 and 83.52 average IoU, which is 12.20 F1 and 11.16 IoU above DDPM-CD. The largest margin is on DEAL-300K: 76.43 versus 20.90 F1.
| Method | Avg. F1 | Avg. IoU | DEAL-300K F1 |
|---|---|---|---|
| PixDiff-Fixed (τ = 10) | 71.10 | 62.09 | 28.04 |
| PixDiff-Otsu | 52.31 | 37.71 | 34.21 |
| Meta-CD | 71.30 | 60.06 | 45.80 |
| DDPM-CD | 77.63 | 72.37 | 20.90 |
| SIGMA | 89.83 | 83.52 | 76.43 |
F1 and IoU are percentages. The full per-dataset table is in the paper.
CoCoGlide. Columns, from left to right: original, edited, PixDiff-Fixed (τ = 10), PixDiff-Fixed (τ = 20), PixDiff-Fixed-morph, PixDiff-Otsu, Meta-CD, DDPM-CD, SIGMA, and ground truth.
DEAL-300K, with the same column order. SIGMA stays closer to the ground-truth mask where pixel-difference baselines and DDPM-CD break down.
SIGMA masks from the public editing corpora are used to fine-tune CAT-Net, MVSS-Net, PSCC-Net, and TruFor. Relative to the public weights, average F1 rises by +4.58 to +30.79, and the mean of these four gains is +20.98. The largest jumps are on MagicBrush: CAT-Net from 5.33 to 61.02 F1, and TruFor from 2.51 to 28.35. The same supervision also improves DEAL-300K and OpenSDI. Against the same image pairs annotated by DDPM-CD, SIGMA is stronger for CAT-Net, PSCC-Net, and TruFor. MVSS-Net's average F1 is higher with DDPM-CD masks (44.43 versus 42.69).
| Method | Weights | AutoSplice | CoCoGlide | DEAL-300K | MagicBrush | OpenSDI | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | F1 | IoU | ||
| CAT-Net | Public Weight | 41.49 | 31.79 | 40.97 | 33.69 | 12.14 | 8.75 | 5.33 | 3.29 | 7.32 | 5.27 | 21.45 | 16.56 |
| Finetune w/ DDPM-CD | 44.27 | 30.48 | 32.83 | 21.99 | 13.83 | 8.40 | 59.19 | 42.99 | 20.56 | 12.68 | 34.14 | 23.31 | |
| Finetune w/ SIGMA | 60.82 | 47.23 | 45.53 | 34.42 | 29.99 | 20.98 | 61.02 | 50.05 | 27.02 | 18.14 | 44.88 | 34.16 | |
| MVSS-Net | Public Weight | 18.04 | 12.22 | 32.80 | 25.30 | 5.43 | 3.61 | 1.79 | 1.01 | 1.46 | 0.91 | 11.90 | 8.61 |
| Finetune w/ DDPM-CD | 53.98 | 41.03 | 35.51 | 25.27 | 16.40 | 10.34 | 91.20 | 85.10 | 25.06 | 16.46 | 44.43 | 35.64 | |
| Finetune w/ SIGMA | 54.14 | 41.43 | 35.63 | 25.38 | 20.05 | 13.29 | 81.95 | 73.72 | 21.69 | 14.51 | 42.69 | 33.67 | |
| PSCC-Net | Public Weight | 56.36 | 43.83 | 50.85 | 40.60 | 20.19 | 13.69 | 14.87 | 9.37 | 14.50 | 9.64 | 31.35 | 23.43 |
| Finetune w/ DDPM-CD | 54.19 | 41.33 | 36.16 | 27.46 | 16.22 | 10.77 | 41.15 | 32.71 | 18.54 | 12.10 | 33.25 | 24.87 | |
| Finetune w/ SIGMA | 57.06 | 45.46 | 42.30 | 31.45 | 21.16 | 14.20 | 36.69 | 26.67 | 22.44 | 15.25 | 35.93 | 26.61 | |
| TruFor | Public Weight | 32.69 | 25.52 | 28.33 | 24.33 | 9.26 | 6.77 | 2.51 | 1.48 | 6.93 | 5.22 | 15.94 | 12.66 |
| Finetune w/ DDPM-CD | 57.06 | 41.97 | 55.33 | 42.31 | 22.49 | 14.36 | 20.63 | 12.36 | 20.13 | 12.78 | 35.13 | 24.76 | |
| Finetune w/ SIGMA | 63.42 | 50.63 | 58.72 | 47.34 | 27.85 | 18.92 | 28.35 | 20.06 | 27.02 | 19.20 | 41.07 | 31.23 | |
F1 and IoU (%). Bold marks the best result within each detector. The mean of the four average-F1 gains over public weights is +20.98.
Four public editing corpora, annotated by SIGMA into about 1.1 million pairs and 24 operation types.
The full annotated set is released on Hugging Face at PatrickZZZ/SIGMA. Two smaller splits of the same annotations are on ModelScope: 10,000 samples (SIGMA_small) and 100,000 samples (SIGMA_partial). These annotations are for academic research only. Any use must also follow the license of each source corpus (PromptfixData, AnyEdit, CrispEdit-2M, and OmniEdit-Filtered-1.2M) and cite both that corpus and SIGMA.
Under JPEG compression, additive Gaussian noise, and Gaussian blur, SIGMA's F1 stays near 0.95 on AutoSplice and near 0.91 on OpenSDI. Pixel differencing drops sharply once noise is added. Remote-sensing change detectors drop under blur.
F1 under JPEG compression, additive white Gaussian noise, and Gaussian blur on AutoSplice and OpenSDI.
@misc{zhuang2026sigma,
title = {SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization},
author = {Zhuang, Peiyu and Yang, Jianquan and Li, Haodong and Cai, Zhuoying and Xie, Ruitao and Zeng, Jishen and Chen, Baoying and Huang, Jiwu and Cao, Xiaochun},
year = {2026},
eprint = {2605.27924},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2605.27924}
}