MMDiff: Multi-modal Remote Sensing Image Generation

Multi-modal Remote Sensing Image Generation via Cross-Modality Spatial Feature Transfer

ISPRS 2026 🔥

Haojun Tang ¹

Wenda Zhao ^1,*

Hengshuai Cui ¹

Haipeng Wang ²

¹Dalian University of Technology

²Unit 92728 of PLA

^*Corresponding author

ISPRS Paper Code HuggingFace Dataset HuggingFace Model

The first model for spatially consistent multi-modal remote sensing image generation via one text prompt

Abstract

Collecting spatially consistent multi-modal remote sensing (MMRS) images remains challenging due to different sensors vary in the imaging principles and acquisition times. This hinders the development of data-driven MMRS technologies, which rely on large-scale training samples. This paper proposes MMDiff, the first text-driven diffusion framework explicitly designed for jointly generating structurally consistent optical (OPT), synthetic aperture radar (SAR), and infrared (IR) remote sensing images from a single text prompt via cross-modality spatial feature transfer. MMDiff first trains the OPT branch with paired optical image-text data to capture rich semantic content, and then trains the SAR/IR branches with simple modality-specific text templates to learn the corresponding style attributes, without relying on complex linguistic descriptions. Specifically, we introduce a LoRA-based modality translation adaptation mechanism to translate the style attributes of optical spatial representations to SAR and IR style attributes while preserving the underlying semantic content. The translated representations are then transferred into the SAR and IR generation branches through the proposed spatial feature transfer mechanism, enabling rich spatial details in the generated SAR/IR images while maintaining cross-modal spatial consistency. Extensive experiments demonstrate that MMDiff achieves superior image quality in terms of modality similarity and semantic consistency compared to the state-of-the-art methods. Furthermore, MMDiff benefits downstream data-driven MMRS applications, e.g., multi-modal image fusion and object classification.

📊 Main Results

1️⃣ Text-to-spatially-consistent Optical, Synthetic Aperture Radar, and Infrared Generation

Text-to-spatially-consistent OPT, SAR, and IR generation

2️⃣ Comparison with Existing Methods

Quantitative comparison across different methods. **Bold** and underline indicate the best and second-best performance, respectively.

3️⃣ Downstream Tasks

Multi-modal image fusion performance before and after fusion. The best results for EN, SD, SF, and AG are highlighted in **bold**, **bold^★**, **bold**, and **boxed bold**, respectively.

Multi-modal object classification on the OPT modality (Top-1 Accuracy). The large values denote the mean accuracy, while the small values denote the standard deviation. Best results are **bolded** and second best results are underlined.

Multi-modal object classification on the SAR modality (Top-1 Accuracy). The large values denote the mean accuracy, while the small values denote the standard deviation. Best results are **bolded** and second best results are underlined.

Multi-modal object classification on the IR modality (Top-1 Accuracy). The large values denote the mean accuracy, while the small values denote the standard deviation. Best results are **bolded** and second best results are underlined.

Method

MMDiff is designed to generate spatially consistent MMRS images in latent space using three parallel diffusion models. Specifically, MMDiff first samples a noisy latent $\mathrm{z}_T^o$ for the optical modality from a standard Gaussian distribution and then uses $\mathrm{z}_T^o$ to initialize $\mathrm{z}_T^s$ and $\mathrm{z}_T^I$ for SAR and IR modalities, where $T$ is the total sampling timestep for consistent MMRS image generation. For the optical modality, the pre-trained optical diffusion model $\epsilon_{\theta_o}$ is used to denoise $\mathrm{z}_T^o$ over $T$ iterations, generating the optical image $I_g^o = \mathcal{D}(\mathrm{z}_0^o)$ using the VAE decoder $\mathcal{D}(\cdot)$ . For the SAR and IR modalities, the modality translation adaptation mechanism is designed to translate the style attributes of the optical spatial representation into SAR- and IR-specific attributes while preserving its semantic content. Subsequently, the proposed spatial feature transfer mechanism uses these translated spatial representations to generate spatially consistent SAR and IR images. Specifically, at each timestep $t$ , the optical spatial representation extracted from the decoder of $\epsilon_{\theta_o}$ is first converted into SAR- and IR-specific representations by modality translation adaptation mechanism without altering its semantic structure. The adapted modality-specific representations are then injected into the decoders of the SAR and IR diffusion models, $\epsilon_{\theta_s}$ and $\epsilon_{\theta_I}$ , respectively. Finally, after denoising for $T$ timesteps, the SAR and IR images $I_g^s = \mathcal{D}(\mathrm{z}_0^s)$ and $I_g^I = \mathcal{D}(\mathrm{z}_0^I)$ are generated, which are spatially consistent with $I_o$ . Here, $\theta_o$ , $\theta_s$ and $\theta_I$ represent the parameters of $\epsilon_{\theta_o}$ , $\epsilon_{\theta_s}$ and $\epsilon_{\theta_I}$ , respectively. During the generation process, MMDiff uses only a single text prompt as the condition for all three diffusion models, without relying on any other spatial constraints.