ICLR 2026 · Fudan University

SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention

Jiahao Li, Jiayi Dong, Peng Ye, Xiaochi Zhou, Haohai Lu, Fei Wang*

College of Computer Science and Artificial Intelligence, Fudan University  ·  *Corresponding author

LLM Prior
Semantic Gene Embeddings

1,536-dim NCBI Gene summaries (GenePT) → OT soft-patched blocks

Latent FM
Gene-Block VAE + DiT OT-FM

AdaLN + condition masking (p = 0.6) & multi-condition CFG across 1–5 covariates

191× Faster
Training & Sampling Efficiency

12.5 min vs. 2,391.2 min on RTX 3090 (~7 min on Colab G4, 20-step ODE)

Tables 1–5
8 Benchmarks · Colab Verified

Unified config across generation, BEC & perturbation (11 datasets reproduced)

01 / Overview & Architecture

LLM biological knowledge meets Latent Flow Matching at 191× efficiency.

SAVE brings three core advantages to multi-condition single-cell RNA-seq modeling: LLM biological knowledge (NCBI Gene summaries clustered via Optimal Transport into pathway-coherent Gene Blocks), a Latent Flow Matching tech stack (Gene-Block Attention VAE + DiT Optimal-Transport Flow Matching with masked conditioning and CFG), and 191× training speedup over flat gene-level Transformers.

01 / LLM KNOWLEDGE

LLM Biological Knowledge

Encodes curated NCBI Gene functional summaries via GenePT (1,536-dim semantic embeddings) and solves iterative Optimal Transport clustering to partition ~20,000 unordered genes into L balanced, pathway-coherent Gene Blocks (K = 3,200).

02 / LATENT FLOW MATCHING

Gene-Block VAE + DiT OT-FM

Compresses structured gene blocks (X ∈ ℝN×L×K) into pathway-structured latents (z ∈ ℝN×L×d), then trains DiT-based Conditional OT Flow Matching (AdaLN, condition masking p = 0.6, multi-condition CFG) across 1–5 covariates and 8 benchmarks with one fixed config.

03 / EFFICIENCY

191× Speedup & 20-Step Sampling

Grouping G ≈ 19,112 genes into L = 6 blocks collapses attention complexity from O(G2) to O(L2), cutting single-RTX-3090 training from 2,391.2 min (~40 h) to 12.5 min (266× / 9.0 min at K = 5,600; ~7 min on Colab G4).

LLM Semantic Gene Blocks vs. ViT Patchify

Standard Transformer models either attend over ~20,000 flat genes (O(G2) memory/time bottleneck) or lack a spatial 2D neighborhood for Vision Transformer (ViT) grid patching. SAVE replaces spatial proximity with LLM semantic similarity + balanced Optimal Transport soft-patching before feeding structured tokens into the Latent VAE + Latent Flow Matching stack.

Figure 1. ViT Patchify (Vision) vs. SAVE Gene Block Mechanism (Genomics). Instead of hard-patching a structured 2D pixel grid by Euclidean proximity, SAVE extracts 1,536-dim LLM semantic embeddings from NCBI gene summaries (Stage 1) and performs balanced Optimal Transport clustering (Stage 2) to form pathway-coupled Gene Blocks for downstream Latent VAE compression and Latent Flow Matching.
  1. Stage 1 — LLM Semantic Embedding: Curated NCBI Gene text descriptions are encoded with a pre-trained LLM (text-embedding-ada-002, GenePT) into continuous 1,536-dim semantic vectors gi, injecting prior biological functional knowledge directly into tokenization.
  2. Stage 2 — Balanced Optimal Transport Soft-Patching → Latent FM: Iterative Optimal Transport minimizes squared Euclidean cost Cij = ‖gi − cj(t)‖22 under uniform marginal constraints (T1L = a, T⊤1G = b), yielding non-overlapping, equal-size functional blocks X ∈ ℝN×L×K that serve as compact tokens for Latent VAE + OT Flow Matching.
Dimension ViT Patchify (Vision Domain) SAVE Gene Block Mechanism (Genomics Domain)
Input Data Structure Structured 2D pixel grid Unordered, high-dimensional & sparse gene set (~20,000 genes)
Grouping Basis Physical proximity (Euclidean grid) Biological semantic similarity (pre-trained LLM prior on NCBI descriptions)
Patching Type Hard-patching (fixed spatial grid) Semantic soft-patching (iterative balanced Optimal Transport clustering)
Token Properties Functionally agnostic image patches Functionally coupled pathway blocks (coherent gene programs)
Downstream Stack & Efficiency Flat patch ViT attention Latent VAE + Latent OT Flow Matching (O(L2) attention, 191× training speedup: 12.5 min vs. 2,391.2 min on RTX 3090)

End-to-End Architecture: Latent VAE + Latent Flow Matching

Once LLM embeddings and Optimal Transport partition profiles into L pathway-coherent Gene Blocks, a Gene-Block Attention VAE compresses expressions into compact latents z ∈ ℝN×L×d where masked DiT Optimal-Transport Flow Matching learns conditional cellular trajectories in just 20 ODE steps.

Figure 2. Overview of SAVE. Left: LLM-embedded NCBI gene summaries are clustered via Optimal Transport into balanced gene blocks (X ∈ ℝN×L×K). Bottom: A Gene-Block Attention VAE compresses profiles into pathway-structured latents z ∈ ℝN×L×d. Top: Latent Conditional Optimal-Transport Flow Matching with AdaLN and condition masking generates unseen multi-condition cell states, corrects batch effects, and transfers perturbations.

02 / Quantitative & Qualitative Results

Consistent gains across generation, extrapolation, integration, and perturbation.

Paper benchmark comparisons across multi-platform scRNA-seq cohorts (bold = best, italic = second best). Full per-dataset Colab Pro+ G4 reproduction numbers are summarized in Section 04.

2.1 Dual-Condition Generation (batch + cell_type)

Paper Table 2 · WD ↓ & MMD ↓

Jointly conditioning on sequencing batch and cell type across three multi-platform scRNA-seq cohorts (Heart, PBMC, and Lung Atlas).

Method Heart WD ↓ Heart MMD ↓ PBMC WD ↓ PBMC MMD ↓ Lung Atlas WD ↓ Lung Atlas MMD ↓
scVI 19.18 1.19 17.75 0.95 22.20 2.27
CFGen 12.57 0.66 17.41 0.65 28.02 1.76
scDiffusion 20.82 0.94 11.38 0.48 13.89 1.71
SAVE 8.30 0.63 5.37 0.29 4.37 1.14
Figure 3. Dual-condition generation on Heart. Top: generated vs. ground-truth overlap; Middle: sequencing batches; Bottom: cell types. SAVE accurately preserves both multi-batch structure and cell-type topology.

2.2 Multi-Condition Generation & Unseen Extrapolation (5 Covariates)

Paper Table 3 · Lung Cancer (Salcher et al. 2022)

Evaluated simultaneously across five covariates (assay, development_stage, disease, uicc_stage, cell_type), including held-out unseen covariate combinations.

Method Seen WD ↓ Seen MMD ↓ Unseen WD ↓ Unseen MMD ↓
CFGen 16.94 ± 4.19 1.49 ± 1.80 23.67 ± 3.08 2.76 ± 2.64
scDiffusion 5.27 ± 2.14 1.06 ± 1.59 5.29 ± 2.03 1.87 ± 2.31
SAVE 3.10 ± 0.96 1.06 ± 1.48 4.63 ± 0.95 2.11 ± 2.11
Figure 4. Five-condition generation on Lung Cancer. (a) Seen combination (Smart-seq2, 60-year-old, lung adenocarcinoma, stage IV); (b) Unseen held-out combination (Smart-seq2, 70-year-old, squamous cell lung carcinoma, stage I).

2.3 Out-of-Sample Perturbation Prediction (PBMC IFN-β)

Paper Table 5 · Leave-One-Out PCC ↑ / R² ↑

Leave-one-cell-type-out counterfactual prediction of IFN-β stimulation response across all seven PBMC cell types.

Method CD8T CD14+Mono FCGR3A+Mono B CD4T Dendritic NK Average
control 0.94 / 0.88 0.75 / 0.32 0.78 / 0.38 0.91 / 0.81 0.94 / 0.87 0.79 / 0.51 0.93 / 0.84 0.86 / 0.66
scDisInFact 0.85 / 0.43 0.77 / 0.47 0.71 / 0.43 0.84 / 0.45 0.85 / 0.43 0.76 / 0.47 0.79 / 0.44 0.80 / 0.45
MFM 0.78 / 0.60 0.77 / 0.53 0.56 / 0.08 0.70 / 0.49 0.70 / 0.49 0.72 / 0.51 0.72 / 0.51 0.71 / 0.46
trVAE 0.98 / 0.96 0.82 / 0.39 0.83 / 0.32 0.90 / 0.78 0.97 / 0.94 0.83 / 0.46 0.97 / 0.92 0.90 / 0.68
scGEN 0.96 / 0.88 0.95 / 0.80 0.93 / 0.77 0.91 / 0.83 0.92 / 0.85 0.96 / 0.89 0.93 / 0.84 0.94 / 0.84
CellOT 0.99 / 0.97 0.99 / 0.98 0.77 / 0.02 0.95 / 0.89 0.97 / 0.94 0.96 / 0.88 0.96 / 0.90 0.94 / 0.80
SAVE 0.98 / 0.97 0.97 / 0.89 0.91 / 0.53 0.97 / 0.94 0.96 / 0.97 0.96 / 0.81 0.96 / 0.90 0.96 / 0.86
Figure 5. Perturbation prediction on PBMC IFN-β. Top: UMAP alignment against ground-truth stimulated cells across all 7 held-out cell types; Bottom: predicted expression distributions of key interferon-response markers (ISG15, TNFSF10).

2.4 Batch Effect Correction & Block Size Scaling

Paper Table 4 (scIB ↑) · RTX 3090 Efficiency

Without explicit batch-integration objectives, SAVE's condition-aware latent space disentangles technical batch variation from biological identity.

Method Lung Bio Lung Batch Lung scIB Heart Bio Heart Batch Heart scIB PBMC Bio PBMC Batch PBMC scIB
Scanorama 0.70 0.88 0.77 0.72 0.82 0.76 0.71 0.93 0.80
Harmony 0.65 0.93 0.76 0.76 0.89 0.81 0.71 0.95 0.80
scVI 0.58 0.83 0.68 0.66 0.81 0.72 0.47 0.78 0.59
trVAE 0.69 0.90 0.78 0.75 0.83 0.78 0.75 0.91 0.82
SAVE 0.73 0.93 0.81 0.76 0.86 0.80 0.75 0.95 0.83

Training efficiency and block size scaling measured on Heart (19,112 genes) on a single NVIDIA RTX 3090 (24 GB VRAM):

Block Size (K) Num. Blocks (L) Training Time Speedup Heart WD ↓ Heart MMD ↓
1 (naive gene-level) 19,112 2391.2 min 1.0× — —
1600 12 16.4 min 146× 9.64 0.65
3200 (default) 6 12.5 min 191× 8.30 0.63
5600 4 9.0 min 266× 8.41 0.63

Attention Ablation: Removing Gene Block Attention (SAVE w/o Att.) degrades Heart conditional generation performance from WD 8.30 / MMD 0.63 to WD 8.89 / MMD 0.65.

03 / Biological Interpretability

Condition-modulated attention recovers coherent biological pathways.

Extracting condition-modulated attention weights from the Flow Matching Transformer on Heart (16,000 HVGs partitioned into 5 blocks of size 3,200) shows that SAVE attends to biologically meaningful gene programs across cell types and batches.

Figure 6. Condition-modulated Gene Block attention across Heart cell types and batches. Dominant attended semantic gene block for each observed (batch, cell_type) pair.

Physiological Alignment

Atrial and ventricular cardiomyocytes consistently attend to Block 1 (Fatty Acid β-Oxidation), matching the heart's primary lipid metabolism signature across batches.

Functional Specificity

Endothelial cells and fibroblasts dynamically shift attention toward Block 4 (Adenylate Cyclase-Activating GPCR Signaling) and Block 3 (Protein Localization to Plasma Membrane), reflecting extracellular signaling and structural roles.

Biological Filtering

Block 2 (Broad/General Cellular Functions without a dominant enriched pathway) is universally suppressed across conditions, demonstrating autonomous filtering of non-discriminative gene programs.

04 / Verified ICLR 2026 Reproduction

One-click Google Colab & local CLI verification.

Reproducibility · Colab Pro+ Verified

All 5 Tables (11 Datasets) Completed

Reproduce all 5 ICLR 2026 tables in Google Colab or locally

All five quantitative benchmarks (Tables 1–5, 11 datasets) completed end-to-end on Colab Pro+ (NVIDIA RTX PRO 6000 Blackwell Server Edition, 94.97 GB VRAM, PyTorch 2.14.1+cu130, CUDA 13.0) with TF32, SDPA, bfloat16 Flow Matching autocast, and automatic GPU/CPU parallelism (88m 15s wall-clock for parallel Tables 1–4; ~7m for single-run Heart).

Ref: reproduce-iclr2026@307c925 (merged to main) Workers: gpu_workers="auto" (4) · cpu_workers="auto" (8) Notebook: notebooks/SAVE_ICLR2026_Reproduce_Colab.ipynb
Benchmark Key Metric(s) Paper Target Colab G4 Reproduced
Table 1 (pbmc3k / dentategyrus / tabula_muris) WD ↓ / MMD ↓ 19.14/1.80 · 9.16/0.17 · 10.86/0.04 12.76/0.48 · 8.08/0.11 · 10.65/0.04
Table 2 (heart / pbmc / lung_atlas) WD ↓ / MMD ↓ 8.30/0.63 · 5.37/0.29 · 4.37/1.14 8.83/0.64 · 5.39/0.30 · 4.42/1.16
Table 3 (lung_cancer Seen / Unseen) WD ↓ / MMD ↓ 3.10/1.06 · 4.63/2.11 3.07/1.08 · 5.39/2.16
Table 4 (lung_atlas / heart / pbmc BEC) scIB ↑ 0.8128 · 0.7983 · 0.8309 0.8097 · 0.8019 · 0.8125
Table 5 (pbmc_ifn 7-fold LOO Average) PCC ↑ / R² ↑ / WD ↓ 0.96 / 0.86 / 5.15 0.943 / 0.807 / 5.77
Local CLI Quickstart (Tables 1–5)
git clone https://github.com/fdu-wangfeilab/sc-save.git && cd sc-save
uv sync --extra all   # or: pip install -e ".[all]"

# Run all 5 paper benchmarks end-to-end (or pass --tasks table2)
python examples/reproduce_iclr2026.py \
  --tasks all \
  --data-root ./data \
  --output-root ./outputs/iclr2026 \
  --gpu-workers auto \
  --cpu-workers auto \
  --prep-workers 4 \
  --device 0

05 / Citation

Cite this work.

If you use SAVE or this codebase in your research, please cite our ICLR 2026 paper.

BibTeX
@inproceedings{
li2026save,
title={{SAVE}: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention},
author={Jiahao Li and Jiayi Dong and Peng Ye and Xiaochi Zhou and Haohai Lu and Fei Wang},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=l7QEoK4uDP}
}
Figure