1,536-dim NCBI Gene summaries (GenePT) → OT soft-patched blocks
ICLR 2026 · Fudan University
SAVE: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention
College of Computer Science and Artificial Intelligence, Fudan University · *Corresponding author
AdaLN + condition masking (p = 0.6) & multi-condition CFG across 1–5 covariates
12.5 min vs. 2,391.2 min on RTX 3090 (~7 min on Colab G4, 20-step ODE)
Unified config across generation, BEC & perturbation (11 datasets reproduced)
01 / Overview & Architecture
LLM biological knowledge meets Latent Flow Matching at 191× efficiency.
SAVE brings three core advantages to multi-condition single-cell RNA-seq modeling: LLM biological knowledge (NCBI Gene summaries clustered via Optimal Transport into pathway-coherent Gene Blocks), a Latent Flow Matching tech stack (Gene-Block Attention VAE + DiT Optimal-Transport Flow Matching with masked conditioning and CFG), and 191× training speedup over flat gene-level Transformers.
LLM Biological Knowledge
Encodes curated NCBI Gene functional summaries via GenePT (1,536-dim semantic embeddings) and solves iterative Optimal Transport clustering to partition ~20,000 unordered genes into L balanced, pathway-coherent Gene Blocks (K = 3,200).
Gene-Block VAE + DiT OT-FM
Compresses structured gene blocks (X ∈ ℝN×L×K) into pathway-structured latents (z ∈ ℝN×L×d), then trains DiT-based Conditional OT Flow Matching (AdaLN, condition masking p = 0.6, multi-condition CFG) across 1–5 covariates and 8 benchmarks with one fixed config.
191× Speedup & 20-Step Sampling
Grouping G ≈ 19,112 genes into L = 6 blocks collapses attention complexity from O(G2) to O(L2), cutting single-RTX-3090 training from 2,391.2 min (~40 h) to 12.5 min (266× / 9.0 min at K = 5,600; ~7 min on Colab G4).
LLM Semantic Gene Blocks vs. ViT Patchify
Standard Transformer models either attend over ~20,000 flat genes (O(G2) memory/time bottleneck) or lack a spatial 2D neighborhood for Vision Transformer (ViT) grid patching. SAVE replaces spatial proximity with LLM semantic similarity + balanced Optimal Transport soft-patching before feeding structured tokens into the Latent VAE + Latent Flow Matching stack.
- Stage 1 — LLM Semantic Embedding: Curated NCBI Gene text descriptions are encoded with a pre-trained LLM (
text-embedding-ada-002, GenePT) into continuous 1,536-dim semantic vectors gi, injecting prior biological functional knowledge directly into tokenization. - Stage 2 — Balanced Optimal Transport Soft-Patching → Latent FM: Iterative Optimal Transport minimizes squared Euclidean cost Cij = ‖gi − cj(t)‖22 under uniform marginal constraints (T1L = a, T⊤1G = b), yielding non-overlapping, equal-size functional blocks X ∈ ℝN×L×K that serve as compact tokens for Latent VAE + OT Flow Matching.
| Dimension | ViT Patchify (Vision Domain) | SAVE Gene Block Mechanism (Genomics Domain) |
|---|---|---|
| Input Data Structure | Structured 2D pixel grid | Unordered, high-dimensional & sparse gene set (~20,000 genes) |
| Grouping Basis | Physical proximity (Euclidean grid) | Biological semantic similarity (pre-trained LLM prior on NCBI descriptions) |
| Patching Type | Hard-patching (fixed spatial grid) | Semantic soft-patching (iterative balanced Optimal Transport clustering) |
| Token Properties | Functionally agnostic image patches | Functionally coupled pathway blocks (coherent gene programs) |
| Downstream Stack & Efficiency | Flat patch ViT attention | Latent VAE + Latent OT Flow Matching (O(L2) attention, 191× training speedup: 12.5 min vs. 2,391.2 min on RTX 3090) |
End-to-End Architecture: Latent VAE + Latent Flow Matching
Once LLM embeddings and Optimal Transport partition profiles into L pathway-coherent Gene Blocks, a Gene-Block Attention VAE compresses expressions into compact latents z ∈ ℝN×L×d where masked DiT Optimal-Transport Flow Matching learns conditional cellular trajectories in just 20 ODE steps.
02 / Quantitative & Qualitative Results
Consistent gains across generation, extrapolation, integration, and perturbation.
Paper benchmark comparisons across multi-platform scRNA-seq cohorts (bold = best, italic = second best). Full per-dataset Colab Pro+ G4 reproduction numbers are summarized in Section 04.
2.1 Dual-Condition Generation (batch + cell_type)
Paper Table 2 · WD ↓ & MMD ↓
Jointly conditioning on sequencing batch and cell type across three multi-platform scRNA-seq cohorts (Heart, PBMC, and Lung Atlas).
| Method | Heart WD ↓ | Heart MMD ↓ | PBMC WD ↓ | PBMC MMD ↓ | Lung Atlas WD ↓ | Lung Atlas MMD ↓ |
|---|---|---|---|---|---|---|
| scVI | 19.18 | 1.19 | 17.75 | 0.95 | 22.20 | 2.27 |
| CFGen | 12.57 | 0.66 | 17.41 | 0.65 | 28.02 | 1.76 |
| scDiffusion | 20.82 | 0.94 | 11.38 | 0.48 | 13.89 | 1.71 |
| SAVE | 8.30 | 0.63 | 5.37 | 0.29 | 4.37 | 1.14 |
2.2 Multi-Condition Generation & Unseen Extrapolation (5 Covariates)
Paper Table 3 · Lung Cancer (Salcher et al. 2022)Evaluated simultaneously across five covariates (assay, development_stage, disease, uicc_stage, cell_type), including held-out unseen covariate combinations.
| Method | Seen WD ↓ | Seen MMD ↓ | Unseen WD ↓ | Unseen MMD ↓ |
|---|---|---|---|---|
| CFGen | 16.94 ± 4.19 | 1.49 ± 1.80 | 23.67 ± 3.08 | 2.76 ± 2.64 |
| scDiffusion | 5.27 ± 2.14 | 1.06 ± 1.59 | 5.29 ± 2.03 | 1.87 ± 2.31 |
| SAVE | 3.10 ± 0.96 | 1.06 ± 1.48 | 4.63 ± 0.95 | 2.11 ± 2.11 |
2.3 Out-of-Sample Perturbation Prediction (PBMC IFN-β)
Paper Table 5 · Leave-One-Out PCC ↑ / R² ↑Leave-one-cell-type-out counterfactual prediction of IFN-β stimulation response across all seven PBMC cell types.
| Method | CD8T | CD14+Mono | FCGR3A+Mono | B | CD4T | Dendritic | NK | Average |
|---|---|---|---|---|---|---|---|---|
| control | 0.94 / 0.88 | 0.75 / 0.32 | 0.78 / 0.38 | 0.91 / 0.81 | 0.94 / 0.87 | 0.79 / 0.51 | 0.93 / 0.84 | 0.86 / 0.66 |
| scDisInFact | 0.85 / 0.43 | 0.77 / 0.47 | 0.71 / 0.43 | 0.84 / 0.45 | 0.85 / 0.43 | 0.76 / 0.47 | 0.79 / 0.44 | 0.80 / 0.45 |
| MFM | 0.78 / 0.60 | 0.77 / 0.53 | 0.56 / 0.08 | 0.70 / 0.49 | 0.70 / 0.49 | 0.72 / 0.51 | 0.72 / 0.51 | 0.71 / 0.46 |
| trVAE | 0.98 / 0.96 | 0.82 / 0.39 | 0.83 / 0.32 | 0.90 / 0.78 | 0.97 / 0.94 | 0.83 / 0.46 | 0.97 / 0.92 | 0.90 / 0.68 |
| scGEN | 0.96 / 0.88 | 0.95 / 0.80 | 0.93 / 0.77 | 0.91 / 0.83 | 0.92 / 0.85 | 0.96 / 0.89 | 0.93 / 0.84 | 0.94 / 0.84 |
| CellOT | 0.99 / 0.97 | 0.99 / 0.98 | 0.77 / 0.02 | 0.95 / 0.89 | 0.97 / 0.94 | 0.96 / 0.88 | 0.96 / 0.90 | 0.94 / 0.80 |
| SAVE | 0.98 / 0.97 | 0.97 / 0.89 | 0.91 / 0.53 | 0.97 / 0.94 | 0.96 / 0.97 | 0.96 / 0.81 | 0.96 / 0.90 | 0.96 / 0.86 |
ISG15, TNFSF10).2.4 Batch Effect Correction & Block Size Scaling
Paper Table 4 (scIB ↑) · RTX 3090 EfficiencyWithout explicit batch-integration objectives, SAVE's condition-aware latent space disentangles technical batch variation from biological identity.
| Method | Lung Bio | Lung Batch | Lung scIB | Heart Bio | Heart Batch | Heart scIB | PBMC Bio | PBMC Batch | PBMC scIB |
|---|---|---|---|---|---|---|---|---|---|
| Scanorama | 0.70 | 0.88 | 0.77 | 0.72 | 0.82 | 0.76 | 0.71 | 0.93 | 0.80 |
| Harmony | 0.65 | 0.93 | 0.76 | 0.76 | 0.89 | 0.81 | 0.71 | 0.95 | 0.80 |
| scVI | 0.58 | 0.83 | 0.68 | 0.66 | 0.81 | 0.72 | 0.47 | 0.78 | 0.59 |
| trVAE | 0.69 | 0.90 | 0.78 | 0.75 | 0.83 | 0.78 | 0.75 | 0.91 | 0.82 |
| SAVE | 0.73 | 0.93 | 0.81 | 0.76 | 0.86 | 0.80 | 0.75 | 0.95 | 0.83 |
Training efficiency and block size scaling measured on Heart (19,112 genes) on a single NVIDIA RTX 3090 (24 GB VRAM):
| Block Size (K) | Num. Blocks (L) | Training Time | Speedup | Heart WD ↓ | Heart MMD ↓ |
|---|---|---|---|---|---|
1 (naive gene-level) |
19,112 | 2391.2 min | 1.0× | — | — |
1600 |
12 | 16.4 min | 146× | 9.64 | 0.65 |
3200 (default) |
6 | 12.5 min | 191× | 8.30 | 0.63 |
5600 |
4 | 9.0 min | 266× | 8.41 | 0.63 |
Attention Ablation: Removing Gene Block Attention (SAVE w/o Att.) degrades Heart conditional generation performance from WD 8.30 / MMD 0.63 to WD 8.89 / MMD 0.65.
03 / Biological Interpretability
Condition-modulated attention recovers coherent biological pathways.
Extracting condition-modulated attention weights from the Flow Matching Transformer on Heart (16,000 HVGs partitioned into 5 blocks of size 3,200) shows that SAVE attends to biologically meaningful gene programs across cell types and batches.
(batch, cell_type) pair.Physiological Alignment
Atrial and ventricular cardiomyocytes consistently attend to Block 1 (Fatty Acid β-Oxidation), matching the heart's primary lipid metabolism signature across batches.
Functional Specificity
Endothelial cells and fibroblasts dynamically shift attention toward Block 4 (Adenylate Cyclase-Activating GPCR Signaling) and Block 3 (Protein Localization to Plasma Membrane), reflecting extracellular signaling and structural roles.
Biological Filtering
Block 2 (Broad/General Cellular Functions without a dominant enriched pathway) is universally suppressed across conditions, demonstrating autonomous filtering of non-discriminative gene programs.
04 / Verified ICLR 2026 Reproduction
One-click Google Colab & local CLI verification.
Reproducibility · Colab Pro+ Verified
All 5 Tables (11 Datasets) CompletedReproduce all 5 ICLR 2026 tables in Google Colab or locally
All five quantitative benchmarks (Tables 1–5, 11 datasets) completed end-to-end on Colab Pro+ (NVIDIA RTX PRO 6000 Blackwell Server Edition, 94.97 GB VRAM, PyTorch 2.14.1+cu130, CUDA 13.0) with TF32, SDPA, bfloat16 Flow Matching autocast, and automatic GPU/CPU parallelism (88m 15s wall-clock for parallel Tables 1–4; ~7m for single-run Heart).
| Benchmark | Key Metric(s) | Paper Target | Colab G4 Reproduced |
|---|---|---|---|
Table 1 (pbmc3k / dentategyrus / tabula_muris) |
WD ↓ / MMD ↓ | 19.14/1.80 · 9.16/0.17 · 10.86/0.04 | 12.76/0.48 · 8.08/0.11 · 10.65/0.04 |
Table 2 (heart / pbmc / lung_atlas) |
WD ↓ / MMD ↓ | 8.30/0.63 · 5.37/0.29 · 4.37/1.14 | 8.83/0.64 · 5.39/0.30 · 4.42/1.16 |
Table 3 (lung_cancer Seen / Unseen) |
WD ↓ / MMD ↓ | 3.10/1.06 · 4.63/2.11 | 3.07/1.08 · 5.39/2.16 |
Table 4 (lung_atlas / heart / pbmc BEC) |
scIB ↑ | 0.8128 · 0.7983 · 0.8309 | 0.8097 · 0.8019 · 0.8125 |
Table 5 (pbmc_ifn 7-fold LOO Average) |
PCC ↑ / R² ↑ / WD ↓ | 0.96 / 0.86 / 5.15 | 0.943 / 0.807 / 5.77 |
git clone https://github.com/fdu-wangfeilab/sc-save.git && cd sc-save
uv sync --extra all # or: pip install -e ".[all]"
# Run all 5 paper benchmarks end-to-end (or pass --tasks table2)
python examples/reproduce_iclr2026.py \
--tasks all \
--data-root ./data \
--output-root ./outputs/iclr2026 \
--gpu-workers auto \
--cpu-workers auto \
--prep-workers 4 \
--device 0
05 / Citation
Cite this work.
If you use SAVE or this codebase in your research, please cite our ICLR 2026 paper.
@inproceedings{
li2026save,
title={{SAVE}: A Generalizable Framework for Multi-Condition Single-Cell Generation with Gene Block Attention},
author={Jiahao Li and Jiayi Dong and Peng Ye and Xiaochi Zhou and Haohai Lu and Fei Wang},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=l7QEoK4uDP}
}