Benchmarking of ISC metrics

Author
Affiliation

Josep Garnica

University of Geneva

Published

July 16, 2026

1 Benchmarking of ISC metrics

This tutorial summarizes the benchmarking framework used to evaluate Inter-Sample Consistency (ISC) metrics for cell type annotation quality in scRNA-seq.

Each ISC metric returns one scalar score per cell type. Higher values indicate stronger inter-sample consistency, meaning that cell type definitions are more reproducible across donors or samples.

NoteWhy this matters

Because ISC is built on cross-sample dissimilarity, donor-to-donor variability is part of the metric definition rather than a downstream confounder.

2 Evaluation framework: 8 tasks

To systematically compare ISC behaviors, the benchmark defined eight evaluation criteria (tasks), each designed to probe a different failure mode or robustness property:

  1. sensitivity to cell type signal degradation by progressively shuffling original labels
  2. sensitivity to cell type over-partitioning by artificially splitting populations
  3. robustness to annotation granularity (coarse vs fine labels)
  4. robustness to cellular complexity (low vs high inter-cell-type variability tissues)
  5. robustness to number of samples
  6. robustness to cells per cell type
  7. robustness to batch effects
  8. robustness to biological perturbations (condition-induced shifts)

For each task, metric performance was summarized with a normalized task performance score from 0 (worst) to 1 (best), enabling direct comparison across tasks.

Benchmarking design used to evaluate ISC metric behavior across eight criteria. Semi-synthetic perturbations were applied to 31 real scRNA-seq datasets to assess: (1) sensitivity to cell type signal degradation via progressive label reshuffling; (2) sensitivity to cell type over-partitioning by splitting a cell type into closely related subtypes; (3) robustness to annotation granularity through hierarchical label merging; (4) robustness to cellular complexity by removing subsets of cell types; (5) robustness to the number of samples via sample subsampling; (6) robustness to the number of cells per cell type via cell subsampling; (7) robustness to batch effects by comparing within-batch and pooled analyses; and (8) robustness to biological perturbations by comparing condition-specific and combined datasets.

Figure 2B. Evaluation tasks used to benchmark ISC metrics.

3 Benchmark design across real datasets

The benchmark evaluated:

  • 6 ISC metrics
  • 6 dissimilarity functions
  • 31 scRNA-seq datasets
  • 17 studies
  • 6 biological systems (blood, colorectal cancer, melanoma, breast cancer, basal cell carcinoma, respiratory tissue)

Each dataset represented one batch and a relatively homogeneous biological condition. This reduced strong confounding from large batch/condition structure in baseline comparisons.

The eight tasks were then generated through controlled perturbations or combinations of these datasets to produce semi-synthetic scenarios with known changes in annotation quality, composition, batch structure, and biological heterogeneity.

4 Main benchmark outcomes

4.1 Global ranking

A global ranking across tasks identified the top-performing metric+dissimilarity combinations as:

  1. silhouette + RCM
  2. 2-label silhouette + RCM
  3. silhouette + pseudobulk cosine

This indicates that silhouette-family ISC scores provide the best overall balance of sensitivity and robustness, especially when paired with reciprocal classification matching (RCM).

4.2 Efficiency and scalability

Pseudobulk-based methods were markedly more efficient than distribution-based distances:

  • typically under 1 second per dataset
  • typically less than 100 MB memory

Wasserstein distance performed competitively in some tasks, but its higher runtime and memory demands did not improve overall benchmark ranking enough to offset the computational cost.

(a) Heatmap summarizing the performance of 25 method combinations (5 inter-sample consistency metrics with 5 dissimilarity functions) across eight evaluation tasks. Average similarity metric and Pseudobulk:Pearson dissimilarity were removed from visualization, given their high correlation with 2-label silhouette and Pseudobulk:Cosine, respectively. Each row represents a method combination ordered by overall weighted score (left column). Task columns (middle) show median scores as colored circles, with circle size proportional to score magnitude (0-1 scale; legend) and color representing within-task rank (dark blue = best, white = worst). Resource columns (right) display median computational time (ms) and peak memory usage (MB) as colored bars, with darker shades indicating higher resource consumption. Overall scores represent weighted averages (Task 1: 20%, Task 2: 20%, Tasks 3-8: 10% each). n = 31 baseline datasets from 17 independent studies. (b) Method-specific responses to perturbation intensity across the eight evaluation tasks. Each column represents one task (labeled at top); rows show the best-performing (top) and worst-performing (bottom) method for that task. Y-axis: inter-sample consistency score. X-axis: perturbation rate or condition. Continuous perturbations (Tasks 1, 3-6) are displayed as line plots with ribbons showing 5th–95th percentile (light blue) and 25th–75th percentile (dark blue) ranges across all cell types and datasets. Two-level perturbations (Tasks 2, 7-8) are shown as box plots with jittered points comparing baseline (Split/Single) versus perturbed (Complete/Multi/Paired) conditions. Task 2 (Over-partitioning): Split = artificially split cell type, Complete = original annotation. Tasks 7-8: Single = within-condition consistency, Multi/Paired = cross-condition consistency.

Benchmarking of inter-sample consistency metrics across annotation quality and dataset complexity perturbations.

5 Decomposing metric and dissimilarity contributions

To separate scoring-function effects from representation effects, benchmark performance was decomposed into marginal contributions.

Key findings:

  • 2-label silhouette contributed most to task 1 (signal degradation sensitivity)
  • silhouette contributed most to task 2 (over-partitioning sensitivity)
  • RCM dissimilarity performed best for task 2 and for robustness to batch/condition shifts (tasks 7 and 8)
  • pseudobulk cosine and Euclidean dissimilarities performed best for task 1

These results support two complementary ISC dimensions:

  1. task-1-oriented dimension: best captured by 2-label silhouette + pseudobulk cosine
  2. task-2-oriented dimension: best captured by silhouette + RCM

Most methods were robust to granularity, cellular complexity, sample number, and cells per cell type. Neighborhood purity was the main exception when sample count approached neighborhood size.

Residual-effect decomposition of benchmark performance. Heatmaps show centered main effects (Z-scores) for consistency metrics (top) and dissimilarity methods (bottom) across the eight evaluation tasks. Cell numbers indicate within-task ranks. Warm colors indicate positive residual effects, and cool colors indicate negative residual effects.

Component-level effects across benchmark tasks.

6 Stability of ranking with dataset subsampling

To test robustness to dataset selection, ranking stability was evaluated across progressively larger subsets of datasets.

  • rank agreement increased with subset size
  • median correlation plateaued near 0.95 by 10 datasets
  • changes were minimal beyond that point

This supports that the 31-dataset benchmark is sufficient to produce stable, representative metric rankings.

Saturation analysis of ranking stability as a function of dataset number. Left, distribution of Spearman rank correlations between method rankings computed from disjoint dataset subsets at increasing subset size (Number of datasets). Right, interquartile range (IQR) of these correlations versus subset size; blue points indicate observed values, and the red curve shows the fitted trend extended to larger subset sizes by a generalized additive model (GAM). Open red circles indicate extrapolated IQR beyond the observed range.

Benchmark ranking stability saturation.

7 Practical recommendations

From the benchmark, two combinations are especially useful in practice:

  1. global-structure sensitivity: 2-label silhouette + pseudobulk cosine
  2. local-structure / over-partitioning sensitivity: silhouette + RCM

Using both provides complementary evidence when diagnosing annotation consistency problems across samples.

8 Take-home message

The benchmarking analysis identifies ISC metric configurations that are both robust and broadly transferable across biological systems, evaluation tasks, and dataset settings. This provides a practical and quantitative basis for evaluating and improving scRNA-seq cell type annotations.

10 More information