Defining Cell Type Inter-Sample Consistency Metrics

Author
Affiliation

Josep Garnica

University of Geneva

Published

July 16, 2026

1 Defining ISC Metrics and Evaluation Criteria

This tutorial summarizes the manuscript rationale for how inter-sample consistency (ISC) is defined and evaluated in scTypeEval.

We designed a panel of ISC metrics adapted from commonly used cluster validation approaches:

  • average silhouette width-based metrics: silhouette and 2label_silhouette
  • graph-based neighborhood purity
  • clustering agreement
  • nearest-medoid agreement

These families are commonly used to quantify compactness and separability at the single-cell level within one dataset. In ISC, they are adapted to explicitly account for variation across samples or datasets (see Methods in the manuscript).

NoteKey Idea

ISC does not score cell types directly from raw expression. It first evaluates cross-sample relationships in a dissimilarity space, then computes consistency metrics on top of that space.

2 Input to ISC Metrics: Cell Type Dissimilarity Matrices

All ISC metrics take as input a cell-type by cell-type dissimilarity matrix. In these matrices, expression-defined cell type profiles are compared across samples.

Conceptually, the pipeline is:

  1. define cell type labels per sample
  2. compute cross-sample cell type dissimilarities
  3. apply ISC metrics on the resulting matrix

3 Two Ways to Build Dissimilarity Matrices

3.1 1. Single-cell distribution-based dissimilarity

This approach compares distributions of single cells across samples, for example with:

  • WasserStein

3.2 2. Pseudobulk or classifier-based dissimilarity

This approach compares aggregated profiles or reciprocal prediction behavior across samples, including:

  • Pseudobulk:Euclidean
  • Pseudobulk:Cosine
  • Pseudobulk:Pearson
  • reciprocal-classification dissimilarities (recip_classif:Match, recip_classif:Score)

4 Reciprocal Classification Dissimilarity

For reciprocal classification, each pair of samples is evaluated bidirectionally:

  1. train classifier from sample A and predict in sample B
  2. train classifier from sample B and predict in sample A
  3. combine reciprocal predictions into dissimilarities

The reciprocal outcomes can be encoded as:

  • binary match-based dissimilarity (recip_classif:Match)
  • score-based dissimilarity (recip_classif:Score)

Overview of the ISC framework. Cell types are represented per sample either as single-cell expression distributions or as aggregated pseudobulk profiles. Pairwise dissimilarities between all cell type-sample combinations are computed using distribution-based distances (Wasserstein/EMD) or pseudobulk-based measures (Euclidean distance, cosine similarity, Pearson correlation, or reciprocal classification-based dissimilarities). These dissimilarities define a cell type-by-cell type matrix spanning all samples, which serves as input to sample-aware consistency metrics, including silhouette, 2-label silhouette, nearest-medoid agreement, clustering agreement, and neighborhood purity. Each metric yields a normalized consistency score per cell type, with higher values indicating greater inter-sample reproducibility.

Definition of inter-sample consistency (ISC) metrics.