We employed a straightforward ranking-based strategy to evaluate the relative performance of the selected methods across the six metrics. Specifically, within each dataset, the methods were ranked on the basis of their raw outputs for each metric. The overall ranking of a method in a given dataset was then calculated by averaging its ranks across the six metrics. To obtain a final ranking across all the datasets, we averaged the overall rankings from each dataset. This ranking-based approach addresses potential inconsistencies in value ranges between different metrics, ensuring a fair comparison.
To facilitate fair and convenient comparison between newly developed tools and those included in our benchmark, we provide the raw results of all benchmarked methods across six evaluation metrics. Once users obtain prediction results from their own models, they can use the code provided in our Podman image (https://github.com/bm2-lab/scPerturBench) to compute performance on the same six metrics and directly compare their results with those of existing methods.
We calculated the metrics in two ways: for all genes and for the top 100 most differentially expressed genes (DEGs), ranked by absolute effect sizes. The top 100 DEGs are typically intervention-specific and represent the most significant responses to the perturbations. Metrics calculated based on this smaller, biologically meaningful gene set are generally preferred in the field, as genes with modest expression changes contribute little to understanding perturbation effects. For the evaluation based on all genes, we did not compute the Wasserstein distance and Common-DEGs metrics for the following reasons:
Wasserstein distance becomes computationally expensive and prone to errors in high-dimensional settings;
Computing the Common-DEGs metric is not informative in this context, as all methods yield a perfect score (i.e., a value of 1).
All the data used to generate our plots are available on our GitHub (https://github.com/bm2-lab/scPerturBench).