Benchmark

This subpackage contains functions to compare labels.

AI compare cell type labels

This module contains functions that use LLMs to compare two cell type labels.

ai_compare_cell_type_labels_pairwise(adata, ...)

Compare cell type labels by finding unique combinations between labels in cols1 and cols2, applying the comparison, and mapping the results back to adata.obs.

Calculate label agreement metrics

This module contains functions to calculate reliability metrics on columns adata.

kappa_adata(adata, cols)	Calculate pairwise Cohen's Kappa, average pairwise Kappa, and Fleiss' Kappa for categorical data in adata.obs[cols].
krippendorff_alpha_adata(adata, cols[, ...])	Calculate Krippendorff's Alpha for categorical data in adata.obs[cols].

Utility functions

This module contains utility functions for label comparison.

create_label_df(adata, cols1, cols2)

Creates a DataFrame of unique label combinations from the specified columns in cols1 and cols2, only including combinations that exist in adata.obs[[*cols1, *cols2]].