Datasets & Pseudobulks

How single cells become conditioned training/inference units: SEACells metacells, multi-level categorical grouping, paired pseudobulking for delta/velocity training, the parquet build, and the DuckDB-backed random-access store.

Metacells & pseudobulks

run_meta_cells

run_meta_cells(adata, obsm, groupby=None, max_fragments=3000000, group_size_cutoff=200, min_seacells_per_group=5, large_group_split_threshold=3000, n_fragment_key='n_fragments', meta_fold=75, random_state=0, verbose=False)

Run meta cell calculation on the given AnnData object. This function groups cells by the specified groupby key and calculates meta cells for each group. It also handles the case where the number of fragments exceeds the specified maximum number of fragments.

Parameters:

Name	Type	Description	Default
adata	AnnData	The AnnData object containing the data.	required
groupby	str	The key in adata.obs to group cells by.	None
obsm	str	The key in adata.obsm to use for the kernel matrix.	required
max_fragments	int	The maximum number of fragments allowed for each meta cell.	3000000
group_size_cutoff	int	If specified, groups with fewer cells than this cutoff will be skipped. If None, all groups will be processed regardless of size.	200
min_seacells_per_group	int	The minimum number of SEACells to use for each group. If the group has fewer cells, it will use that number. This prevents errors when a group is too small to form a meta cell.	5
large_group_split_threshold	int	If the group size is larger than this threshold, it will be randomly split into smaller groups.	3000
n_fragment_key	str	The key in adata.obs to use for the number of fragments.	'n_fragments'
meta_fold	int	The number of folds to group cells into.	75

Returns:

Type	Description
Series	A series containing the meta cell assignments for each cell.

prepare_multi_level_categorical_groups

prepare_multi_level_categorical_groups(cell_meta, embedding, group_cols, hard_group_cols=None, min_cell_count=500, dendrogram_method='ward', _is_top=True)

Group multiple levels of categorical variables into a single categorical variable by using a tree-based approach on cell embedding, with a minimum cell count threshold.

Specifically, this function: 1. Iteratively groups cells based on the each level of categorical variables. 2. Within each level, it uses hierarchical clustering on group average embeddings to split data into groups if the resulting group size passes the minimum cell count threshold. 3. It returns a final cell-to-group mapping and a dictionary of group-to-categorical mappings.

Parameters:

Name	Type	Description	Default
cell_meta	DataFrame	DataFrame containing cell metadata with categorical variables.	required
embedding	DataFrame	DataFrame containing cell embeddings. Index should match with cell_meta.	required
group_cols	list	List of categorical variable names to group by, in order of priority.	required
min_cell_count	int	Minimum number of cells required in a group for further splitting.	500
dendrogram_method	str	Method to use for hierarchical clustering. Default is 'ward'.	'ward'
_is_top	bool	Flag to indicate if this is the top-level call. Do not set this manually. It is used internally for recursive calls.	True

Returns:

Name	Type	Description
cell_to_group	Series	Series mapping each cell to its corresponding group name.
group_to_cells	dict	Dictionary mapping group names to lists of cell indices.

EnsemblePairedPseudobulker

Bases: PseudobulkerMixin

Generate paired pseudobulks from predefined pseudobulk records, the predefined pseudobulk will be used as p1, while p0 will be randomly sampled from all available pseudobulks.

create_pseudobulk_records_from_design

create_pseudobulk_records_from_design(designs) -> dict[str, dict]

Create pseudobulk pairs for each pseudobulk in the self.pseudobulk_records.

Data build & storage

CSRRowMerge

AdataPseudobulkMerger

merge

merge()

Merge all chroms for all samples and groups.

GenomeChunkDatasetGenerator

A generator class for creating genome-chunk ray dataset for single-cell or bulk data.

Parameters:

Name	Type	Description	Default
genome	Union[str, Genome]	The genome associated with the dataset.	required

add_bigwig

add_bigwig(prefix, name, path, sparse=True, compress_level=5, **kwargs)

Add BigWig files. BigWig will be aggregated based on the prefix.

Parameters:

Name	Type	Description	Default
kwargs	Dict[str, str]	The dataset name and the path to the BigWig file.	{}

add_snap_adata

add_snap_adata(prefix, path, barcode_whitelist=None, **ds_kwargs)

Add SnapATAC AnnData dataset that contains insersion sites as a sparse matrix.

Parameters:

Name	Type	Description	Default
prefix	str	The dataset name.	required
path	str	The path to the AnnData file.	required
barcode_whitelist	List[str]	The list of barcodes to include in the dataset, by default None.	None

add_chrom_sparse

add_chrom_sparse(prefix, dataset_dir, **ds_kwargs)

Add chromatin sparse matrix. Chromatin sparse matrix will be aggregated based on the prefix.

Parameters:

Name	Type	Description	Default
prefix	str	The dataset name.	required
dataset_dir	str	The path to the chromatin sparse matrix dataset_dir.	required

add_allc

add_allc(prefix, name, path, sparse=True, compress_level=5, **ds_kwargs)

Add ALLC files. ALLC will be aggregated based on the prefix.

Parameters:

Name	Type	Description	Default
prefix	str	The dataset name.	required
name	str	The name of the ALLC file.	required
path	str	The path to the ALLC file.	required
sparse	bool	Whether the sample-by-pos matrix is sparse, by default True.	True

generate

generate() -> None

Generate the ray dataset.

GenomeParquetDB

Bases: GenomeParquetDBNoParallel

A DuckDB-based interface for querying genomic regions from a Parquet dataset.

This class extends the GenomeParquetDBNoParallel class to support parallel processing

query_parquet_regions

query_parquet_regions(regions: list[str], return_ordered=True, pseudobulk_subset_plan=None) -> Generator

Given a list of regions, find which Parquet file(s) contain any of those regions, read the data from those files, and yield the rows as dictionaries.

IMPORTANT: the regions will be deduplicated in the SQL query, so if repeated regions are passed in, they will be returned only once.

Parameters:

Name	Type	Description	Default
regions	list[str]	the region coordinates must be exactly the same as regions in parquet files. for more general region query, use "query_regions" method.	required
return_ordered		If False, the results are returned in an unordered fashion.	True
pseudobulk_subset_plan			None

compressed_bytes_to_array

compressed_bytes_to_array(bytes: bytes, dtype: str) -> np.ndarray

Decompress bytes and convert to numpy array.

Parameters:

Name	Type	Description	Default
bytes	bytes	The compressed bytes to be decompressed.	required
dtype	str	The data type of the resulting numpy array.	required

Returns:

Type	Description
ndarray	The decompressed numpy array.