Bring your own data to NicheDECODE

Use an annotated spatial reference to train a model and infer niche proportions in your own bulk RNA-seq cohort.

View the template notebook · Download notebook

After cloning the repository, open tutorials/bring_your_own_data.ipynb in Jupyter. Edit its configuration cell, prepare the four inputs below, and run the cells in order. The notebook uses tutorials/byod_utils.py alongside the existing model. It predicts the relative abundance of predefined niches; it does not assign cells to new coordinates or discover new niches in bulk samples.

Environment and scope

Follow the installation tutorial to create the NicheDECODE environment. The current model requires CUDA. Verify torch.cuda.is_available() before training. SpatialDE, NaiveDE, FRoGS and CytoSPACE are external preparation tools, not bundled in environment.yml. Use their installation instructions and record their versions; CytoSPACE can be run in a separate environment.

Use this template to train a new model for your reference and target cohort. Protein data require modality-specific preparation; use the CODEX example as the starting point.

1 Prepare the input files

data/custom/
  reference.h5ad
  bulk_counts.csv
  spatialde.csv
  go_embeddings.csv
  spot_cell_counts.csv   # spot-resolution references only

A small format illustration (values are illustrative, not a usable training dataset):

gene_id,sample_A,sample_B
EPCAM,120,80
COL1A1,50,210
PTPRC,90,60

To construct an h5ad from tables, align all metadata by ID before assigning it:

import anndata as ad
import pandas as pd

# Reference expression table: gene rows, cell/spot columns.
counts = pd.read_csv('reference_counts.csv', index_col=0)
metadata = pd.read_csv('reference_metadata.csv', index_col=0)
# metadata columns: x, y, niche; index: cell/spot ID
assert counts.index.is_unique and counts.columns.is_unique and metadata.index.is_unique
metadata = metadata.reindex(counts.columns)
assert metadata[['x', 'y', 'niche']].notna().all().all()
reference = ad.AnnData(counts.T.astype('float32'), obs=metadata)
reference.layers['counts'] = reference.X.copy()
reference.obsm['spatial'] = metadata[['x', 'y']].to_numpy(float)
reference.write_h5ad('reference.h5ad')

For an existing bulk h5ad, rows must be samples and columns genes; assign its raw-count layer to X. Do not pass log-expression, z-scores, TPM or FPKM through this raw-count template. Convert identifiers consistently upstream and resolve duplicate mappings explicitly. Missing genes are excluded when selecting a new shared panel, not filled with fabricated zero expression. Inspect the printed overlap and preserve niche-discriminative genes; there is no universal sufficient gene-count threshold.

2 Define and format niche annotations

Obtain spatial domains from an appropriate spatial method, such as NicheCompass, CellCharter, STAligner or SpaNiche, or use expert-reviewed anatomical/histological annotations. Follow the chosen method's own workflow, inspect spatial coherence, markers and biological meaning, then export one label per reference cell/spot. This guide imports those labels; it does not implement a niche-discovery algorithm.

cell_or_spot_id,niche
spot_001,tumor_interior
spot_002,tumor_stroma_boundary
spot_003,stroma

labels = pd.read_csv('niche_annotations.csv', index_col=0)
assert labels.index.is_unique
reference.obs['niche'] = labels['niche'].reindex(reference.obs_names)
assert reference.obs['niche'].notna().all()
reference.obs['niche'] = reference.obs['niche'].astype('category')

Cell type and niche are different annotations: a niche may contain several cell types. Cluster numbers from different sections are not automatically equivalent; harmonize their meaning before combining data. The template does not automatically remove rare niches. If merging or excluding a niche, justify it biologically, record the decision and recognize that outputs then describe proportions relative to the retained label set.

3 Prepare spot cell counts with CytoSPACE

Skip this section for single-cell spatial references. For spot data, prepare matching scRNA counts, scRNA cell-type labels, spatial counts and spatial coordinates using the official CytoSPACE input instructions. Expression matrices use genes as rows and cells/spots as columns; identifiers must agree between inputs. The upstream Seurat export helpers are also available. Run, for example:

cytospace -sp scRNA_counts.txt -ctp scRNA_celltypes.txt -stp ST_counts.txt -cp ST_coordinates.txt -o cytospace_output

Convert its assigned-cell table to a spot count table (one row per assigned cell):

assignments = pd.read_csv('cytospace_output/assigned_locations.csv')
assert assignments['UniqueCID'].is_unique
cell_counts = assignments.groupby('SpotID').size().rename('cell_count')
cell_counts.rename_axis('spot_id').to_csv('data/custom/spot_cell_counts.csv')

CytoSPACE documents UniqueCID and SpotID in its output specification. Inspect unassigned spots and resolve them through QC before continuing. Do not silently set missing estimates to one.

Set SPOT_RESOLUTION=True in the notebook. Spot expression is summed as measured; only the supervision labels are weighted by estimated cell numbers. For example, a selected tumor spot with 8 estimated cells and a stromal spot with 2 cells yield labels (0.8, 0.2), not (0.5, 0.5). Each spot has one niche annotation, so the target is an estimated cell-number-weighted domain composition. For random mixing, the labels are recomputed after selecting spots; sampled spot fractions are not assumed to be cell fractions.

4 Prepare spatial and functional priors

SpatialDE features

Fit spatial features on the reference only. Following the SpatialDE preprocessing example, use raw counts, variance stabilization and library-size regression, then export its gene-wise results:

import numpy as np
import pandas as pd
import NaiveDE, SpatialDE
from tutorials.byod_utils import dense, SPATIAL_FEATURES

counts = pd.DataFrame(dense(reference.layers['counts']),
                      index=reference.obs_names, columns=reference.var_names)
# Remove all-zero genes and empty observations upstream.
info = pd.DataFrame(reference.obsm['spatial'], columns=['x', 'y'], index=counts.index)
info['total_counts'] = counts.sum(axis=1)
stabilized = NaiveDE.stabilize(counts.T).T
residuals = NaiveDE.regress_out(info, stabilized.T, 'np.log(total_counts)').T
results = SpatialDE.run(info[['x', 'y']], residuals)
results[['g'] + SPATIAL_FEATURES].to_csv('data/custom/spatialde.csv', index=False)

The required fields are FSV, max_delta, max_ll, max_mu_hat, max_s2_t_hat, s2_FSV, s2_logdelta, LLR, pval, qval. Keep full fit results rather than only significant genes. Inspect missing/nonfinite fits. The notebook aligns rows to the expression gene order, scales each feature with the repository's auto_log_then_minmax, and freezes that prior for prediction.

SpatialDE may require substantial memory for large sections. If using a representative reference subset, record the sampling strategy and verify that relevant domains remain represented. Do not concatenate unrelated sections into a single coordinate plane. This template intentionally uses one reference section.

GO functional embeddings

Obtain GO embeddings from FRoGS. Its provided human GO file data/gene_vec_go_256.csv is headerless, with a gene ID followed by embedding values. Convert it to the template format:

emb = pd.read_csv('FRoGS/data/gene_vec_go_256.csv', header=None, index_col=0)
emb.index = emb.index.astype(str)
emb.index.name = 'gene_id'
emb.columns = [f'embedding_{i}' for i in range(emb.shape[1])]
emb.to_csv('data/custom/go_embeddings.csv')

Match FRoGS identifiers to your expression identifiers with a documented mapping before use. For another species, use appropriate GO-derived embeddings or a justified ortholog mapping. The notebook constructs a cosine k-nearest-neighbor graph (default k=5), explicitly preserves graph node order, and reports embedding coverage. Genes lacking embeddings can remain in the expression encoder but are absent from the graph. Negative similarity weights or invalid vectors trigger an error for inspection.

5 Set pseudo-tissue construction parameters

Start with these settings, inspect the printed sample sizes and niche distributions, then tune on reference-derived validation data. Avoid windows across section boundaries. To use multiple sections, generate pseudo-tissues separately and establish a consistent reference-prior strategy before combining them; the single-section template does not automate this.

6 Train, predict and interpret outputs

The notebook splits reference-derived pseudo-tissues for training and early stopping, trains a new model, and predicts the unlabelled bulk samples through a dedicated inference function. It does not create fake ground-truth labels. An overlapping-window split is only an optimization validation split. For independent evaluation, separate whole sections/donors before pseudo-tissue generation and prior construction.

Use a fresh RUN_NAME for each run. The template stops if output directories/checkpoints exist. Do not reuse a checkpoint after changing gene order, normalization, niches or priors. Store the inputs and their provenance with the output manifest. A high proportion is relative to the trained niche set; it does not measure absolute tissue volume, and an absent niche label cannot be discovered at prediction time.

Troubleshooting

• No windows: inspect coordinate spread, lower MIN_OBSERVATIONS or MIN_NICHES, or adjust width.
• Missing annotations/cell counts: inspect exact cell/spot IDs and section prefixes; align by ID, never row order.
• Few shared genes: check species, gene identifier conversion, CSV orientation and reference panel coverage.
• CUDA unavailable: check the active Jupyter kernel and CUDA-enabled PyTorch installation.
• Out of memory: reduce batch size and pseudo-tissue count; select reference-informative genes before generating dense model inputs.
• No bulk accuracy metrics: expected when measured niche composition is unavailable. Pseudo-tissue validation is not a substitute for independent biological validation.