c4

c4 is a Python toolkit for genetics research on the UCSF C4 cluster. These pages cover its data loading layer: one catalog file that lists every dataset the lab uses, and a small API for finding, loading and merging them. They also cover its HLA tools, for working with HLA typing from HLA-HD and HIBAG, and fine-mapping of GWAS loci with SuSiE.

Example data

Every example on these pages uses a made-up cohort called demo, with fake patients, samples and paths. Real cohorts work the same way.

If you have access to the c4 repository (it is private to the lab), the demo is in examples/demo/ and you can follow along. From the repository root:

export C4_CATALOG=examples/demo/catalog.yaml
python examples/demo/run_examples.py    # runs every example on these pages

The docs show the demo's data folder as /data/lab; in the repository it is examples/demo/data.

In 30 seconds

from c4.cohort import cohort

demo = cohort("demo")                          # prints what's available
pheno = demo.load("pheno")                     # one table, with patient_id added
df = demo.load("pheno", "pcs", "hla.hlahd")    # several tables, merged on patient_id
vcf = demo.path("dna.imputed", chrom=6)        # genetic data: a file path
demo.describe("OS_mo")                         # what does this column mean?
demo.check()                                   # is everything where the catalog says?

The idea

One catalog. Every data file is listed once in c4/catalog.yaml, grouped by cohort: phenotype tables, principal components, HLA calls by software, and DNA by data type. Paths live there, not in code, so when data moves only the catalog changes.

One patient ID. Different data types identify people differently: clinical tables use a patient ID, the exome VCF uses sequencing sample names, the array uses genotyping IDs. Each cohort has an ID map that links them all to a single patient_id. Every table you load gets a patient_id column, so merging anything with anything is a join on that one column.

Loud, not silent. Merges print how many patients matched. Duplicate patients stop a merge instead of quietly multiplying rows. Missing files and unlinked samples are reported rather than skipped.

Where to go next

I want to…	Read
Load data and merge tables	Loading and merging
Understand how exome, array and clinical IDs connect	How IDs link
Get a few SNPs for everyone, as a table	Extracting variants
Load data in R	From R
Add a new file or cohort, or move data	The data catalog
Run a command (c4 dosages, survival, clump, …)	Command line
Make sense of an error message	Troubleshooting
Work with HLA genotypes, or compare two HLA callers	HLA calls
Fine-map GWAS loci (credible sets, PIPs)	Fine-mapping
See every function and argument	API reference

Setup

c4 needs Python 3.11+. In a clone of the repository, this installs it and every Python package it uses:

pip install -e .

Add ".[dev]" instead of . to also get pytest. Extracting variants and the survival/PRS commands also need bcftools on your PATH (on C4: module load bcftools). Fine-mapping needs plink2 and R with the susieR package (module load plink2 r).

Then check that the catalog matches what is on disk:

python -c "from c4.cohort import cohort; cohort('demo').check()"