The data catalog¶

Every data file c4 knows about is listed in one YAML file, c4/catalog.yaml, versioned with the code. Code never contains data paths; it asks the catalog.

Where it's read from¶

In order:

cohort(name, catalog=...), if you pass a path (or an already-loaded dict)
the C4_CATALOG environment variable
c4/catalog.yaml in the package

Paths in the catalog are relative to its root (unless absolute). A relative root is taken relative to the catalog file itself, which is how the demo catalog finds examples/demo/data. Set C4_DATA_ROOT to use a different root without editing the file, for example to test against a copy of the data:

export C4_DATA_ROOT=/scratch/me/data_copy

Layout¶

catalog_version: 1
root: /data/lab                 # paths below are relative to this

resources:                      # shared reference files, not tied to a cohort
  chain_hg19_to_hg38:
    path: ref/hg19ToHg38.over.chain
    from_build: b37
    to_build: b38

cohorts:
  demo:                         # one block per cohort
    aliases: [DEMO]
    owner: A. Researcher
    id_map: {...}               # links every ID type to patient_id
    pheno: {...}                # phenotype / clinical tables
    pcs: {...}                  # principal components
    hla: {...}                  # HLA calls, one block per software
    dna: {...}                  # genetic data, one block per data type
    survival: {...}             # settings for survival analyses

Inside a cohort, anything with a path: is an entry: one file, or one templated family of files. Entries are named by their position, joined with dots: pcs, pheno.main, hla.hibag.raw, dna.imputed. That's the name you pass to .load() and .path().

Entries under dna: are genetic data, used through .path(). Everything else is a table, used through .load().

The demo catalog, annotated¶

cohorts:
  demo:
    aliases: [DEMO]                 # other names cohort() accepts (case-insensitive)
    owner: A. Researcher            # who to ask about this data

    id_map:
      path: demo/ids/demo_id_map.tsv
      sep: "\t"
      built_by: notebooks/resolve_demo_ids.ipynb   # how the map was made

    pheno:
      default: main                 # "pheno" on its own means "pheno.main"
      main:
        path: demo/pheno/demo_pheno_2025-01-15.tsv
        sep: "\t"
        id: {column: ID, type: patient}
        data_dict:                  # used by .describe()
          path: demo/pheno/demo_data_dictionary.txt
          format: markdown
          term_col: Variable
          def_col: Description
          type_col: Type

    pcs:
      path: demo/pca/demo_array_pruned.eigenvec
      sep: "\t"
      id: {column: IID, type: array}

    dna:
      imputed:
        path: demo/dna/chr{chrom}.dose.vcf.gz
        format: vcf
        build: b38
        imputed_from: array         # imputed_id should equal array_id (checked)
        params: {chrom: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]}
      exome:
        path: TODO                  # known to exist, not added yet
        format: TODO
        build: TODO

    hla:
      hlahd:                        # one block per HLA caller
        source: exome
        default: standardized
        standardized:
          path: demo/hla/hlahd_calls_standardized.tsv
          sep: "\t"
          id: {column: seqID, type: exome}
      hibag:
        source: array
        default: raw
        raw:
          path: "demo/hla/hibag_{panel}.csv"
          sep: ","
          id: {column: sample.id, type: array}
          loader: HIBAG
          params: {panel: [european, african, asian, multiethnic]}
          defaults: {panel: european}

    survival:                       # used by `c4 survival` / `c4 prs --dataset demo`
      dna: imputed                  # which dna: entry holds the dosages
      tables: [pheno, pcs]          # merged on patient_id; the first defines the study population
      covariates: [PC1, PC2, PC3, female, age]
      endpoints:
        OS: {time: OS_mo, event: OS_event}

Entry fields¶

Field	Used for	Notes
`path`	all	Relative to `root`, or absolute. `TODO` means "exists, not added yet". May contain `{placeholders}`.
`sep`	tables	Column separator. Default tab. Write `"\t"` in double quotes and regexes like `'\s+'` in single quotes.
`header`	tables	Header row number. Default `0`.
`id`	tables	`{column: <name>, type: patient \\| exome \\| array \\| imputed}`. Required for `.load()`.
`loader`	tables	`standard` (default), `HLA` (default under `hla:`; gene columns `A` or `A_1`/`A_2` become `HLAgenotype` objects), `HLA-HD` (same as `HLA`), or `HIBAG` (`allele1_A`/`allele2_A` become `A`, with `.prob` from `prob_A`).
`params`	templated files	Allowed values for each placeholder, e.g. `{panel: [european, african]}`.
`defaults`	templated files	Value used when a placeholder isn't passed.
`data_dict`	phenotype tables	Data dictionary for `.describe()`: `path`, `format` (`markdown` or `space_delim` for `.txt`), `term_col`, `def_col`, optional `type_col`.
`format`, `build`	genetic data	Shown in the table, e.g. `vcf`, `b38`.
`id`	genetic data	`{type: ...}`: which ID map column holds the files' sample names. Defaults to the entry's key (`dna.imputed` → `imputed_id`); `type: patient` means the files use `patient_id` itself.
`imputed_from`	genetic data	The data type an imputed set was made from; `.check()` confirms the IDs match.
`source`, `description`, `built_by`	anything	Free text for people. c4 doesn't use them.

A group (a block without a path, like pheno or hla.hibag) can have a default: naming one of its entries, so the short name loads it.

Templated files¶

Use {placeholder} in a path for files that differ by one value, and list the allowed values under params:

path: demo/dna/chr{chrom}.dose.vcf.gz
params: {chrom: [1, 2, ..., 22]}

.path("dna.imputed", chrom=6) fills it in, and .paths("dna.imputed") gives all of them. A placeholder with a defaults value can be left out. The status column and .check() test every combination.

Quote paths that start with {

In YAML, a value beginning with { is read as a mapping. Write path: "{panel}_calls.csv" with quotes. Paths with the placeholder later on (demo/chr{chrom}.vcf.gz) are fine unquoted.

Common edits¶

Add a file to an existing cohort¶

Add an entry in the right section with path, sep and id.
If it needs special parsing, set loader.
Run cohort("<name>").check().

Add a cohort¶

Build its ID map (one row per patient; see How IDs link).
Copy an existing cohort block, rename it, and fill in what exists. Leave anything you don't have yet as TODO rather than deleting it, so the gap is visible.
Run .check().

Move data¶

Change the paths (or root, if everything moved together), run .check(), and commit. No code changes.

Replace a file with a newer version¶

Point path at the new file and commit. The git history keeps a record of which version was current when, so older analyses can still be traced. Put a date in data file names (demo_pheno_2025-01-15.tsv) to make this obvious.

Shared resources¶

Files that aren't specific to a cohort, like liftOver chain files or the 1000 Genomes panel table, go under resources: and are fetched by name:

from c4.cohort import resource_path

resource_path("chain_hg19_to_hg38")

Liftover chains also carry from_build and to_build. The survival and PRS commands use them to lift variants given in one build onto genotype data in another, so c4 survival --variant-build b37 against a b38 VCF finds the hg19→hg38 chain without a --chain-file. liftover_chains() lists them:

from c4.cohort import liftover_chains

liftover_chains()   # {("b37", "b38"): .../hg19ToHg38.over.chain, ...}

Reference panels¶

A panel with no patients of our own, like 1000 Genomes, is a cohort block with kind: reference. It loads like any other cohort, and list_cohorts(kind="cohort") leaves it out:

from c4.cohort import cohort

kgp = cohort("1000g")
kgp.load("pheno")                       # sample, pop, super_pop, gender
kgp.path("dna", chrom=6)                # b38 (30x high coverage) by default
kgp.path("dna.b37", chrom=6)            # phase 3
kgp.samples("b38")                      # the 2,504 unrelated samples

Its sample names are the same in every file, so the ID map has just patient_id, qc_note and drop, and each dna: entry sets id: {type: patient}. The b38 VCFs hold 3,202 samples. The 698 that aren't in phase 3 (mostly relatives) are marked drop=1, so both builds cover the same 2,504 people. .path() returns the whole VCF, so subset it with .samples().

Survival settings¶

A cohort's survival: block makes it available to c4 survival --dataset and c4 prs --dataset, and to Survival.from_cohort() in Python:

Field	Meaning
`dna`	The `dna:` entry holding the dosages, e.g. `imputed`. Its VCF sample names are matched to patients through the id_map's `<dna>_id` column (`imputed_id`).
`tables`	Tables merged on `patient_id` to supply outcomes and covariates. The first one defines who is in the study. Default `[pheno]`.
`covariates`	Columns adjusted for in every model. Columns starting with `PC` are standardised.
`endpoints`	`{name: {time: <column>, event: <column>}}`, chosen with `--survival`.

Patients marked drop=1 in the ID map are left out, as everywhere else. A qc_note on its own doesn't remove anyone.

from c4.survival import Survival

surv = Survival.from_cohort("demo")