How IDs link

The same person shows up under different IDs in different data:

Data	ID used	Example
Clinical / phenotype tables	patient ID	P003
Exome VCF, HLA-HD calls	sequencing sample name	EX-0004
Genotyping array, PCs, HIBAG calls	array sample ID	ARR-0003
Imputed VCF	imputed sample ID	ARR-0003 (same as array here)

Each cohort has one ID map that ties these together, and every table in the catalog says which kind of ID it uses. That's all c4 needs to put a patient_id on any table.

flowchart LR
    P["pheno<br/>ID (patient)"] --> M
    C["pcs<br/>IID (array)"] --> M
    H["hla.hlahd<br/>seqID (exome)"] --> M
    B["hla.hibag<br/>sample.id (array)"] --> M
    M{{"id_map<br/>patient_id · exome_id · array_id · imputed_id"}} --> O["patient_id<br/>on every table"]

The ID map

One TSV per cohort, one row per patient, with fixed columns:

Column	Meaning
patient_id	The study ID for the person. Matches the phenotype tables.
exome_id	Their exome sample name, as it appears in the exome files
array_id	Their array sample ID, as it appears in the array files
imputed_id	Their sample name in the imputed VCF
qc_note	Optional free text for the record, e.g. an old sample ID. Never removes anyone.
drop	Optional: 1 leaves the patient out of everything loaded; blank or 0 keeps them.

A blank cell means that patient has no sample of that type. The demo ID map:

demo.id_map()

  patient_id exome_id  array_id imputed_id                                       qc_note drop
0       P001  EX-0001  ARR-0001   ARR-0001                                           NaN  NaN
1       P002  EX-0002  ARR-0002   ARR-0002                                           NaN  NaN
2       P003  EX-0004  ARR-0003   ARR-0003                         old exome_id: EX-0003  NaN
3       P004  EX-0003  ARR-0004   ARR-0004                         old exome_id: EX-0004  NaN
4       P005  EX-0005       NaN        NaN                                           NaN  NaN
5       P006  EX-0006  ARR-0006   ARR-0006  sex mismatch (genetic male, recorded female)    1
6       P007      NaN  ARR-0007   ARR-0007                                           NaN  NaN
7       P008      NaN       NaN        NaN                                           NaN  NaN

P005 has an exome but no array, P007 the reverse, and P008 has no genetic data. P003 and P004's notes record a sample swap (below). P006 failed a sex check and is marked drop=1, so .load() leaves it out (Dropped patients and QC notes).

How a table is linked

Each table entry in the catalog declares its ID column and its ID type:

pcs:
  path: demo/pca/demo_array_pruned.eigenvec
  id: {column: IID, type: array}

On load, c4 reads the IID column, looks each value up in the ID map's array_id column, and writes the matching patient_id. For a table with type: patient, the column already holds patient IDs and is copied as is.

The four types are patient, exome, array and imputed, matching the ID map's column names (exome → exome_id, and so on).

A worked example

Exomes EX-0003 and EX-0004 were swapped at sequencing: the tube labelled for P003 actually held P004's DNA, and vice versa. Comparing exome genotypes against each patient's array data revealed it.

The exome files are not renamed. They keep the names they were sequenced under, which still match the sequencing core's records and every file derived from them. Instead the ID map records who each sample really belongs to:

patient_id	exome_id	array_id
P003	EX-0004	ARR-0003
P004	EX-0003	ARR-0004

Anything keyed by exome ID now lands on the right person automatically:

demo.load("pheno", "hla.hlahd")[["patient_id", "seqID", "A"]]

  patient_id    seqID                A
0       P001  EX-0001  A*02:01/A*24:02
1       P002  EX-0002  A*01:01/A*03:01
2       P003  EX-0004  A*11:01/A*24:02
3       P004  EX-0003  A*02:01/A*02:01
...

Fixing a swap, or undoing one later, is a two-line edit to the ID map, which is tracked in git. The qc_note column is a good place to keep the old assignment (old exome_id: EX-0003).

Rules for an ID map

c4 checks these when it reads the map, and stops with an error if they're broken:

• One row per patient. A patient_id appears at most once.
• Each sample belongs to one patient. A given exome_id (or array or imputed ID) appears at most once.

And by convention:

• Genetic IDs are the names in the files, not the names they "should" have had.
• Pick one sample per type per patient. If someone was sequenced twice, choose the sample to use and note the other in the QC record, not as a second row.
• drop decides who is used; qc_note explains. Only drop=1 removes a patient. Put the reason in qc_note, so anyone who finds a patient missing can see why. drop must be 1, 0 or blank; anything else is an error.
• Generate it with a script or notebook, and point the catalog's built_by field at it, so the reasoning behind each decision can be found.

When imputed and array IDs are the same

If the imputed data came from the array, the sample names usually carry over. Keep both columns anyway, and say so in the catalog:

dna:
  imputed:
    path: demo/dna/chr{chrom}.dose.vcf.gz
    imputed_from: array

.check() then confirms the two columns really are identical, and a future re-imputation that renames samples only needs a new imputed_id column.