CLI: genome commands¶
Ten commands in two groups: one prepares a reference on this machine, the other registers
annotations against it and reads what they carry. Every one of them takes an assembly name
as its first argument, except assembly list, which is the command that tells you what to
put there.
$ genome assembly register sacCer3
registered sacCer3 in /Users/hanqing/liulab_data/genome/sacCer3
source https://hgdownload.soe.ucsc.edu/goldenPath/sacCer3/bigZips/sacCer3.fa.gz
sha256 6ff72f079c3268431fc514a1a88730f8290e717663d343fa8a3590af65c422c3
files sacCer3.2bit, sacCer3.chrom.sizes, sacCer3.fa, sacCer3.fa.fai
The files land in the shared data directory, where every project on the machine reads the same copy.
Each synopsis below is its group's own --help, so it leaves the leading genome off:
assembly register sacCer3 is run as genome assembly register sacCer3. --json, exit
codes and the split between stdout and stderr are on the CLI overview.
genome assembly¶
register prepares an assembly and prints where it landed, as above. list is what to run
before it: the assemblies the metadata table offers, set against the ones already prepared
on this machine, downloading nothing to answer. files prints where one registered
assembly's four files are, for handing a path to another tool. verify re-reads the
FASTA and recomputes its sha256, which is the full-file check; registering and reopening go
by size and are instant. table-row downloads an assembly and prints the line to paste into
the shipped metadata table, which is how a new assembly gets a pinned checksum.
Assembly says what the four files hold and what a chimera is.
$ genome assembly list
assemblies in /Users/hanqing/liulab_data/genome
hg38 offered, not registered Homo sapiens GRCh38
hg19 offered, not registered Homo sapiens GRCh37
mm39 offered, not registered Mus musculus GRCm39
mm10 offered, not registered Mus musculus GRCm38
sacCer3 registered Saccharomyces cerevisiae R64-1-1
ce11 registered Caenorhabditis elegans WBcel235
ecHT115 offered, not registered Escherichia coli HT115 ASM435494v1
ce11_ecHT115 offered, not registered
registered here: 2 — prepare another with `genome assembly register <name>`, or re-check one with `genome assembly verify <name>`
an assembly the table does not list registers too, from the UCSC golden path — with no pinned checksum behind it
A name no row lists reads as registered, not offered, and a directory in the tree with no
registration record beside it reads as here, not registered. Neither is an error, and
neither costs the exit code.
$ genome assembly files sacCer3
registered sacCer3 in /Users/hanqing/liulab_data/genome/sacCer3
fasta /Users/hanqing/liulab_data/genome/sacCer3/sacCer3.fa
fai /Users/hanqing/liulab_data/genome/sacCer3/sacCer3.fa.fai
twobit /Users/hanqing/liulab_data/genome/sacCer3/sacCer3.2bit
chrom_sizes /Users/hanqing/liulab_data/genome/sacCer3/sacCer3.chrom.sizes
$ genome assembly files sacCer3 --json | jq -r .genome_files.fasta
/Users/hanqing/liulab_data/genome/sacCer3/sacCer3.fa
Every path is absolute. --json prints the object register --json does, with the four
paths under genome_files. files never prepares anything: an assembly not registered
here exits 1 and names the genome assembly register command.
$ genome assembly verify sacCer3
/Users/hanqing/liulab_data/genome/sacCer3/sacCer3.fa: sha256 6ff72f079c3268431fc514a1a88730f8290e717663d343fa8a3590af65c422c3 matches the digest the metadata table pins for it
Registering an assembly that is already here downloads nothing and prints the same summary
off its record. A directory whose files changed after they were registered exits 1
instead. --force is the repair, and it keeps whatever is provably good: a FASTA whose
checksum still matches is reused and only the derived files are rebuilt.
The five commands, with every argument and option:
assembly¶
Prepare an assembly on disk, check one, and compute its metadata row.
Usage¶
assembly [OPTIONS] COMMAND [ARGS]...
Arguments¶
No arguments available
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--install-completion |
Install completion for the current shell. | No | - |
--show-completion |
Show completion for the current shell, to copy it or customize the installation. | No | - |
Commands¶
| Name | Description |
|---|---|
register |
Prepare an assembly on disk: fetch or... |
list |
List the assemblies the metadata table... |
files |
Print where a registered assembly's FASTA,... |
verify |
Re-read an assembly's FASTA and check its... |
table-row |
Download an assembly and print its... |
Subcommands¶
register¶
Prepare an assembly on disk: fetch or build, verify, index, and record it.
Downloads the FASTA from the source the metadata table pins (or the UCSC golden
path for an assembly no row lists), checks it against the pinned sha256, derives
the .fai, .2bit and chrom.sizes, and writes the registration record that says
all of it finished. An assembly that is already registered is reported from its
record without fetching anything.
Naming a chimera builds it, and there is no flag for listing its parts: an
assembly whose name is its component assemblies sorted and joined by _ —
ce11_ecHT115 — is concatenated from components already prepared here, merged
annotation included, and nothing is downloaded. The name carries the fact, so typing
the components in the wrong order is refused by naming the canonical spelling, and a
component this machine has not prepared is refused by naming the command that
prepares it. What an assembly already registered here is comes from its record and
never from its name, so a plain hg38_mm10 seeded years ago stays what it was.
Exits with code 1 when the directory holds a registration that cannot be trusted —
files with no record, or a record that disagrees with what is on disk — and when a
chimera cannot be built from this machine's parts. Re-run with --force to repair a
directory: an unpacked FASTA that still matches the pinned checksum is kept and only
the derived files are rebuilt, a chimera is concatenated again from its components,
and neither is a way past the checks above.
Usage¶
assembly register [OPTIONS] ASSEMBLY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'hg38'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--source TEXT |
Seed from this FASTA (local path or http(s)/ftp/sftp URL) instead of the source the metadata table pins. | No | - |
--force |
Register again from scratch — the repair for a directory that raises. | No | False |
--json |
Emit JSON instead of plain text. | No | False |
list¶
List the assemblies the metadata table offers against the ones prepared here.
Two questions, two answers, side by side: which assemblies the lab pins a source and a checksum for, and which are actually prepared on this machine — including ones no row lists, which is what a registration from the UCSC golden path is. It takes no assembly name, since neither half has one to scope it.
A directory in the assembly tree with no registration record beside it reads as
here, not registered: nothing vouches for what is in it, and reading it as an
assembly would be a claim this command has not checked. Registering it again from
scratch with --force is what makes it trustworthy.
Whether a registration is still intact is not asked here. That is genome assembly
verify, which re-reads the FASTA and recomputes its digest; doing it cheaply enough
to belong on this command would report unchecked in the words of checked. This one
reads the table, the directory names and one small record each, so it stays instant.
Nothing is downloaded, prepared or built to answer, and exit is 0 on a machine where
nothing is registered — which is a fresh install's ordinary state, and is printed as
such rather than as an error.
Usage¶
assembly list [OPTIONS]
Arguments¶
No arguments available
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--json |
Emit JSON instead of plain text. | No | False |
files¶
Print where a registered assembly's FASTA, .fai, .2bit and chrom.sizes are.
Nothing is downloaded, prepared or created, so an assembly not registered here is
refused rather than registered. Paths are absolute, and --json prints the object
genome assembly register --json does. Whether the bytes are intact is what
genome assembly verify checks.
Exits with code 1 when the assembly is not registered here, naming the command that
registers it, and when its directory cannot be trusted, naming the --force repair.
Usage¶
assembly files [OPTIONS] ASSEMBLY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'sacCer3'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--json |
Emit JSON instead of plain text. | No | False |
verify¶
Re-read an assembly's FASTA and check its sha256 against the digest expected of it.
Registering an assembly and reopening it go by presence and size, which is what makes them instant. This is the deliberate re-verification for when integrity is actually in doubt: it reads the whole file and computes its digest.
What that digest is held to is the metadata table's pin, and failing that the one this assembly's own registration recorded — three states with three sentences, since being held to the lab's pin and being held to what this machine last produced are different results.
A chimera is also checked against its components, which is the one failure no digest of its own bytes can show: those bytes do not change when a component is registered again underneath them. That line always prints for a chimera, whether it proved the components unchanged or found nothing to compare, so silence is never read as a pass.
Exits with code 1 when the digest is not the one expected, when a component is no longer the one this was built from, when there is nothing registered to verify, or when the assembly's directory cannot be trusted.
Usage¶
assembly verify [OPTIONS] ASSEMBLY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'sacCer3'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--fasta TEXT |
Check this FASTA instead of the assembly's registered one — a copy from a mirror or handed over by hand, checkable before anything is built on it. | No | - |
--json |
Emit JSON instead of plain text. | No | False |
table-row¶
Download an assembly and print its finished metadata table row.
Fetches the FASTA — from the pinned source when the table has one for this assembly, otherwise from the UCSC golden path — unpacks it, and computes the sha256 of the unpacked file. Prints one tab-separated line to paste into the shipped metadata table, or the same row as a JSON object.
A checksum the table already pins is reported, never enforced — this is the
command to reach for when an upstream file has legitimately changed and the pin has
to be regenerated, so refusing on a mismatch would refuse exactly when it is needed.
Use genome assembly verify to check a FASTA you already hold against the official
row.
A chimera is refused, and refused before anything is downloaded: it pins nothing, so this command has no job to do for one. The refusal describes the row — the name, and every other column blank — rather than printing it, because a row this command printed would look like one it had computed something for.
Exits with code 1 if the download fails, or if the assembly named is a chimera.
Usage¶
assembly table-row [OPTIONS] ASSEMBLY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'sacCer3'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--json |
Emit JSON instead of the TSV line. | No | False |
genome annotation¶
register fetches one of the annotations the table lists for an assembly and builds its
gffutils database; register-gtf does the same for a GTF of your own, which nothing is
pinned for. list puts what is offered beside what is registered here, and downloads
nothing to answer, so it works for an assembly you have never prepared. gene-list and
gene-categories read the curated gene categories that ship with the package. What a
category is and where it comes from is on Annotations.
$ genome annotation list ce11
annotations for ce11 in /Users/hanqing/liulab_data/genome/ce11
wormbase_ws298 registered WormBase WS298
default: wormbase_ws298
The last line of a registration always says whether the GTF's chromosome names were checked
against the assembly's, and when they were not, why. Registering an annotation before its
assembly is allowed, and reports that there was no chrom.sizes to check against;
registering the assembly is what fixes that.
gene-list prints ids and nothing else on stdout, so the output pipes or redirects. The
heading and the per-source counts go to stderr:
Asking for a category the annotation does not declare exits 1, and so does asking an
annotation that is not registered here. Neither prints an empty list. genome annotation
gene-categories ce11 says which categories may be asked for.
Every argument and option of the five:
annotation¶
Register an assembly's annotations, list them, and read a gene category.
Usage¶
annotation [OPTIONS] COMMAND [ARGS]...
Arguments¶
No arguments available
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--install-completion |
Install completion for the current shell. | No | - |
--show-completion |
Show completion for the current shell, to copy it or customize the installation. | No | - |
Commands¶
| Name | Description |
|---|---|
register |
Register one of an assembly's annotations... |
register-gtf |
Register an annotation from a local GTF:... |
list |
List what the annotation table offers for... |
gene-list |
Print the gene ids an annotation puts in... |
gene-categories |
List the gene categories an annotation... |
Subcommands¶
register¶
Register one of an assembly's annotations by name: fetch, verify, build, record.
Downloads the GTF from the URL the annotation table pins for this assembly, checks
the unpacked file against the pinned sha256 and its chromosome names against the
assembly's, builds the gffutils database, and writes the registration record that
says all of it finished. An annotation that is already registered is reported from
its record without fetching anything. A GTF the table does not list is registered
from a path instead, with genome annotation register-gtf.
The chromosome check needs the assembly's chrom.sizes, so it can only run once
the assembly itself is registered. Registering an annotation first is allowed and
reports that there was nothing to check the names against, which registering the
assembly first is what fixes. Standing the check down yourself reports that instead,
and advises nothing: you meant it.
Exits with code 1 when the table lists no such annotation, when the GTF is not the
digest pinned for it, when it names chromosomes the assembly does not carry, or when
the directory holds a registration that cannot be trusted — files with no record, or
a record that disagrees with what is on disk. Re-run with --force to repair it.
Usage¶
annotation register [OPTIONS] ASSEMBLY NAME
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'hg38'. | Yes |
NAME |
Registered annotation name, e.g. 'gencode_v50'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--force |
Register again from scratch — the repair for a directory that raises. | No | False |
--check-chromosomes / --no-check-chromosomes |
Refuse a GTF naming sequences this assembly does not carry, before paying for the database build. Pass --no-check-chromosomes to register one whose mismatch you have inspected and accept — the annotation is built as it stands and the record says the check was stood down, so nothing later mistakes it for a verified one, and nothing mistakes it for an annotation registered before its assembly either. | No | check-chromosomes |
--infer-genes / --no-infer-genes |
Reconstruct gene features from exon lines. Off by default: GENCODE, Ensembl and RefSeq GTFs declare them already and inferring is the slow path. Turn it on for a bare exon-level GTF — one whose only lines are exons — which otherwise registers as a database of exons and nothing else. | No | no-infer-genes |
--infer-transcripts / --no-infer-transcripts |
Reconstruct transcript features from exon lines. Off by default: GENCODE, Ensembl and RefSeq GTFs declare them already and inferring is the slow path. Turn it on for a bare exon-level GTF — one whose only lines are exons — which otherwise registers as a database of exons and nothing else. | No | no-infer-transcripts |
--json |
Emit JSON instead of plain text. | No | False |
register-gtf¶
Register an annotation from a local GTF: place, check, build, record.
The escape hatch for a GTF the annotation table does not list — genome annotation
register is the way in for one it does. Nothing is downloaded and no checksum is
compared against, because an unlisted GTF has none pinned for it: the file you name is
placed under .gz is decompressed on the way), its
chromosome names are checked against the assembly's, the gffutils database is built,
and the record that says all of it finished is written last. It is addressed by
genome
annotation list as registered but not offered.
The assembly is named rather than inferred — it is what says which reference these
gene models describe — and naming it is also what lets the chromosome check find the
assembly's chrom.sizes without being told where it is.
Exits with code 1 when the GTF is not there, when it names chromosomes the assembly
does not carry, or when the directory holds a registration that cannot be trusted.
Re-run with --force to repair it.
Usage¶
annotation register-gtf [OPTIONS] ASSEMBLY GTF NAME
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'hg38'. | Yes |
GTF |
Path to the GTF to register, plain or .gz. | Yes |
NAME |
Registered name to address it by afterwards, e.g. 'wormbase_ws298'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--force |
Register again from scratch — the repair for a directory that raises. | No | False |
--check-chromosomes / --no-check-chromosomes |
Refuse a GTF naming sequences this assembly does not carry, before paying for the database build. Pass --no-check-chromosomes to register one whose mismatch you have inspected and accept — the annotation is built as it stands and the record says the check was stood down, so nothing later mistakes it for a verified one, and nothing mistakes it for an annotation registered before its assembly either. | No | check-chromosomes |
--infer-genes / --no-infer-genes |
Reconstruct gene features from exon lines. Off by default: GENCODE, Ensembl and RefSeq GTFs declare them already and inferring is the slow path. Turn it on for a bare exon-level GTF — one whose only lines are exons — which otherwise registers as a database of exons and nothing else. | No | no-infer-genes |
--infer-transcripts / --no-infer-transcripts |
Reconstruct transcript features from exon lines. Off by default: GENCODE, Ensembl and RefSeq GTFs declare them already and inferring is the slow path. Turn it on for a bare exon-level GTF — one whose only lines are exons — which otherwise registers as a database of exons and nothing else. | No | no-infer-transcripts |
--json |
Emit JSON instead of plain text. | No | False |
list¶
List what the annotation table offers for an assembly against what is registered here.
Two questions, two answers, side by side: which annotations the lab supports for this assembly, and which are actually registered on this machine. The default annotation is named last, with the command that registers it when it is one of the ones this machine does not have — which is the ordinary state of a fresh install.
An annotation whose directory is here but cannot be trusted — files with no record,
or a record that disagrees with what is on disk — reads as broken rather than as
one nobody has fetched, and the line under it says what is wrong and names the
command that repairs it. This is where such a thing is discovered, so it is reported
and not raised over: exit is still 0, and one broken annotation never hides the
ones beside it.
Nothing is downloaded, prepared or built to answer this, so it works for an assembly that has never been registered here.
Usage¶
annotation list [OPTIONS] ASSEMBLY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'hg38'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--json |
Emit JSON instead of plain text. | No | False |
gene-list¶
Print the gene ids an annotation puts in one category, one per line.
The genes come from a curated gene list shipped inside this package, not from the GTF's own biotype attribute — which is spelled two ways across four publishers, sorts genes by three taxonomies that do not agree, and is missing altogether from some annotations, so a caller deriving categories itself reports none for those and never finds out.
Only the ids go to stdout, so the output pipes: the heading and the per-source
attribution go to stderr. For a merged annotation there is one source per contributing
component, and the ids are concatenated in that order and never de-duplicated — two
components carrying one gene id is a real ambiguity, not something to collapse here.
--json carries the same answer with the sources kept apart.
genome annotation gene-categories says which categories may be asked for;
they are the curated list's to declare and differ between annotations.
Exits with code 1 when the annotation is not registered here, when no curated gene list ships for it, and when it declares categories but not this one — three different facts, each with its own message, and none of them an empty list of genes.
Usage¶
annotation gene-list [OPTIONS] ASSEMBLY CATEGORY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'ce11'. | Yes |
CATEGORY |
Gene category, e.g. 'rRNA'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--annotation TEXT |
Ask about this registered annotation instead of the assembly's default one. An assembly with no default and none named has nothing to answer about, and says so. | No | - |
--json |
Emit JSON instead of plain text. | No | False |
gene-categories¶
List the gene categories an annotation declares, with a gene count for each.
What may be asked of genome annotation gene-list, since the categories are a property
of the annotation rather than of this package: human GENCODE splits rRNA pseudogenes
out and mouse does not, yeast carries a precursor category nothing else has, and a
bacterium has no mitochondrial category at all.
A merged annotation shows the per-component split beside each count, so a category one
component declares and another does not is visible as such rather than as a smaller
number. --json carries every category with its gene ids and its sources — the same
answer genome annotation gene-list gives for one of them, for all of them at once.
Exits with code 1 when the annotation is not registered here, and when no curated gene list ships for it — which is not the same answer as its declaring no genes, and is why an empty list is never printed.
Usage¶
annotation gene-categories [OPTIONS] ASSEMBLY
Arguments¶
| Name | Description | Required |
|---|---|---|
ASSEMBLY |
Assembly name, e.g. 'ce11'. | Yes |
Options¶
| Name | Description | Required | Default |
|---|---|---|---|
--annotation TEXT |
Ask about this registered annotation instead of the assembly's default one. An assembly with no default and none named has nothing to answer about, and says so. | No | - |
--json |
Emit JSON instead of plain text. | No | False |