§ 02 · Practice
Pipelines that still run in five years.
A pipeline is software. We build yours with version control, tests, containers and CI, so it runs the same way for every sample, on every machine, and can be handed to the next person without a walkthrough.
§ 02.1 / Scope
Capabilities.
New pipelines
Nextflow to nf-core standards, Snakemake or WDL, depending on your team and infrastructure. Modular, documented, parameterised.
Legacy migration
Turn a folder of Bash, R and Python scripts into a tested workflow without changing the science it encodes.
Containers & environments
Docker and Apptainer images, Conda and pixi lockfiles, and a strategy for keeping them current.
Execution anywhere
SLURM, PBS, LSF on-prem; AWS Batch, Google Batch and Azure in the cloud; Seqera Platform for orchestration.
Testing & CI
nf-test and pytest suites, small test datasets, GitHub Actions or GitLab CI that run on every change.
Cost & performance
Runtime and cost profiling, resource right-sizing, spot and retry strategies that cut cloud bills without losing runs.
§ 02.2 / Output
Deliverables.
- —A versioned pipeline repository with CI, tests and docs
- —Container images and resolved, reproducible environments
- —Runbooks for your cluster or cloud account
- —Runtime and cost benchmarks on your data
Tools & Standards.
nf-core and community standards where they exist, custom software where they do not.
- Nextflow
- nf-core
- Snakemake
- WDL / Cromwell
- Docker
- Apptainer
- SLURM
- AWS Batch
- Seqera Platform
- GitHub Actions
- nf-test
- pixi
§ 02.3 / Questions
Frequently asked questions.
- Nextflow or Snakemake?
- Nextflow if you want the nf-core ecosystem and cloud execution; Snakemake if your team is Python-first and runs mainly on a cluster. We work in both and will recommend one for your case.
- Can you take over a pipeline someone else wrote?
- Yes. We start with a review and a test harness so we can change it safely, then modernise it in increments.
- Do you support the pipeline after handover?
- Yes. Retainers cover dependency updates, new modules and on-call for failed runs.
§ Practice Scoping · 02
Scope a project in pipeline engineering.
Tell us about your assay types, dataset scale, and timelines. We reply with a preliminary scoping memo within two working days.
Scope Pipeline Engineering →