asf-tools

ASF Tools command line tool-kit for ASF operations.

Overview

ASF Tools is a Python-based command-line application designed to streamline and automate repetitive tasks within ASF operations. It provides a suite of utilities for managing sequencing data, interacting with internal services such as the LIMS, and supporting automation workflows for both Illumina and Nanopore (ONT) platforms.

• Authors: Chris Cheshire, Areda Elezi

Installation

Requirements:

• Python 3.11
• Linux or macOS (tested in containerized environments)

Development Setup:

1. Clone the repository:

   git clone https://github.com/FrancisCrickInstitute/asf-tools.git
   cd asf-tools

2. Install dependencies for development and testing:

   pip install -e ".[dev,tests]"

Formatting and Linting:

• Format code with black:

  black .

• Sort imports with isort:

  isort .

• Lint with ruff:

  ruff check .

Usage

ASF Tools can be run as a containerized application (recommended for production) or directly via the CLI for development.

Running in Production

ASF Tools is packaged for use on Nemo. The easiest way to interact is via the scripts in asf-automation-scripts in the scripts folder. All operations must be run from the scripts folder where the config.sh file is located. For example:

./asf-automation-scripts/asf_tools.sh [ARGS]

CLI Commands

Illumina: Viral Genomics Samplesheet

Generate a samplesheet from a folder of Illumina FASTQ files, supporting multi-lane samples and optional path prefix substitution.

asf_tools gen-viral-genomics-samplesheet SOURCE_DIR TARGET_DIR [--curr-prefix CURR_PREFIX --new-prefix NEW_PREFIX]

• SOURCE_DIR: Directory containing FASTQ files.
• TARGET_DIR: Directory to write the samplesheet.
• --curr-prefix: (Optional) Prefix in FASTQ file paths to replace (e.g., /mnt/data).
• --new-prefix: (Optional) Replacement prefix for FASTQ file paths (e.g., /Users/you/data).

Behavior:

• Each (sample id, lane) pair appears as a row in the samplesheet.
• If both --curr-prefix and --new-prefix are provided, the prefix in read1 and read2 paths is substituted.
• The samplesheet is sorted by sample id, read1, and read2 for consistency.

ONT: Nanopore Data Management

gen-demux-run

Creates a run directory for the ONT demux pipeline.

Syntax

asf_tools ont gen-demux-run [OPTIONS]

Options

• -s, --source_dir <PATH>: Source directory to look for runs (required).
• -t, --target_dir <PATH>: Target directory to write runs (required).
• -p, --pipeline_dir <PATH>: Pipeline code directory (required).
• -n, --nextflow_cache <PATH>: Nextflow cache directory (required).
• -w, --nextflow_work <PATH>: Nextflow work directory (required).
• -c, --container_cache <PATH>: Nextflow Singularity cache directory (required).
• -r, --runs_dir <PATH>: Host path for the runs folder (required).
• --use_api: Utilize the Clarity API to generate the samplesheet.
• --contains <STRING>: Filter run folders by a substring.
• --samplesheet_only: Update samplesheets only for all runs in the target folder.
• --nextflow_version <VERSION>: Specify the Nextflow version to use in the sbatch header.

deliver-to-targets

Symlinks demux outputs to the user directory.

Syntax

asf_tools ont deliver-to-targets [OPTIONS]

Options

• -s, --source_dir <PATH>: Source directory (required).
• -t, --target_dir <PATH>: Target directory (required).
• -d, --host_delivery_folder <PATH>: Host delivery folder path.
• -i, --interactive: Run in interactive mode to select runs manually.

Development Workflow

• Follow Test-Driven Development (TDD): write tests before implementation.
• All new code must be accompanied by tests in the tests/ directory.
• Use pytest for running tests:

  pytest

• Ensure code is formatted with black and isort, and linted with ruff.
• Aim for 100% code coverage.

Project Structure

• asf_tools/: Main source code.
• tests/: Flat directory of test modules mirroring the source structure.
• docs/: Documentation and design notes.
• output/: Delivery and output files (ignored in version control).
• pyproject.toml: Project configuration.
• Dockerfile: Container build file.

License

See LICENSE for details.

Contact

Please contact chris.cheshire@crick.ac.uk for any questions.