Scanning Tools

This repository contains command-line tools and utilities for working with whole-slide imaging datasets. These tools support image extraction, data linking, and deidentification.

Installation

After cloning the repository, install with pip to generate command-line scripts

pip install ./scantools

Docker

A Docker image is provided for running the CLIs described below

docker build -t scantools ./scantools
docker container run --rm -it --user $(id -u):$(id -g) -v /data:/data scantools

Contents

• Image extraction
• Deidentification
• Barcode reading
• Hashing
• Scan dating
• Masking

Image extraction

The dumpwsi command-line tool extracts associated label, macro, and thumbnail images from whole-slide image formats supported by OpenSlide. This is useful for barcode reading, quality control, and generating thumbnail visualizations to confirm deidentification.

dumpwsi generates jpeg or png label/macro/thumbnail images in a directory structure that mirrors the input. Outputs are generated in sequence based on input file timestamps so that they can be sorted by scan order. Existing outputs are overwritten by default.

Note: dumpwsi leaves the associated images intact within the input file. It does not redact the input file.

Generate .jpg label images for all .svs files in the folder /inputs and in subfolders, saving them to the folder /outputs:

$ dumpwsi /inputs /outputs

Generate label images for all .tiff files:

$ dumpwsi /inputs /outputs -e .tiff

Generate macro images only:

$ dumpwsi /inputs /outputs -m

Generate thumbnail and label images:

$ dumpwsi /inputs /outputs -t -l

Skip existing output images:

$ dumpwsi /inputs /outputs -s

Write label images as .png:

$ python dumpwsi.py /inputs /outputs -f png

Set the number of worker processes to 8

$ dumpwsi /inputs /outputs -w 8

Deidentification

The deidentify command-line tool redacts label and macro images from a .svs image. It has been tested on images from Leica AT2 and GT450 scanners.

In-place deidentification all .svs slides in the folder /inputs:

$ deidentify /inputs/*.svs

Note: Deidentification operates on the files in-place and does not generate a copy of the file. Dump label images prior to deidentification to preserve a record of the slide provenance.

Note: Very rarely a scan will incorporate PHI into the image pyramid which cannot be detected or removed by this tool. Typically this is due to a misplaced label containing PHI in the lower portion of the label. For GT450 scans you can dump and review macro images which indicate the scan area on a macro photograph of the slide.

Barcode reading

barcodedecodes barcode contents from slide label images.

barcode takes as input a folder and generates a .csv file containing the filename and barcode contents for label images in that folder and subfolders.

Decode all images ending in _label.png in the folder /inputs and its subfolders, trying both aztec and data matrix formats, saving output to barcodes.csv. If barcodes.csv exists, skip entries and append to file end:

$ barcode /inputs barcodes.csv

Decode images ending in _label.jpg:

$ barcode /inputs barcodes.csv -e _label.jpg

Decode aztec symbology only:

$ barcode /inputs barcodes.csv -a

Decode data matrix only:

$ barcode /inputs barcodes.csv -d

Set the calibration interval to the first 128 inputs (default is 512)

$ barcode /inputs barcodes.csv -c 128

Set the number of worker processes to 4

$ barcode /inputs barcodes.csv -w 4

Hashing

hashwsi generates sha or md5 hashes from specified levels of a whole-slide image. Hashes are useful for identifying files based on contents, since files may undergo renaming operations during research. The hashing can be fast since it can be applied to a portion of the file such as a smaller level from the WSI pyramid. Hashing a pyramid level also ensures that the hash is invariant to changes in file metadata or redaction of associated images during deidentification.

hashwsi takes as input a folder and generates a .csv file containing the filename, hash algorithm used, level analyzed, and resulting hash for whole-slide images in that folder and subfolders.

Generate a sha1 hash (default) for all .svs files in the folder /inputs and in subfolders, saving output to hashes.csv. If hashes.csv exists, skip entries and append to file:

$ hashwsi /inputs ./hashes.csv

Use an md5 hash instead:

$ hashwsi /inputs ./hashes.csv -a md5

Hash all .tiff files:

$ hashwsi /inputs ./hashes.csv -e .tiff

Specify hashing of level 1 (default is level 2, larger is higher resolution):

$ hashwsi /inputs ./hashes.csv -l 1

Require a 10000 bytes in level to hash, increment level until requirement satisfied:

$ hashwsi /inputs ./hashes.csv -m 10000

Set the number of worker processes to 8

$ hashwsi /inputs ./hashes.csv -w 8

Scan dating

The scandate command-line tool generates extracts the original scan date from Aperio .svs files. scandate takes as input a folder and generates a .csv file containing the filename and scanning date for all .svs images in that folder and subfolders.

Generate dates for all .svs files in the folder /inputs and in subfolders, saving output to dates.csv. If dates.csv exists, skip entries and append to file:

$ scandate /inputs ./dates.csv

Set the number of worker processes to 8

$ scandate /inputs ./dates.csv -w 8

Masking

The mask command-line tool generates binary mask images of tissue regions, excluding background, marker, and dark print. This is done using a combination of Otsu thresholding of the grayscale image and thresholding of saturation and value channels in the HSV color space. mask takes as input a folder and generates a jpeg or png mask images in a mirror directory structure at the specified output folder.

Generate .jpg masks for all .svs files in the folder /inputs and in subfolders, saving them to the folder /outputs. Existing mask images will be skipped:

$ mask /inputs /outputs

Generate mask images for all .tiff files:

$ mask /inputs /outputs -e .tiff

Overwrite any existing images:

$ mask /inputs /outputs -n

Write mask images as .png:

$ mask /inputs /outputs -f png

Change the magnification to 2.5:

$ mask /inputs /outputs -m 2.5

Change the object removal and hole filling threshold to 2000 pixels:

$ mask /inputs /outputs -m 2.5 -l 2000 -o 2000

Change the saturation and value threshold parameters:

$ mask /inputs /outputs -s 0.7 -v 0.5

Set the number of worker processes to 8

$ mask /inputs /outputs -w 8