SCOUT++

SCOUT++ is a feature-engineered derivative dataset and reproducibility toolkit built from the public ARL-SCOUT dataset.

The goal of SCOUT++ is to provide model-ready data files, aligned extracted images, preprocessing code, and documentation for research on situated human-robot dialogue, multimodal instruction understanding, and robot command classification.

SCOUT++ is not an official ARL SCOUT release.

Relationship to ARL SCOUT

The original SCOUT dataset, the Situated Corpus Of Understanding Transactions (seen here), is a multimodal human-robot dialogue corpus focused on collaborative exploration tasks. In the original SCOUT experiments, human Commanders gave instructions to a remotely located robot through a Wizard-of-Oz setup involving a Dialogue Manager and Robot Navigator.

SCOUT++ is derived from SCOUT and adds feature engineering, cleaned task-specific data files, image alignment outputs, model input/output files, and scripts used for GPT-based and neural-network-based experiments.

Users of SCOUT++ should cite both:

1. The original SCOUT dataset/paper.
2. The SCOUT++ paper and dataset release.

What SCOUT++ Includes

This repository contains the public SCOUT++ release artifacts and supporting reproducibility files:

• full aligned Navigator-view image files;
• core GPT/model input rows;
• GPT text-only output results;
• GPT multimodal output results;
• sample data and sample images;
• preprocessing and feature-engineering scripts;
• GPT text-only and multimodal prompting scripts;
• frame extraction utilities for aligned Navigator-view images;
• documentation for schema, transformations, limitations, provenance, script usage, and release validation;
• SHA-256 checksums for public artifacts.

The repository intentionally excludes raw videos, private environment files, external embedding files, intermediate workbooks, and duplicate spreadsheet exports.

Repository Structure

SCOUT++/
  README.md
  LICENSE
  CITATION.cff
  pyproject.toml
  requirements.txt

  checksums/
    checksums_sha256.txt

  data/
    raw/
      README.md
      videos/
        .gitkeep
    images/
      *.jpg
    processed/
      gpt_input.csv
      gpt_text_output_results.csv
      output_results.csv
      output_results_full.csv
    samples/
      sample_output_results.csv
      images/

  docs/
    dataset_card.md
    schema.md
    transformations.md
    known_limitations.md
    release_checklist.md
    script_usage.md
    provenance.md

  notebooks/
    trial.ipynb

  scripts/
    GPTMultiModalPrompt.py
    GPTTextPrompt.py
    get_frames.py
    gpt_text_output_fix.py
    normalizer.py
    regenerate_missing_images.py
    tokenizermodule2.py

  src/
    scoutpp/

  tests/

Data Availability

The public GitHub release includes the main SCOUT++ processed CSV files and the aligned image folder:

data/processed/gpt_input.csv
data/processed/gpt_text_output_results.csv
data/processed/output_results.csv
data/processed/output_results_full.csv
data/images/*.jpg

A smaller sample is also included for quick inspection:

data/samples/sample_output_results.csv
data/samples/images/

The following files are intentionally excluded from the public release:

.env
data/interim/*
data/processed/*.xlsx
data/processed/meta.tsv
data/processed/vecs.tsv
data/raw/videos/*
glove.6B.*.txt
.DS_Store
__pycache__/
*.pyc

Dataset Files

File	Description
data/processed/gpt_input.csv	Core GPT/model input file containing Commander text, reference DM-to-RN command labels, and aligned image filenames.
data/processed/output_results_full.csv	Full multimodal GPT output file with Commander text, reference labels, GPT predictions, GPT explanations, and aligned image paths.
data/processed/output_results.csv	Smaller multimodal output/evaluation file.
data/processed/gpt_text_output_results.csv	Text-only GPT output file with full GPT responses and parsed predictions/reasons.
data/images/*.jpg	Full aligned Navigator-view image files used by the public processed outputs.
data/samples/sample_output_results.csv	Small sample file for quick inspection.
data/samples/images/*.jpg	Sample images corresponding to the sample data.

The exact schema is documented in:

docs/schema.md

Dataset Size

The release-preparation validation found:

• data/processed/gpt_input.csv: 12,000 rows;
• data/processed/output_results_full.csv: 12,000 rows;
• data/processed/output_results.csv: 61 rows;
• data/processed/gpt_text_output_results.csv: 11,980 rows;
• data/images/: 12,003 local image files;
• output_results_full: 11,989 unique referenced images, 0 missing referenced images;
• output_results: 61 referenced images, 0 missing referenced images.

These counts should be revalidated before each new public release.

Sample Data

A small sample is included for inspection:

data/samples/sample_output_results.csv
data/samples/images/

This sample is intended to show the expected structure of the multimodal model-output data without requiring users to inspect the full dataset first.

Reproducing and Reusing the Dataset

The repository contains scripts used during dataset construction. The main processing components include:

• text normalization;
• tokenization;
• GPT text-only prompting;
• GPT multimodal prompting;
• output cleanup;
• aligned frame extraction from source videos.

The public release prioritizes clean reuse of the final processed CSV files and aligned images. It does not include all local intermediate workbooks used during development.

The original experiment videos are not included. If frame regeneration is needed, place local source videos under:

data/raw/videos/

Extracted and released aligned images are stored under:

data/images/

Script usage notes are documented in:

docs/script_usage.md

Environment Setup

Create a Python environment and install dependencies:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For scripts that call the OpenAI API, create a local .env file on your machine:

OPENAI_API_KEY=your_key_here

Do not commit .env.

A safe template is provided in:

.env.example

Checksums

SHA-256 checksums for public artifacts are provided in:

checksums/checksums_sha256.txt

Users can verify downloaded or cloned files by comparing local SHA-256 hashes against this file.

Important Release Notes

This repository should not include:

• real API keys;
• .env files;
• raw source videos;
• intermediate workbooks;
• duplicate .xlsx exports;
• meta.tsv or vecs.tsv unless they are documented as meaningful artifacts in a future release;
• large third-party embedding files such as GloVe;
• operating-system or Python cache files.

Intended Use

SCOUT++ is intended for research on:

• human-robot interaction;
• situated dialogue;
• multimodal command understanding;
• natural language command classification;
• data-efficient learning for robotics;
• language-and-vision model evaluation in navigation-like tasks.

Out-of-Scope Use

SCOUT++ should not be used as:

• a general-purpose surveillance dataset;
• a biometric identification dataset;
• a dataset for claims outside the SCOUT task domain;
• an official replacement for the original ARL SCOUT release;
• evidence of safety-critical robot performance without additional validation.

Citation

If you use SCOUT++, please cite the SCOUT++ release and the associated SCOUT++ paper when available.

Also cite the original SCOUT paper:

Lukin, Stephanie M., Claire Bonial, Matthew Marge, Taylor Hudson,
Cory J. Hayes, Kimberly A. Pollard, Anthony Baker, Ashley N. Foots,
Ron Artstein, Felix Gervits, Mitchell Abrams, Cassidy Henry,
Lucia Donatelli, Anton Leuski, Susan G. Hill, David Traum,
and Clare R. Voss. 2024. SCOUT: A Situated and Multi-Modal
Human-Robot Dialogue Corpus.

A complete citation file is maintained in:

CITATION.cff

License

License details are provided in:

LICENSE

SCOUT++ is a derivative, unofficial research release. Users should also respect the release terms and citation expectations of the original ARL-SCOUT corpus.