PromptSecurity CLI

PromptSecurity is operated through a single CLI entrypoint: python -m experiments. The workflow is placeholder-first: the CLI creates placeholder JSON files, then executes them. Results are stored under experiments/placeholders/.

The anonymous release also includes an evaluation-result dataset under dataset/promptsecurity_eval/. See dataset/README.md for the file layout, record counts, and loading examples.

Note: the CLI prints a short usage example block on every run. Use --show-examples if you only want that output and then exit.

Quick Start

# List available components
python -m experiments --list all

# Run a single experiment (auto-create placeholder + execute)
python -m experiments --model gpt-4o --attack ArtPrompt --defense no_defense --dataset harmbench --judger harmbench_judger

# Limit to 5 samples
python -m experiments --model gpt-4o --attack ArtPrompt --sample-limit 5

# Generate only, run later
python -m experiments --model gpt-4o --attack ArtPrompt --generate-only
python -m experiments --run-placeholders

# Run existing placeholders with filters
python -m experiments --run-placeholders --placeholder-status pending --run-model gpt-4o --run-limit 3

# Dashboard
python -m experiments --dashboard --dashboard-model gpt-4o --dashboard-limit 10

Core CLI Workflow

1) Create and run placeholders

python -m experiments \
  --model gpt-4o \
  --attack ArtPrompt \
  --defense no_defense \
  --dataset harmbench \
  --judger harmbench_judger

If you omit any of the five elements, defaults are applied: model=gpt-4o, attack=no_attack, defense=no_defense, dataset=harmbench, judger=harmbench_judger.

2) Generate only (no execution)

python -m experiments --model gpt-4o --attack ArtPrompt --generate-only

3) Run existing placeholders

python -m experiments --run-placeholders
python -m experiments --run-placeholders --placeholder-status pending
python -m experiments --run-placeholders --run-attack ArtPrompt --run-limit 5

4) Resume failed experiments

python -m experiments --resume

Batch Mode (Combinatorial)

If you pass multiple values for any of --model/--attack/--defense/--dataset/--judger, the CLI generates all combinations:

python -m experiments \
  --model gpt-4o claude-3-5-sonnet \
  --attack no_attack ArtPrompt \
  --defense no_defense smooth_llm \
  --dataset harmbench jbb \
  --sample-limit 5

Multi-judger in a single experiment

By default, multiple judgers create combinations. Use --multi-judger to evaluate all specified judgers in the same run:

python -m experiments --model gpt-4o --attack ArtPrompt --judger harmbench_judger gpt_judger_harmful_binary --multi-judger

Phase Experiments

Phase 1 and Phase 4 are placeholder-based:

python -m experiments --phase 1 --sample-limit 30
python -m experiments --phase 4 --generate-only

Dashboard and Reporting

python -m experiments --dashboard
python -m experiments --dashboard --dashboard-dataset harmbench --dashboard-sort asr
python -m experiments --dashboard --placeholders-dir /path/to/custom/placeholders

Inspect Methods

python -m experiments --list attacks
python -m experiments --list models
python -m experiments --info ArtPrompt attack

Config Files

Use --config with JSON or YAML:

python -m experiments --config my_experiment.json

Example JSON:

{
  "model": "gpt-4o",
  "attack": "ArtPrompt",
  "defense": "no_defense",
  "dataset": "harmbench",
  "judger": "harmbench_judger",
  "sample_limit": 10,
  "seed": 42
}

Example YAML:

model: gpt-4o
attack: ArtPrompt
defense: no_defense
dataset: harmbench
judger:
  - harmbench_judger
  - gpt_judger_harmful_binary
sample_limit: 10
seed: 42

Environment Variables

You can provide defaults via env vars:

export PS_MODEL=gpt-4o
export PS_ATTACK=ArtPrompt
export PS_DEFENSE=no_defense
export PS_DATASET=harmbench
export PS_JUDGER=harmbench_judger
export PS_SAMPLE_LIMIT=10
export PS_VERBOSE=true
python -m experiments

Useful Flags

• --sample-limit: limit number of samples per experiment
• --seed: control deterministic sampling (default 42)
• --verbose: show detailed execution logs
• --max-length: cap verbose text output length
• --generate-only: only create placeholders
• --run-placeholders: execute existing placeholders
• --list-placeholders: list placeholder files and status
• --placeholder-status: filter placeholders by status
• --run-model/--run-attack/--run-defense/--run-dataset/--run-judger: filter runs
• --run-limit: limit how many placeholders to run
• --baseline/--security-test/--defense-eval: preset combos

Advanced: Parallel Placeholder Runner

For multi-worker runs, use the dedicated runner:

python -m experiments.core.placeholder_runner --run-all --workers 4
python -m experiments.core.placeholder_runner --run experiments/placeholders/your_file.json

Help

python -m experiments --help