feat(ltx2): add CLAP-only T2AV reward recipe

datasets/README.md

@@ -63,6 +63,7 @@ needs a `.gitignore` line, which keeps that decision explicit.
 
 | Folder | What |
 |---|---|
+| [`audiocaps/`](audiocaps/README.md) | AudioCaps captions for LTX-2.3 CLAP audio-reward RL |
 | [`arxivqa_mc/`](arxivqa_mc/README.md) | ArxivQA scientific-figure multiple-choice (BAGEL GRPO) |
 | [`asearcher/`](asearcher/README.md) | ASearcher deep-research prompts (agentic RL) |
 | [`daily_omni_av/`](daily_omni_av/README.md) | Daily-Omni audio-video QA |

datasets/audiocaps/README.md

@@ -0,0 +1,95 @@
+# AudioCaps for LTX-2.3 CLAP training
+
+This converter prepares the official human-written captions from
+[AudioCaps](https://github.com/cdjkim/audiocaps) for the LTX-2.3 CLAP reward recipe.
+AudioCaps was introduced at NAACL 2019 and contains captions for AudioSet clips.
+
+The generated JSONL is a local artifact and must not be committed. This recipe does not
+download or consume the source audio: LTX-2.3 generates audio from each caption and CLAP
+scores that generated waveform against the same official caption.
+
+The event-grounded variant also joins the AudioCaps clips to their human-confirmed AudioSet
+labels. It uses those labels as positive sound-event targets and RMS-normalizes generated
+audio before CLAP scoring so increasing waveform gain cannot improve the reward by itself.
+
+## Source and terms
+
+By default, the converter reads the official CSV files at AudioCaps commit
+`d004db3ea1b01cf4fd0347dd8d27db90cadc8809`:
+
+- train: 49,838 clips with one caption each
+- validation: 495 clips with five captions each (2,475 caption rows)
+- test: 975 clips with five captions each (4,875 caption rows)
+
+The upstream repository says its code and dataset are free to use for academic purposes and
+asks users to cite the AudioCaps paper. Review the upstream terms before redistributing or
+using the data outside that scope.
+
+## Cook
+
+From the repository root:
+
+```bash
+python datasets/audiocaps/prepare_audiocaps.py --out-dir data/audiocaps
+```
+
+The default manifests contain the full official train split and 64 distinct clips sampled
+deterministically from the official validation split. Validation has five captions per clip;
+the converter chooses one deterministically so a clip is not counted five times. Pass
+`--keep-all-eval-captions --eval-limit 0` to retain every validation caption.
+
+For the event-grounded recipe, download `class_labels_indices.csv`, `eval_segments.csv`,
+`balanced_train_segments.csv`, and `unbalanced_train_segments.csv` from the official
+[AudioSet download page](https://research.google.com/audioset/download.html), place them in
+one directory, and run:
+
+```bash
+python datasets/audiocaps/prepare_audiocaps.py \
+  --out-dir data/audiocaps-event-grounded \
+  --audioset-metadata-dir /path/to/audioset-metadata
+```
+
+The converter requires an exact AudioSet segment match for every selected AudioCaps clip and
+fails instead of silently emitting an ungrounded row.
+
+Each row uses the original AudioCaps caption as both the generation prompt and CLAP target:
+
+```json
+{
+  "prompt": "Multiple clanging and clanking sounds",
+  "metadata": {
+    "negative_audio_captions": ["... seven captions from other rows ..."],
+    "audio_event_labels": ["Door", "Sliding door"],
+    "audiocap_id": "58146",
+    "source_dataset": "AudioCaps",
+    "source_split": "train"
+  }
+}
+```
+
+The CLAP scorer reads the positive text directly from `prompt`; no separate
+audio-caption field or hand-written rewrite is required for this dataset.
+
+The deterministic negative captions provide the hardest-negative retrieval margin used by
+the CLAP recipe and its retrieval diagnostics. The event-grounded recipe additionally scores
+the least-aligned positive AudioSet label, encouraging every labeled event to remain audible.
+
+## Train
+
+```bash
+DATA_PATH=data/audiocaps/train.jsonl \
+EVAL_DATA_PATH=data/audiocaps/eval.jsonl \
+bash examples/run_experiment_single_node.sh \
+  diffusion/ltx2/ltx2_3_t2av_clap_trainside
+```
+
+Train and evaluation use the official AudioCaps train/validation split boundary.
+
+To train with RMS normalization and AudioSet event coverage:
+
+```bash
+DATA_PATH=data/audiocaps-event-grounded/train.jsonl \
+EVAL_DATA_PATH=data/audiocaps-event-grounded/eval.jsonl \
+bash examples/run_experiment_single_node.sh \
+  diffusion/ltx2/ltx2_3_t2av_clap_event_grounded_trainside
+```

datasets/audiocaps/prepare_audiocaps.py

@@ -0,0 +1,197 @@
+"""Convert official AudioCaps captions to UniRL prompt JSONL files."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import io
+import json
+import os
+import random
+import urllib.request
+from typing import Dict, Iterable, List, Optional, Tuple
+
+DEFAULT_SOURCE = "https://raw.githubusercontent.com/cdjkim/audiocaps/d004db3ea1b01cf4fd0347dd8d27db90cadc8809"
+AudioSetKey = Tuple[str, int]
+
+
+def _audioset_key(youtube_id: str, start_time: str | float) -> AudioSetKey:
+    return str(youtube_id).strip(), round(float(start_time) * 1000)
+
+
+def _read_audioset_labels(metadata_dir: str) -> Dict[AudioSetKey, List[str]]:
+    """Read the three official AudioSet segment files and resolve their label names."""
+    label_path = os.path.join(metadata_dir, "class_labels_indices.csv")
+    with open(label_path, encoding="utf-8", newline="") as handle:
+        display_names = {row["mid"]: row["display_name"] for row in csv.DictReader(handle)}
+
+    labels_by_segment: Dict[AudioSetKey, List[str]] = {}
+    filenames = ("eval_segments.csv", "balanced_train_segments.csv", "unbalanced_train_segments.csv")
+    for filename in filenames:
+        with open(os.path.join(metadata_dir, filename), encoding="utf-8", newline="") as handle:
+            rows = csv.reader((line for line in handle if not line.startswith("#")), skipinitialspace=True)
+            for row in rows:
+                key = _audioset_key(row[0], row[1])
+                mids = [mid.strip() for mid in row[3].strip().strip('"').split(",")]
+                labels = [display_names[mid] for mid in mids]
+                previous = labels_by_segment.setdefault(key, labels)
+                if previous != labels:
+                    raise ValueError(f"Conflicting AudioSet labels for segment {key!r}.")
+    return labels_by_segment
+
+
+def _read_rows(source: str, split: str) -> List[Dict[str, str]]:
+    filename = f"{split}.csv"
+    if os.path.isdir(source):
+        with open(os.path.join(source, filename), encoding="utf-8", newline="") as handle:
+            return list(csv.DictReader(handle))
+
+    url = f"{source.rstrip('/')}/dataset/{filename}"
+    with urllib.request.urlopen(url, timeout=60) as response:  # noqa: S310 - user-selectable dataset source
+        content = response.read().decode("utf-8-sig")
+    return list(csv.DictReader(io.StringIO(content)))
+
+
+def _select_rows(
+    rows: List[Dict[str, str]],
+    limit: int,
+    seed: int,
+    *,
+    one_caption_per_clip: bool,
+) -> List[Dict[str, str]]:
+    valid = [row for row in rows if str(row.get("caption") or "").strip()]
+    if one_caption_per_clip:
+        rows_by_clip: Dict[tuple[str, str], List[Dict[str, str]]] = {}
+        for row in valid:
+            rows_by_clip.setdefault((row["youtube_id"], row["start_time"]), []).append(row)
+        rng = random.Random(seed)
+        valid = [rng.choice(rows_by_clip[clip_key]) for clip_key in sorted(rows_by_clip)]
+    if limit <= 0 or limit >= len(valid):
+        return valid
+    indices = list(range(len(valid)))
+    random.Random(seed).shuffle(indices)
+    return [valid[index] for index in sorted(indices[:limit])]
+
+
+def _negative_captions(
+    rows: List[Dict[str, str]],
+    index: int,
+    count: int,
+    *,
+    seed: int,
+    split: str,
+) -> List[str]:
+    target = rows[index]["caption"].strip()
+    rng = random.Random(f"{seed}:{split}:{rows[index]['audiocap_id']}")
+    negatives: List[str] = []
+    attempted_indices = set()
+    while len(attempted_indices) < len(rows):
+        candidate_index = rng.randrange(len(rows))
+        if candidate_index in attempted_indices:
+            continue
+        attempted_indices.add(candidate_index)
+        caption = rows[candidate_index]["caption"].strip()
+        if candidate_index != index and caption != target and caption not in negatives:
+            negatives.append(caption)
+            if len(negatives) == count:
+                return negatives
+    raise ValueError(f"Could not find {count} distinct negative captions for AudioCaps row {index}.")
+
+
+def _write_split(
+    rows: List[Dict[str, str]],
+    out_path: str,
+    *,
+    source_split: str,
+    negative_count: int,
+    seed: int,
+    audioset_labels: Optional[Dict[AudioSetKey, List[str]]],
+) -> None:
+    os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
+    with open(out_path, "w", encoding="utf-8") as output:
+        for index, row in enumerate(rows):
+            caption = row["caption"].strip()
+            metadata = {
+                "negative_audio_captions": _negative_captions(
+                    rows,
+                    index,
+                    negative_count,
+                    seed=seed,
+                    split=source_split,
+                ),
+                "audiocap_id": str(row["audiocap_id"]),
+                "youtube_id": str(row["youtube_id"]),
+                "start_time": float(row["start_time"]),
+                "source_dataset": "AudioCaps",
+                "source_split": source_split,
+            }
+            if audioset_labels is not None:
+                key = _audioset_key(row["youtube_id"], row["start_time"])
+                if key not in audioset_labels:
+                    raise ValueError(f"AudioCaps row {row['audiocap_id']} has no matching AudioSet segment {key!r}.")
+                metadata["audio_event_labels"] = audioset_labels[key]
+            record = {
+                "prompt": caption,
+                "prompt_id": f"audiocaps:{source_split}:{row['audiocap_id']}",
+                "metadata": metadata,
+            }
+            output.write(json.dumps(record, ensure_ascii=False) + "\n")
+    print(f"wrote {len(rows)} records -> {out_path}")
+
+
+def _iter_specs(args: argparse.Namespace) -> Iterable[tuple[str, str, int, int, bool]]:
+    yield args.train_split, "train.jsonl", args.train_limit, args.seed, False
+    yield args.eval_split, "eval.jsonl", args.eval_limit, args.seed + 1, not args.keep_all_eval_captions
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--source", default=DEFAULT_SOURCE, help="AudioCaps repo URL or local directory")
+    parser.add_argument("--out-dir", default="data/audiocaps")
+    parser.add_argument("--train-split", default="train")
+    parser.add_argument("--eval-split", default="val")
+    parser.add_argument("--train-limit", type=int, default=0, help="0 keeps the full source split")
+    parser.add_argument("--eval-limit", type=int, default=64, help="0 keeps the full source split")
+    parser.add_argument("--negative-count", type=int, default=7)
+    parser.add_argument(
+        "--audioset-metadata-dir",
+        help="directory containing the three official AudioSet segment CSVs and class_labels_indices.csv",
+    )
+    parser.add_argument(
+        "--keep-all-eval-captions",
+        action="store_true",
+        help="keep all five validation captions per clip instead of choosing one deterministically",
+    )
+    parser.add_argument("--seed", type=int, default=42)
+    args = parser.parse_args()
+
+    if args.negative_count < 1:
+        parser.error("--negative-count must be at least 1")
+
+    audioset_labels = _read_audioset_labels(args.audioset_metadata_dir) if args.audioset_metadata_dir else None
+
+    for source_split, filename, limit, seed, one_caption_per_clip in _iter_specs(args):
+        source_rows = _read_rows(args.source, source_split)
+        selected_rows = _select_rows(
+            source_rows,
+            limit,
+            seed,
+            one_caption_per_clip=one_caption_per_clip,
+        )
+        if len(selected_rows) <= args.negative_count:
+            parser.error(
+                f"{source_split} produced {len(selected_rows)} rows, but --negative-count={args.negative_count} "
+                "requires at least one more row"
+            )
+        _write_split(
+            selected_rows,
+            os.path.join(args.out_dir, filename),
+            source_split=source_split,
+            negative_count=args.negative_count,
+            seed=args.seed,
+            audioset_labels=audioset_labels,
+        )
+
+
+if __name__ == "__main__":
+    main()

examples/diffusion/ltx2/ltx2_3_t2av_clap_ast_event_grounded_trainside.yaml

@@ -0,0 +1,19 @@
+# @package _global_
+# LTX-Video-2.3 T2AV Flow-GRPO with CLAP alignment plus an independent
+# AudioSet AST event classifier. The AST term closes the gap where CLAP score
+# rises while a concrete event such as applause becomes less audible.
+
+defaults:
+  - ltx2_3_t2av_clap_event_grounded_trainside
+  - _self_
+
+save_dir: ${oc.env:SAVE_DIR,checkpoints/ltx2_3_t2av_clap_ast_event_grounded_trainside}
+
+logging:
+  tags: [ltx2.3, t2av, flowgrpo, trainside, audio, clap, ast, event-grounded, rms-normalized]
+
+reward:
+  backend:
+    config:
+      ast_event_weight: 1.0
+      ast_model_id: MIT/ast-finetuned-audioset-10-10-0.4593

examples/diffusion/ltx2/ltx2_3_t2av_clap_event_grounded_trainside.yaml

@@ -0,0 +1,31 @@
+# @package _global_
+# LTX-Video-2.3 T2AV Flow-GRPO with loudness-invariant, event-grounded CLAP reward.
+#
+# Cook AudioCaps with official AudioSet segment metadata before launching:
+#   python datasets/audiocaps/prepare_audiocaps.py \
+#     --out-dir data/audiocaps-event-grounded \
+#     --audioset-metadata-dir /path/to/audioset-metadata
+#
+# Launch (1 node x 8 GPUs):
+#   DATA_PATH=data/audiocaps-event-grounded/train.jsonl \
+#   EVAL_DATA_PATH=data/audiocaps-event-grounded/eval.jsonl \
+#     bash examples/run_experiment_single_node.sh \
+#       diffusion/ltx2/ltx2_3_t2av_clap_event_grounded_trainside
+
+defaults:
+  - ltx2_3_t2av_clap_trainside
+  - _self_
+
+save_dir: ${oc.env:SAVE_DIR,checkpoints/ltx2_3_t2av_clap_event_grounded_trainside}
+
+logging:
+  tags: [ltx2.3, t2av, flowgrpo, trainside, audio, clap, event-grounded, rms-normalized]
+
+reward:
+  backend:
+    config:
+      event_prompts_metadata_key: audio_event_labels
+      event_coverage_weight: 1.0
+      audio_normalization: rms
+      target_rms_dbfs: -20.0
+      peak_limit: 0.95