Preference Data Curation Pipeline

Production-style pipeline for curating RLHF preference data from open datasets. Handles loading, quality filtering, near-duplicate removal, eval contamination checks, and DPO-ready JSONL output — the data side of the post-training stack.

---

Why this matters

Post-training quality is bounded by data quality. A model trained on noisy, duplicate, or contaminated preference pairs learns spurious shortcuts instead of genuine reward signal. This pipeline enforces:

• Quality gates — length bounds, score margin requirements, refusal filtering, repetition detection
• Near-dedup — MinHash LSH removes semantically identical prompts that would inflate training signal
• Contamination check — strips eval set prompts from train before any fine-tuning run
• Reproducible splits — deterministic train/val split with per-source provenance in every record

---

Supported datasets

All loaded from HuggingFace datasets — no local files required.

Dataset	Records (full)	Type	Key feature
Anthropic/hh-rlhf (helpful-base)	~44k	Human preference pairs	Chosen/rejected already labeled
Anthropic/hh-rlhf (harmless-base)	~43k	Human preference pairs	Safety-focused pairs
openbmb/UltraFeedback	~64k	GPT-4 scored responses	4 responses/prompt with 0–5 scores
OpenAssistant/oasst1	~88k msgs	Ranked conversation tree	Human rank → preference pairs
HuggingFaceH4/ultrachat_200k	~200k	Instruction following	High-quality SFT warm-up data

---

Pipeline stages

load_sources()         HH-RLHF + UltraFeedback + OASST1
      ↓
apply_filters()        length · score gap · refusal · repetition
      ↓
dedup_minhash()        MinHash LSH (threshold=0.8) on prompt field
      ↓
remove_contamination() strip eval prompts from train (optional)
      ↓
write_splits()         train.jsonl + val.jsonl  (95/5 split)

---

Output format

Each JSONL line:

{
  "prompt":   "What is the capital of France?",
  "chosen":   "The capital of France is Paris ...",
  "rejected": "I don't know.",
  "source":   "hh-rlhf/helpful-base",
  "metadata": {"score_chosen": 4.5, "score_rejected": 2.0}
}

Three templates available:

Template	Use case
raw	Framework-agnostic; use with TRL DPOTrainer directly
chatml	Qwen2.5 / ChatML-format models
llama3	Llama-3 instruct models

---

Quick start

pip install datasets datasketch

# Default: HH-RLHF + UltraFeedback + OASST1, 5k per source
python scripts/run_pipeline.py

# ChatML format for Qwen2.5 DPO training
python scripts/run_pipeline.py \
    --sources hh_rlhf_helpful ultrafeedback oasst1 \
    --max_per_source 10000 \
    --template chatml \
    --output_dir output/qwen_dpo

# Smoke test (500 samples, fast exact dedup)
python scripts/run_pipeline.py \
    --sources hh_rlhf_helpful \
    --max_per_source 500 \
    --dedup_method exact \
    --output_dir output/smoke

# Analyze the result
python scripts/analyze_dataset.py output/default/train.jsonl

---

Dataset analysis (analyze_dataset.py)

==================================================
Dataset: output/default/train.jsonl
==================================================
Total records:    14,832
With rejected:    93.2%
With scores:      38.5%

Sources:
  hh-rlhf/helpful-base                  4,871
  hh-rlhf/harmless-base                 4,203
  ultrafeedback                         3,417
  oasst1                                2,341

Prompt length (chars): {'mean': 312.4, 'p50': 201.0, 'p75': 412.0, 'max': 2048.0}
Chosen length (chars): {'mean': 487.1, 'p50': 341.0, 'p75': 621.0, 'max': 4096.0}
Score gap (chosen - rejected): {'mean': 1.83, 'p50': 1.5, 'p75': 2.5, 'max': 5.0}

---

Filter configuration

Key parameters in configs/default.json:

Parameter	Default	Effect
min_prompt_len	10 chars	Drop trivially short prompts
max_prompt_len	2048 chars	Drop context-window-busting prompts
min_response_len	20 chars	Drop one-word responses
min_score_gap	0.5	UltraFeedback: chosen must beat rejected by ≥0.5
min_edit_distance_ratio	0.1	Drop near-identical chosen/rejected pairs
filter_refusals	true	Drop chosen responses that open with "I'm sorry / I cannot..."
filter_repetition	true	Drop responses with >40% repeated 5-grams

See configs/qwen_dpo.json for stricter settings tuned for DPO on Qwen2.5.

---

Project structure

preference-data-pipeline/
├── src/
│   ├── loader.py       # HuggingFace dataset loaders (HH-RLHF, UltraFeedback, OASST, UltraChat)
│   ├── filter.py       # Quality filters (length, score gap, refusal, repetition)
│   ├── dedup.py        # MinHash LSH + exact dedup + contamination removal
│   ├── formatter.py    # Normalise → DPO JSONL; raw / chatml / llama3 templates
│   └── pipeline.py     # Orchestrator: load → filter → dedup → write
├── scripts/
│   ├── run_pipeline.py     # CLI entry point
│   └── analyze_dataset.py  # Dataset stats + ASCII histograms
├── configs/
│   ├── default.json    # HH-RLHF + UltraFeedback + OASST, raw format
│   └── qwen_dpo.json   # Stricter filters, ChatML template for Qwen2.5 DPO
├── tests/
│   └── test_pipeline.py    # 14 unit tests — no HF downloads required
└── README.md

---

Integration with DPO training

The output train.jsonl plugs directly into TRL's DPOTrainer:

from datasets import load_dataset
from trl import DPOTrainer, DPOConfig

dataset = load_dataset("json", data_files={"train": "output/qwen_dpo/train.jsonl"})

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=DPOConfig(beta=0.1, ...),
    train_dataset=dataset["train"],
)
trainer.train()

---

Tests

python -m pytest tests/ -v
# 14 passed — filter, dedup, formatter, write_splits

---

Related repos

Repo	Connection
rlhf-synthesis-optimization	Trains DPO/GRPO/PPO on preference data — use this pipeline's output as input
LLM_FineTuning_SFT_Production	SFT stage; UltraChat output from this pipeline works as SFT data
efficient-post-training-suite	Full 6-stage pipeline; this repo covers Stage 1 (data preparation)