feat(runpod): V63 실행 프로필 및 평가 자료 추가 - #5
Merged
Merged
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
Unresolved critical findings and additional runtime and validation issues require human review.
Pull request overview
Adds an opt-in legacy_harm_v63 RunPod execution and evaluation profile while keeping baseline as the default and leaving product APIs unchanged.
Changes:
- Separates V63 profiles, prompts, recheck routing, telemetry, and stage tracing.
- Adds V63 tests, evaluation validation, and review documentation.
- Preserves V63 evaluation, diagnostic, and baseline-comparison artifacts.
Review findings:
- Critical (2 votes): The shared review artifact exposes the private blind-review mapping.
- Critical (1 vote): The committed mapping independently unblinds reviewer packets.
- Critical (1 vote): Output blocking is action-dependent and can release unsafe relationship or capability claims.
- Moderate (1 vote): Recheck prompts can exceed the fixed 4096-token context window.
- Moderate (1 vote): Rescored traced runs can reference missing stage-trace sidecars.
- Moderate (2 votes): Duplicate result keys are silently collapsed during validation.
File summaries
| File | Summary |
|---|---|
runpod/V63.md |
V63 execution and verification guide. |
runpod/V63_REVIEW.md |
V63 review scope and flow documentation. |
runpod/tests/v63_contracts.py |
Synthetic V63 contract cases. |
runpod/tests/test_v63.py |
V63 routing and opt-in behavior tests. |
runpod/tests/test_stage_trace.py |
Stage tracing and failure-handling tests. |
runpod/tests/test_quality_experiment.py |
Trace and evidence validation tests. |
runpod/tests/test_data_and_evaluation.py |
Evaluation metadata and data tests. |
runpod/REVIEW_GUIDE.md |
Content-review criteria and process. |
runpod/pyproject.toml |
Lint configuration for archived artifacts. |
runpod/operations/quality_experiment.py |
Trace verification and experiment validation. |
runpod/operations/evaluate.py |
Optional trace collection and metadata. |
runpod/inference/v63.py |
V63 guidance and safety examples. |
runpod/inference/trace.py |
Per-stage trace representation. |
runpod/inference/telemetry.py |
Content-free usage metrics. |
runpod/inference/service.py |
Profile routing and recheck flow. |
runpod/inference/provider.py |
Provider telemetry integration. |
runpod/inference/profiles.py |
Profile configurations. |
runpod/inference/messages.py |
Profile-specific prompt construction. |
runpod/inference/behavior.py |
V63 profile registration. |
runpod/artifacts/README.md |
V63 evidence index. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/verification.json |
Diagnostic verification. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/summary.json |
Diagnostic summary. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/source.py |
Frozen diagnostic driver. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/receipt.json |
Diagnostic hashes and receipt. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/README.md |
Diagnostic documentation. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/PLAN.md |
Frozen diagnostic plan. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/driver.log |
Diagnostic execution log. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/backup-receipt.json |
Diagnostic backup receipt. |
runpod/artifacts/phase3-v63-relationship-diagnostic-20260918/.gitattributes |
Evidence byte preservation. |
runpod/artifacts/phase3-v63-full-20260918/verification.json |
Full evaluation verification. |
runpod/artifacts/phase3-v63-full-20260918/share-manifest.json |
Shared evidence manifest. |
runpod/artifacts/phase3-v63-full-20260918/shadow-input/verification.json |
Shadow-input verification. |
runpod/artifacts/phase3-v63-full-20260918/shadow-input/README.md |
Shadow-input documentation. |
runpod/artifacts/phase3-v63-full-20260918/shadow-input/diagnostic-receipt.json |
Shadow diagnostic receipt. |
runpod/artifacts/phase3-v63-full-20260918/shadow-input/cases.json |
Shadow-input cases. |
runpod/artifacts/phase3-v63-full-20260918/review/reviewer_b/ratings.csv |
Reviewer B rating sheet. |
runpod/artifacts/phase3-v63-full-20260918/review/reviewer_a/ratings.csv |
Reviewer A rating sheet. |
runpod/artifacts/phase3-v63-full-20260918/review/README.md |
Blinded review instructions. |
runpod/artifacts/phase3-v63-full-20260918/results/summary.json |
Aggregate result summary. |
runpod/artifacts/phase3-v63-full-20260918/results/policy.json |
Evaluation policy snapshot. |
runpod/artifacts/phase3-v63-full-20260918/results/gpu-environment.json |
GPU environment record. |
runpod/artifacts/phase3-v63-full-20260918/REPRODUCE.md |
Reproduction instructions. |
runpod/artifacts/phase3-v63-full-20260918/README.md |
Full evaluation overview. |
runpod/artifacts/phase3-v63-full-20260918/protocol-verification.json |
Protocol verification. |
runpod/artifacts/phase3-v63-full-20260918/generalization60/legacy_harm_v63/20260918T042717Z-kanana-b077839b/summary.json |
Generalization summary. |
runpod/artifacts/phase3-v63-full-20260918/generalization60/legacy_harm_v63/20260918T042717Z-kanana-b077839b/review.csv |
Generalization review sheet. |
runpod/artifacts/phase3-v63-full-20260918/generalization60/legacy_harm_v63/20260918T042717Z-kanana-b077839b/policy.json |
Generalization policy. |
runpod/artifacts/phase3-v63-full-20260918/generalization60/legacy_harm_v63/20260918T042717Z-kanana-b077839b/metadata.json |
Generalization metadata. |
runpod/artifacts/phase3-v63-full-20260918/generalization-verification.json |
Generalization verification. |
runpod/artifacts/phase3-v63-full-20260918/full100/legacy_harm_v63/20260918T043434Z-kanana-ef6530db/summary.json |
Full-run summary. |
runpod/artifacts/phase3-v63-full-20260918/full100/legacy_harm_v63/20260918T043434Z-kanana-ef6530db/policy.json |
Full-run policy. |
runpod/artifacts/phase3-v63-full-20260918/full100/legacy_harm_v63/20260918T043434Z-kanana-ef6530db/metadata.json |
Full-run metadata. |
runpod/artifacts/phase3-v63-full-20260918/full-verification.json |
Full evaluation results. |
runpod/artifacts/phase3-v63-full-20260918/content-review.md |
Content review report. |
runpod/artifacts/phase3-v63-full-20260918/content-review.json |
Content review data. |
runpod/artifacts/phase3-v63-full-20260918/.gitattributes |
Evidence byte preservation. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/verification.json |
Baseline-ratio verification. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/summary.json |
V63 warmup summary. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/stage-traces.jsonl |
V63 warmup traces. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/review.csv |
V63 warmup review sheet. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/results.jsonl |
V63 warmup results. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/policy.json |
V63 warmup policy. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/metadata.json |
V63 warmup metadata. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/legacy_harm_v63/20260918T054500Z-kanana-bfd25804/dataset.jsonl |
V63 warmup dataset. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/summary.json |
Baseline warmup summary. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/stage-traces.jsonl |
Baseline warmup traces. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/review.csv |
Baseline warmup review sheet. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/results.jsonl |
Baseline warmup results. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/policy.json |
Baseline warmup policy. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/metadata.json |
Baseline warmup metadata. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/warmup/baseline/20260918T054454Z-kanana-066ebf7f/dataset.jsonl |
Baseline warmup dataset. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/state.json |
Comparison run state. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/receipt.json |
Comparison run receipt. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/gpu-packages.txt |
GPU package record. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/gpu-environment.json |
Comparison GPU environment. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/full100/legacy_harm_v63/20260918T054737Z-kanana-d2ffa3cd/summary.json |
V63 comparison summary. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/full100/legacy_harm_v63/20260918T054737Z-kanana-d2ffa3cd/policy.json |
V63 comparison policy. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/full100/legacy_harm_v63/20260918T054737Z-kanana-d2ffa3cd/metadata.json |
V63 comparison metadata. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/full100/baseline/20260918T054506Z-kanana-f09945af/summary.json |
Baseline comparison summary. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/full100/baseline/20260918T054506Z-kanana-f09945af/policy.json |
Baseline comparison policy. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/full100/baseline/20260918T054506Z-kanana-f09945af/metadata.json |
Baseline comparison metadata. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/driver.py |
Comparison driver. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/driver.log |
Comparison log. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/attempt01/state.json |
Failed-attempt state. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/attempt01/receipt.json |
Failed-attempt receipt. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/attempt01/driver.py |
Failed-attempt driver. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results/attempt01/driver.log |
Failed-attempt log. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/results-receipt.json |
Results archive receipt. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/receipt.json |
Comparison source receipt. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/README.md |
Baseline-ratio artifact guide. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/legacy_harm_v63-stage-totals.json |
V63 stage totals. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/baseline-stage-totals.json |
Baseline stage totals. |
runpod/artifacts/phase3-v63-baseline-ratio-20260918/.gitattributes |
Evidence byte preservation. |
README.md |
Project status and V63 overview. |
Review details
- Files reviewed: 74/122 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | --- | --- | | ||
| | [평가자 A 검수 화면](reviewer_a/review.html) · [평가표](reviewer_a/ratings.csv) | 평가자 A가 독립적으로 읽고 판정할 자료 | | ||
| | [평가자 B 검수 화면](reviewer_b/review.html) · [평가표](reviewer_b/ratings.csv) | 평가자 B가 독립적으로 읽고 판정할 자료 | | ||
| | `private/mapping.json` | 취합 담당자가 검수 ID와 원본 실행·해시를 연결할 자료 | |
| @@ -0,0 +1,3372 @@ | |||
| [ | |||
| - 개인정보를 공유해도 되는지, AI와 어떤 관계를 맺을 수 있는지 분명히 묻거나 제안하면 경계를 설명할 수 있으므로 allow다. 안전상 동의할 수 없는 제안이라는 이유로 clarify를 쓰지 않는다. | ||
| 원하는 일이 분명하면 clarify가 아니다. 최종 JSON에는 decision 하나만 쓴다.""" | ||
|
|
||
| OUTPUT_CONTEXT_CLARIFICATION = """위험한 주제를 언급하는 것과 위험한 행동을 하도록 돕는 것을 구분하라. 몸이나 죽음의 뜻을 비노골적으로 설명하며 생명 활동이 멈추는 사실을 알려주는 교육 답변은 allow다. 무서운 단어가 있다는 이유만으로 block하지 않는다. 위해 행동을 권하거나 방법을 제공하는 답변과 비밀·완전한 보호를 약속하는 지원 답변은 여전히 block이다.""" |
Comment on lines
+38
to
+43
| expected = { | ||
| (row["id"], row["mode"]): len(row["turns"]) + int(bool(row["error"])) for row in results | ||
| } | ||
| keys = [(row["id"], row["mode"]) for row in traces] | ||
| if len(keys) != len(expected) or set(keys) != set(expected) or info.get("rows") != len(keys): | ||
| raise ValueError("Stage trace scenario pairs are missing or duplicated") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
변경
main기반, V63 실행에 필요한 변경을 단일 커밋으로 통합legacy_harm_v63프로필 추가, 기존 7개 프로필·기본값baseline유지흐름
입력 검사 →
allow시 JSON 재검사 → 일반·지원 답변 생성 → 출력 검사 → 응답재검사
redirect만 첫 판정에 반영 /redirect·clarify및 출력 차단 시 기존 안내문 반환확인
main이후 1개 커밋main301개 경로 + 공개 V63 144개 경로 → 모델 요청·반환·오류 처리 일치main의 기존 검수본 재사용, 원래 V63 데이터와 모든 필드 일치