V3: replace parked imports with first-class statistical History observations

[Loufe]

Parent: #33. Supersedes the adopt-or-drop direction previously recorded here.

Decision

V3 needs one first-class, immutable model for historical statistical observations. Historical evidence is not a FileRecord, standing verdict, output, or run, and its validity does not depend on whether the source still exists on disk.

The only tolerated V2 seam remains the standalone disposable translator (tools/export_history_v3.py). The Rust application accepts only its current native History import schema and contains no V2 compatibility model.

Problem

The current imported-history design correctly tried to preserve years of V2 statistics, including records whose files moved or were deleted, but represented them as ParkedRecord/ImportedProvenance and later tried to adopt them into current operational state.

That conflated two independent questions:

1. What happened historically? This is immutable evidence for History, Statistics, and Estimation.
2. What is true of the content currently at this path? This is a freshness-checked FileRecord/standing-verdict concern.

The previous adopt-or-drop proposal removed the compatibility machinery but also discarded the only carrier of unmatched historical measurements. That contradicts #52's requirement that moved/deleted V2 history remain visible and would permanently weaken estimates.

The nullable summary fields carried by imported VerdictKind::Converted values are a symptom of the same modelling error: historical measurements currently reach projections through a standing verdict.

Model

Add one compact durable observation type, approximately:

struct HistoryRecord {
    id: HistoryId,
    source: HistoricalSource,
    happened_at: Option<UnixMillis>,
    input: HistoricalInput,
    outcome: HistoricalOutcome,
}

struct HistoricalSource {
    path: Option<HistoricalPath>,
    anonymous_id: AnonymousHistoryId,
}

There is no separate label field and no unavailable state. The anonymous identifier is always retained; the optional historical path is present only while readable source data is available. A missing path is simply None. A missing historical timestamp also stays None; import time is not substituted for an unknown event time.

HistoricalInput retains the facts honestly known at observation time, including optional source size, duration, dimensions, source codec, container, and audio codecs.

HistoricalOutcome is typed:

• Converted: output size, analysis/encoding time, CRF, VMAF, target, preset, and other known execution context.
• Analyzed: analysis time, predicted output size, CRF, VMAF, target, and preset.
• NotWorthwhile: analysis time and target/floor evidence.
• Remuxed: output size and remux time.

Discovered/scanned cache state is not History. Failed/stopped activity remains in ConversionRun; it is not a statistical observation. A History UI may present both projections, but the durable models remain separate.

A HistoryRecord has no referential-integrity dependency on FileRecord, ContentKey, StandingVerdict, OutputRecord, or ConversionRun. Moving/deleting a file or replacing/clearing a verdict must have zero effect on History, Statistics, and Estimation.

Native recording

Each reportable native terminal outcome materializes exactly one immutable HistoryRecord in the same durable transaction that records completion. Runs remain canonical for execution/recovery; History is canonical for reporting and estimation. Do not add a second writer that later reconstructs observations from runs or verdicts.

History is append-only per reportable operation. Converting the same source twice produces two observations. User-facing totals therefore count conversions/operations, not unique files. A translated V2 row supplies one observation because V2 retained one available fact per source entry.

V2 translation and native import

The boundary is:

V2 JSON
  -> tools/export_history_v3.py
  -> strict current native V3 History import document
  -> HistoryRecord

Only the Python script understands V2 field names, statuses, hashes, timestamp peculiarities, and units. It must translate both raw-path and already-anonymized V2 entries, including sparse measurements and analysis time.

The Rust/TypeScript application must contain no:

• V2 structs, status mappings, field aliases, or version branches;
• ParkedRecord, ImportedProvenance, adoption rank, or scan-time adoption;
• fallback parsing or migration helper;
• path_hash/filename_hash semantics.

Each imported observation carries a stable opaque HistoryId:

• unseen ID: insert;
• same ID and identical contents: no-op;
• same ID with different contents: report a conflict;
• never silently overwrite or double-count.

Unknown measurements remain absent, never zero. Invalid/non-finite/out-of-range values are rejected or omitted according to the native field contract rather than invented.

Statistics and estimation

Statistics and estimation consume HistoryRecord directly, identically for native and imported observations. They never reach historical evidence through a standing verdict.

Each metric uses its own eligible cohort:

• conversion count: every Converted observation;
• savings: input and output size from the same observation;
• throughput: input size and encoding time from the same observation;
• CRF/VMAF aggregates: observations containing those values;
• encoding estimate: positive duration and encoding time from the same observation;
• analysis estimate: positive duration and analysis time from the same observation.

Encoding efficiency remains encoding_time / source_duration, grouped by compatible execution settings, source codec, and resolution bucket with broader fallbacks. Preset and other known context must be retained so incompatible samples are not silently pooled. Fresh per-file analysis may still override a historical prediction.

Level 1 work in #57 should consume eligible History observations immediately. Queue applicability and historical achievement for the current disk row remain freshness-checked operational questions.

Privacy

• The one-way anonymous identifier is retained on every History observation.
• If History anonymization is enabled, create the observation with source.path = None.
• History scrub performs the same durable, idempotent logical transform on existing observations: set the optional readable path to None.
• Scrub does not change the History ID, anonymous identifier, measurements, outcome, Statistics, or Estimation.
• Scrub makes no physical-storage-erasure promise and does not require journal compaction, database vacuuming, database replacement, or forensic byte-search tests.
• Already-anonymized translated V2 records remain usable for Statistics and Estimation.
• An observation without a readable path has no Open/Reveal action. A path-bearing observation may offer a best-effort action; the path is descriptive historical data and may be stale.
• The external translated import file may itself contain raw paths. UI copy must not imply that scrubbing application state also rewrites that separate file.
• Paths needed for live queue/output recovery are operational state with their own privacy scope; “Scrub History” must not claim to remove those.

Decision restoration is separate

History import performs no stat, ffprobe, ContentKey computation, or tool invocation and succeeds independently of current disk contents.

If restoring current-file standing verdicts from old data is still desired, model it as a separate optional native decision-candidate workflow. It may verify a raw path and create an ordinary verdict, but failure/offline/anonymized input does not alter or remove the imported History observation. Do not recreate parked state.

History delivery

Do not add the resulting potentially multi-megabyte collection to a TypeScript durable fold:

• keep History records in Rust-owned storage;
• serve filtered/paginated History rows by request;
• expose only a History revision/count through the ordinary snapshot stream;
• invalidate/refetch after History changes.

Statistics remains request-driven over the same canonical records.

ADR and issue updates

• Complete V3 architecture research: choose the durable state model and storage engine #89 and record the selected durable-state architecture.
• Add a new ADR that supersedes the imported-record portions of ADR-015.
• Update V3 privacy: history scrub as a durable transform, and wire anonymize_history #51 for the narrow readable-path transform above.
• Update V3 Analysis: derive applicable levels, size/time predictions, and confidence #57 so imported and native observations share estimation eligibility.
• Keep V3 Analysis: add deterministic bulk Analyze and Convert actions #58 queue filtering/verdict applicability operational and independent from History.
• Pull the relevant part of V3: serve first-class History by request/response #77 into this release work.
• Keep any current-file decision restoration outside this issue.

Acceptance

• One first-class HistoryRecord path serves native and imported statistical observations.
• Reportable native terminal outcomes atomically create exactly one observation in the same durable transaction as completion.
• Deleting/moving a source or clearing/replacing a verdict does not change History, Statistics, or Estimation.
• The standalone Python translator covers raw-path, already-anonymized, sparse, and all supported V2 outcome fixtures.
• Rust strictly parses only the current native import schema.
• Re-import is idempotent; changed contents under an existing ID are reported as conflicts.
• Missing values never become zero and every aggregate uses paired measurements from the same observation.
• Imported and native observations have identical statistics/estimation behavior.
• Privacy-on import and scrub leave the optional readable History path absent while retaining the anonymous identifier and all statistical evidence; scrub is idempotent.
• Scrub makes no physical-storage-erasure, compaction, vacuuming, or database-replacement promise.
• History import performs no filesystem/media probing and cannot fail because FFmpeg/ffprobe is unavailable.
• History rows are request-driven/paginated rather than mirrored into TypeScript state.
• Path-only actions are unavailable for anonymous rows and fail gracefully for stale path rows.
• A search across Rust and ui/ finds no V2, parked, adoption, or imported-provenance concepts.
• Generated bindings, fixtures, storage transaction tests, Statistics tests, and Estimation tests cover the new model.

Sequencing

Land as part of the durable-storage cutover selected by #89, before #57/#58. This avoids adding a new first-class model to the journal only to remove that persistence path immediately afterward. It also removes the imported-history seams those issues would otherwise encode and establishes the historical evidence source their estimators need.

[Loufe]

Third-review addendum (at 75c79c3), so the rip-out takes the whole denormalization with it: the nullable carried summary fields on VerdictKind::Converted (sizes, encoding time, crf, vmaf, absorbed by the fold at state.rs:571-609) exist only to serve adopted import verdicts; for native verdicts the carried copy buys nothing because conversion_runs and outputs have no removal delta and are always present. The stated goal ("consumers never branch on provenance", media.rs:152-157) is not achieved anyway: joined_sizes (projection.rs:165-218), collect_stat_facts, and verdict_row all still implement measured-vs-carried priority chains. When provenance goes, also remove the carried fields, the fold absorption, and the fallback chains, plus comments describing run pruning that no delta can perform (projection.rs:55-56).