Skip to content

V3 History: capture and export portable performance observations #90

Description

@Loufe

Parent: #33

Related: #79, #89, #57, #77, #51

Status: open research, no schema decision

This issue records what V3 History may need to retain for:

  1. useful estimates on the user's own machine;
  2. an export the user can inspect and deliberately share;
  3. optional pooling of many users' observations for later estimator research.

It does not select:

Earlier drafts use seven logical headings to inventory possible facts. Those headings are not seven accepted tables. A later two-table sketch was also only a minimal physical mapping, not a settled reduction. The present working range is approximately three clearly motivated relational shapes, with a possible fourth child/profile table if a spike demonstrates value. This issue intentionally keeps the alternatives open.

Corrections to earlier discussion

One canonical export, not two reports

There should be one versioned canonical History bundle. The statistical observations and technical configuration are the same whether the bundle is kept personally or shared for research.

Readable local source names/paths, if the user explicitly includes them, belong in an optional attachment such as local_sources.ndjson. Omitting that attachment produces the pathless research bundle. We should not maintain separate personal and research statistical schemas.

Logical categories are not automatically tables

We need to distinguish:

  • information worth recording;
  • logical concepts useful in Rust/domain code or documentation;
  • physical SQL tables.

A table is justified by independent identity, reuse, one-to-many cardinality, integrity constraints, or demonstrated query value—not by the existence of a conceptual noun.

In particular:

  • runtime_context_id is not currently justified; runtime facts describe one observation and can be inline unless repeated sampling requires child rows;
  • content-complexity summaries describe one source observation and can be inline or a versioned payload;
  • execution settings may remain typed/structured on the observation or attempt with a deterministic fingerprint;
  • CRF-search/sample attempts are genuinely one-to-many and therefore have a stronger claim to a child table;
  • phase measurements may justify a child table if arbitrary/repeated spans must be queried, but fixed totals could be columns on the observation.

Established domain requirements

These requirements are independent of the final physical mapping:

  • History is first-class statistical evidence, not current operational file state.
  • A historical observation survives source deletion, movement, replacement, verdict clearing, and queue cleanup.
  • History has no lifecycle or foreign-key dependency on FileRecord, StandingVerdict, output, queue item, or current path.
  • A reportable native terminal result and its core History observation commit atomically (V3: replace parked imports with first-class statistical History observations #79, V3 architecture research: choose the durable state model and storage engine #89).
  • Failed/stopped runs remain operational evidence and are not automatically clean estimator observations.
  • Native and translated observations enter the same current V3 model. Only the disposable Python translator understands V2.
  • Unknown values remain null/absent, never zero.
  • Privacy scrub removes the optional readable historical path while preserving every technical/statistical fact. It does not promise physical byte erasure.
  • History remains queryable/paginated in the Rust backend rather than mirrored wholesale into frontend state.
  • Exact measurements, predictions, and derived aggregates must remain distinguishable.
  • Collection failure must not abort a conversion; optional probes are best-effort and carry collection provenance.

Prediction stages and leakage boundary

Every field needs an availability stage:

  1. Basic Scan / before CRF search
    • source metadata;
    • planned execution settings;
    • machine/platform/toolchain facts;
    • any genuinely cheap pre-analysis content features.
  2. After CRF search / before full encode
    • sample positions and sample encode timings;
    • attempted CRFs and measured quality scores;
    • selected CRF;
    • ab-av1 predicted size/duration.
  3. After terminal completion
    • actual phase and total times;
    • actual output sizes;
    • observed throughput;
    • retry/fallback outcome and runtime diagnostics.

#57's Level 1 estimator may use only stage-1 facts. Sample performance can improve the post-analysis estimate, but using selected CRF, final size, or measured encode FPS to evaluate a pre-analysis model is target leakage.

Information inventory

The following is a field inventory, not a commitment to a particular table layout.

Observation identity and provenance

  • local opaque History ID;
  • stable random research observation ID for idempotently merging repeated exports;
  • optional copied native run ID for local traceability, without an operational foreign key;
  • native versus translated origin;
  • History schema, collector, and measurement-contract versions;
  • optional event time; unknown translated time stays unknown;
  • analyzed, converted, not-worthwhile, remuxed, or other explicitly typed outcome;
  • clean/contended/incomplete measurement-quality classification where supportable.

No research identity should be derived from a path, filename, content hash, timestamp, or machine hardware.

Source and work-volume facts

  • duration and its provenance;
  • frame count when reliable;
  • rational/average frame rate;
  • variable-frame-rate indication;
  • coded width/height, display rotation, and actual encode width/height;
  • source codec/profile/level and container;
  • source pixel format, bit depth, chroma subsampling, interlace/field order;
  • source bytes and bitrate as descriptive decode/I/O facts, not assumed complexity measures;
  • color primaries, transfer, matrix/range, HDR metadata, and filter-path consequences;
  • audio/subtitle stream counts and copied/transcoded byte totals where pipeline time is modeled.

A useful base work-volume term is approximately frames × encoded pixels, but work volume is not content complexity.

Outcome and timing facts

  • requested and achieved quality target;
  • selected CRF/QP and preset;
  • measured VMAF/XPSNR/other score and metric/model version;
  • input/output total and per-stream bytes;
  • ab-av1 predicted size/duration, explicitly marked as prediction;
  • active operation wall time, excluding queue wait;
  • analysis/search, encode, remux, verification, and finalization totals where reliably scoped;
  • retry/fallback/cache facts and attributable overhead;
  • frames/media duration actually processed and measured throughput.

Every timing needs a documented unit, monotonic-clock requirement, scope, and prediction-versus-measurement status.

Environment facts

The goal is a reusable performance context, not a device inventory.

CPU/hardware candidates:

  • architecture;
  • x86 CPUID vendor, effective family/model/stepping, and CPU-provided brand;
  • physical/logical core topology, sockets, heterogeneous core classes, NUMA/cache domains where useful;
  • ISA capabilities available to the process and implementation/ISA actually selected when tools report it;
  • installed/OS-visible memory capacity;
  • optional speed/channel/cache information only if collection is reliable and evaluation shows value;
  • GPU/accelerator vendor/device/revision identifiers, model, driver/runtime, and actual execution role.

Platform/toolchain candidates:

  • OS family/version/build and kernel;
  • native architecture/target, VM/container status, libc/runtime where relevant;
  • application, ab-av1, FFmpeg/library, encoder, and metric revisions/build fingerprints;
  • effective cpuset/affinity/processor limits;
  • power/governor/QoS state only when trustworthy.

Do not collect or export hostnames, usernames, serial numbers, MAC addresses, account IDs, motherboard/RAM part numbers, device-instance IDs, bus addresses, device UUIDs, or raw command lines.

Execution facts

  • encoder implementation and exact effective preset/tune/rate-control settings;
  • requested/actual decoder and device;
  • filter and metric implementation/device;
  • input/output pixel format, bit depth, and chroma;
  • thread/logical-processor limits, tiles, affinity/pinning/socket/NUMA selection;
  • GOP/keyframe/scenecut/prediction structure;
  • film-grain/denoise, screen-content, super-resolution, lookahead, and other material encoder options;
  • normalized allowlisted filter operations;
  • CRF-search sample count/duration/position strategy, thoroughness, range/step/fallback/cache policy.

Prefer typed high-value fields plus a canonical allowlisted structured settings map and deterministic fingerprint. Do not store raw command lines because they can contain paths and do not provide a stable semantic representation. A fingerprint must account for tool/default-set revision.

Dynamic runtime facts

Potential best-effort quality controls:

  • processors actually available to the process;
  • configured and observed concurrent work;
  • system CPU load and available memory around the measured phase;
  • process user/system CPU time;
  • peak resident memory;
  • process I/O bytes;
  • swap/page/CPU/memory/I/O pressure;
  • battery/AC, power policy, or thermal-throttling indication when trustworthy.

These may explain contaminated wall times, but the expected prediction gain and collection reliability are unproven. They do not currently justify a standalone runtime_contexts table.

CRF-search/sample attempts

ab-av1 already performs representative sample encodes. Candidate attempt-level facts include:

  • target and attempted CRF;
  • sample index/count and full-pass flag;
  • sampling strategy/positions and actual frames/media duration processed;
  • sample encode wall time and FPS;
  • quality-metric wall time and score;
  • predicted full duration/size/percent;
  • decode mode, cache hit, fallback, and completion status.

Reducing the search to only its selected CRF and final prediction discards a direct content-plus-machine calibration signal. Because an observation has multiple attempts/samples, this is the clearest one-to-many relationship found so far.

Content complexity is required research, not an optional afterthought

A five-hour static black frame and a five-hour high-motion, noisy source can have identical duration, resolution, frame rate, codec, and nominal execution settings while imposing very different encoder work. Duration/resolution cohorts alone cannot represent that.

Candidate dimensions:

  • spatial complexity: texture, edges, detail, noise/grain;
  • temporal complexity: motion and frame-to-frame change;
  • structural variability: scene cuts, long static sections, fades, and changing complexity;
  • distribution shape: count, mean, standard deviation, and selected percentiles rather than only one average.

Every feature payload must identify:

  • extractor/algorithm and version;
  • sampling plan and positions;
  • decoded scale and pixel format;
  • frames/media duration examined;
  • tool revision;
  • typed payload/contract version.

Do not define an unversioned universal complexity_score. SI/TI, DCT-energy features, entropy, motion, scene-change rate, grain/noise, and representative encode speed are different measurements and must not be silently treated as interchangeable.

Three possible collection layers

  1. Metadata proxy available at Basic Scan
    • frames, pixels, frame rate, source codec/profile/pixel format/bit depth;
    • bitrate-per-pixel/frame may carry some descriptive signal, but should not be assumed to be encoding complexity.
  2. Reuse ab-av1 sample-encode evidence
    • strongest low-extra-cost combined content/machine signal after analysis;
    • preserve sample time/FPS and sampling contract rather than only ab-av1's final aggregate.
  3. Optional fast sampled pre-analysis scan
    • DCT-energy, motion, entropy, scene/static ratios, or similar descriptors;
    • potentially useful to Level 1, but it adds decode/analysis cost and must be benchmarked before becoming mandatory.

The important distinction is that layer 2 cannot improve a prediction shown before analysis, while layer 3 can only do so if its overhead is worthwhile.

Cross-platform identity and relevance

OS-reported model strings are display evidence, not portable join keys.

For x86/x86-64, direct CPUID can provide structured vendor/family/model/stepping, CPU brand, feature, cache, topology, and hypervisor information. For GPU devices actually used, portable PCI vendor/device/revision identifiers are preferable to installation-specific instance IDs or bus addresses.

Do not collapse observations merely because a fingerprint matches. Retain typed facts and derive relevance hierarchically:

  1. same installation/hardware + platform + compatible toolchain/execution;
  2. same hardware class + platform family + compatible toolchain/execution;
  3. same hardware class across another OS, with lower weight or an OS/toolchain adjustment;
  4. same microarchitecture/ISA/topology class;
  5. global fallback.

Thus a dual-boot Ryzen 5 5600X can contribute a shared hardware baseline while retaining distinct Windows and Linux observations. Different strings must not prevent cross-platform relevance, and equal CPU labels must not imply equal throughput.

Prior art and Rust collection options

These examples support separating reusable machine/environment description from individual measurements, but they do not dictate our SQL normalization:

Rust candidates to spike, not adopted dependencies:

Platform references:

Content/encoding research basis

Licensing matters: VCA is GPL-3.0, so it is research inspiration unless compatibility is deliberately reviewed.

Candidate physical mappings

None is selected.

Candidate A: minimal two-table mapping

  1. environment_profiles
  2. history_observations

Attempts, phases, settings, runtime, and content summaries are versioned structured payloads on the observation. This minimizes joins and migrations, but makes repeated measurements less relational/queryable and risks turning the observation into an opaque bag.

Candidate B: three-table core

  1. environment_profiles
  2. history_observations
  3. encode_attempts

This gives the strongest discovered one-to-many data its own table while keeping execution, runtime, content summaries, and fixed phase totals on the observation.

Candidate C: four-table measurement model

  1. environment_profiles
  2. history_observations
  3. encode_attempts
  4. phase_measurements

This is attractive if arbitrary/repeated analysis/encode/remux/verification spans need independent querying. It is unnecessary if a small fixed set of timing columns answers the real product/model questions.

Candidate D: normalized reusable execution profile

Add execution_profiles to B or C if the same canonical effective settings recur enough to justify deduplication and referential integrity. A fingerprint alone may be sufficient initially; normalization should not be justified by predicted accuracy because table count does not improve the model.

Separate hardware, platform, and toolchain logical fingerprints can still exist inside one environment row. They need independent tables only if reuse/cardinality/update behavior demonstrates a concrete benefit.

Canonical export direction

Internal tables do not define the public bundle. A possible versioned, streaming shape is:

manifest.json
observations.ndjson
environment_profiles.ndjson
encode_attempts.ndjson          # if represented separately
phase_measurements.ndjson       # if represented separately
local_sources.ndjson            # optional, deliberately included

The manifest should record schema/collector versions, units, enumerations, row counts/hashes, feature stages, and export options. NDJSON is being considered as an interchange format, not selected as the application's transactional store.

The pathless statistical core retains hardware, platform, toolchain, execution, and content facts because those are prediction evidence. It excludes paths/names, host/user identifiers, serials, instance IDs, raw command lines, logs, and direct content identity.

Detailed technical profiles can be distinctive. The UI should describe a shared bundle as pathless/pseudonymous, not promise anonymity, and preview the exact profile fields included. Stable random contributor/system tokens, if adopted, make repeated contributions linkable and therefore require explicit disclosure/reset behavior.

Questions still open

  • Is toolchain part of one immutable environment_profile, or should observations reference hardware/platform and toolchain independently?
  • Does an execution profile recur enough to deserve a table, or are typed settings plus a fingerprint simpler?
  • Which phase timings materially improve user-visible estimates?
  • Which dynamic runtime probes are reliable enough on Windows and Linux to justify collection?
  • Can ab-av1 expose stable machine-readable per-sample timing/position data, or would this depend on unstable text parsing/upstream changes?
  • Which content descriptors improve held-out error beyond metadata and ab-av1 sample speed?
  • Can a cheap sampled pre-analysis scan improve Level 1 enough to repay its latency?
  • Should content summaries be selected typed columns, one versioned JSON payload, or both?
  • What privacy language and reset semantics are appropriate for pooled machine-aware datasets?
  • Which physical mapping produces the smallest long-term migration/query burden under the selected V3 architecture research: choose the durable state model and storage engine #89 engine?

Required spike before a decision

  1. Capture one CRF search and one full encode into candidate B and candidate C.
  2. Mark each field by availability stage and collection source.
  3. Verify what ab-av1 exposes directly versus stable JSON, unstable text, wrapper measurement, or unavailable data.
  4. Collect the same hardware class on Windows and Linux and compare normalized identifiers/nullable gaps.
  5. Benchmark sysinfo plus targeted CPUID/platform probes rather than collecting every available system fact.
  6. Preserve per-sample encode timing and evaluate its predictive value separately from source metadata.
  7. Benchmark candidate sampled complexity extractors on static, low-motion, high-motion, noisy/grainy, 1080p, and 4K sources.
  8. Compare held-out error for:
    • current duration/codec/resolution baseline;
    • plus environment/execution;
    • plus sample-encode evidence;
    • plus pre-analysis content descriptors.
  9. Split evaluation by contributor/system group and tool version; do not randomly leak the same machine across train/test.
  10. Export a synthetic fixture containing prohibited identity strings and prove the pathless bundle contains none.
  11. Compare actual schema/query/migration code for candidates A–D under V3 architecture research: choose the durable state model and storage engine #89 rather than choosing by table-count intuition.

Research completion criteria

Closing this issue means the observation/collection/export contract has been selected and documented. It does not mean the estimator, pooled research system, optional content extractor, or storage implementation is complete.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestresearchInvestigation/decision needed before implementation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions