You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
online collection, upload, synchronization, or backup.
Earlier drafts use seven logical headings to inventory possible facts. Those headings are not seven accepted tables. A later two-table sketch was also only a minimal physical mapping, not a settled reduction. The present working range is approximately three clearly motivated relational shapes, with a possible fourth child/profile table if a spike demonstrates value. This issue intentionally keeps the alternatives open.
Corrections to earlier discussion
One canonical export, not two reports
There should be one versioned canonical History bundle. The statistical observations and technical configuration are the same whether the bundle is kept personally or shared for research.
Readable local source names/paths, if the user explicitly includes them, belong in an optional attachment such as local_sources.ndjson. Omitting that attachment produces the pathless research bundle. We should not maintain separate personal and research statistical schemas.
Logical categories are not automatically tables
We need to distinguish:
information worth recording;
logical concepts useful in Rust/domain code or documentation;
physical SQL tables.
A table is justified by independent identity, reuse, one-to-many cardinality, integrity constraints, or demonstrated query value—not by the existence of a conceptual noun.
In particular:
runtime_context_id is not currently justified; runtime facts describe one observation and can be inline unless repeated sampling requires child rows;
content-complexity summaries describe one source observation and can be inline or a versioned payload;
execution settings may remain typed/structured on the observation or attempt with a deterministic fingerprint;
CRF-search/sample attempts are genuinely one-to-many and therefore have a stronger claim to a child table;
phase measurements may justify a child table if arbitrary/repeated spans must be queried, but fixed totals could be columns on the observation.
Established domain requirements
These requirements are independent of the final physical mapping:
History is first-class statistical evidence, not current operational file state.
A historical observation survives source deletion, movement, replacement, verdict clearing, and queue cleanup.
History has no lifecycle or foreign-key dependency on FileRecord, StandingVerdict, output, queue item, or current path.
Failed/stopped runs remain operational evidence and are not automatically clean estimator observations.
Native and translated observations enter the same current V3 model. Only the disposable Python translator understands V2.
Unknown values remain null/absent, never zero.
Privacy scrub removes the optional readable historical path while preserving every technical/statistical fact. It does not promise physical byte erasure.
History remains queryable/paginated in the Rust backend rather than mirrored wholesale into frontend state.
Exact measurements, predictions, and derived aggregates must remain distinguishable.
Collection failure must not abort a conversion; optional probes are best-effort and carry collection provenance.
Prediction stages and leakage boundary
Every field needs an availability stage:
Basic Scan / before CRF search
source metadata;
planned execution settings;
machine/platform/toolchain facts;
any genuinely cheap pre-analysis content features.
After CRF search / before full encode
sample positions and sample encode timings;
attempted CRFs and measured quality scores;
selected CRF;
ab-av1 predicted size/duration.
After terminal completion
actual phase and total times;
actual output sizes;
observed throughput;
retry/fallback outcome and runtime diagnostics.
#57's Level 1 estimator may use only stage-1 facts. Sample performance can improve the post-analysis estimate, but using selected CRF, final size, or measured encode FPS to evaluate a pre-analysis model is target leakage.
Information inventory
The following is a field inventory, not a commitment to a particular table layout.
Observation identity and provenance
local opaque History ID;
stable random research observation ID for idempotently merging repeated exports;
optional copied native run ID for local traceability, without an operational foreign key;
native versus translated origin;
History schema, collector, and measurement-contract versions;
optional event time; unknown translated time stays unknown;
analyzed, converted, not-worthwhile, remuxed, or other explicitly typed outcome;
clean/contended/incomplete measurement-quality classification where supportable.
No research identity should be derived from a path, filename, content hash, timestamp, or machine hardware.
Source and work-volume facts
duration and its provenance;
frame count when reliable;
rational/average frame rate;
variable-frame-rate indication;
coded width/height, display rotation, and actual encode width/height;
source codec/profile/level and container;
source pixel format, bit depth, chroma subsampling, interlace/field order;
source bytes and bitrate as descriptive decode/I/O facts, not assumed complexity measures;
color primaries, transfer, matrix/range, HDR metadata, and filter-path consequences;
audio/subtitle stream counts and copied/transcoded byte totals where pipeline time is modeled.
A useful base work-volume term is approximately frames × encoded pixels, but work volume is not content complexity.
Outcome and timing facts
requested and achieved quality target;
selected CRF/QP and preset;
measured VMAF/XPSNR/other score and metric/model version;
input/output total and per-stream bytes;
ab-av1 predicted size/duration, explicitly marked as prediction;
active operation wall time, excluding queue wait;
analysis/search, encode, remux, verification, and finalization totals where reliably scoped;
retry/fallback/cache facts and attributable overhead;
frames/media duration actually processed and measured throughput.
Every timing needs a documented unit, monotonic-clock requirement, scope, and prediction-versus-measurement status.
Environment facts
The goal is a reusable performance context, not a device inventory.
CPU/hardware candidates:
architecture;
x86 CPUID vendor, effective family/model/stepping, and CPU-provided brand;
ISA capabilities available to the process and implementation/ISA actually selected when tools report it;
installed/OS-visible memory capacity;
optional speed/channel/cache information only if collection is reliable and evaluation shows value;
GPU/accelerator vendor/device/revision identifiers, model, driver/runtime, and actual execution role.
Platform/toolchain candidates:
OS family/version/build and kernel;
native architecture/target, VM/container status, libc/runtime where relevant;
application, ab-av1, FFmpeg/library, encoder, and metric revisions/build fingerprints;
effective cpuset/affinity/processor limits;
power/governor/QoS state only when trustworthy.
Do not collect or export hostnames, usernames, serial numbers, MAC addresses, account IDs, motherboard/RAM part numbers, device-instance IDs, bus addresses, device UUIDs, or raw command lines.
Execution facts
encoder implementation and exact effective preset/tune/rate-control settings;
Prefer typed high-value fields plus a canonical allowlisted structured settings map and deterministic fingerprint. Do not store raw command lines because they can contain paths and do not provide a stable semantic representation. A fingerprint must account for tool/default-set revision.
Dynamic runtime facts
Potential best-effort quality controls:
processors actually available to the process;
configured and observed concurrent work;
system CPU load and available memory around the measured phase;
process user/system CPU time;
peak resident memory;
process I/O bytes;
swap/page/CPU/memory/I/O pressure;
battery/AC, power policy, or thermal-throttling indication when trustworthy.
These may explain contaminated wall times, but the expected prediction gain and collection reliability are unproven. They do not currently justify a standalone runtime_contexts table.
sampling strategy/positions and actual frames/media duration processed;
sample encode wall time and FPS;
quality-metric wall time and score;
predicted full duration/size/percent;
decode mode, cache hit, fallback, and completion status.
Reducing the search to only its selected CRF and final prediction discards a direct content-plus-machine calibration signal. Because an observation has multiple attempts/samples, this is the clearest one-to-many relationship found so far.
Content complexity is required research, not an optional afterthought
A five-hour static black frame and a five-hour high-motion, noisy source can have identical duration, resolution, frame rate, codec, and nominal execution settings while imposing very different encoder work. Duration/resolution cohorts alone cannot represent that.
temporal complexity: motion and frame-to-frame change;
structural variability: scene cuts, long static sections, fades, and changing complexity;
distribution shape: count, mean, standard deviation, and selected percentiles rather than only one average.
Every feature payload must identify:
extractor/algorithm and version;
sampling plan and positions;
decoded scale and pixel format;
frames/media duration examined;
tool revision;
typed payload/contract version.
Do not define an unversioned universal complexity_score. SI/TI, DCT-energy features, entropy, motion, scene-change rate, grain/noise, and representative encode speed are different measurements and must not be silently treated as interchangeable.
bitrate-per-pixel/frame may carry some descriptive signal, but should not be assumed to be encoding complexity.
Reuse ab-av1 sample-encode evidence
strongest low-extra-cost combined content/machine signal after analysis;
preserve sample time/FPS and sampling contract rather than only ab-av1's final aggregate.
Optional fast sampled pre-analysis scan
DCT-energy, motion, entropy, scene/static ratios, or similar descriptors;
potentially useful to Level 1, but it adds decode/analysis cost and must be benchmarked before becoming mandatory.
The important distinction is that layer 2 cannot improve a prediction shown before analysis, while layer 3 can only do so if its overhead is worthwhile.
Cross-platform identity and relevance
OS-reported model strings are display evidence, not portable join keys.
For x86/x86-64, direct CPUID can provide structured vendor/family/model/stepping, CPU brand, feature, cache, topology, and hypervisor information. For GPU devices actually used, portable PCI vendor/device/revision identifiers are preferable to installation-specific instance IDs or bus addresses.
Do not collapse observations merely because a fingerprint matches. Retain typed facts and derive relevance hierarchically:
same installation/hardware + platform + compatible toolchain/execution;
same hardware class + platform family + compatible toolchain/execution;
same hardware class across another OS, with lower weight or an OS/toolchain adjustment;
same microarchitecture/ISA/topology class;
global fallback.
Thus a dual-boot Ryzen 5 5600X can contribute a shared hardware baseline while retaining distinct Windows and Linux observations. Different strings must not prevent cross-platform relevance, and equal CPU labels must not imply equal throughput.
Prior art and Rust collection options
These examples support separating reusable machine/environment description from individual measurements, but they do not dictate our SQL normalization:
Rust candidates to spike, not adopted dependencies:
sysinfo provides cross-platform system, CPU, memory, process, and disk information. Its own guidance recommends refreshing only needed data, and unsupported/virtual environments can return empty values: https://docs.rs/sysinfo/latest/sysinfo/
raw-cpuid parses x86 CPUID directly without external dependencies and exposes vendor, features, caches, topology, brand, and hypervisor information: https://docs.rs/raw-cpuid/latest/raw_cpuid/
hwlocality exposes rich hwloc topology/affinity/NUMA/cache information, but the Rust binding is currently 1.0.0-alpha, links a C library, and a vendored build adds native build-tool requirements. It is useful prior art but may be excessive for the first collector: https://docs.rs/hwlocality/latest/hwlocality/ https://hwloc.readthedocs.io/
An SVT-AV1 encoding-time model uses video parameters, encoder configuration, and spatial/temporal content descriptors, reporting 19.6% encoding-time prediction error in its evaluated setting: https://arxiv.org/abs/2401.16067
Content-adaptive preset work uses DCT-energy spatial/temporal features together with allocated CPU threads and target speed: https://arxiv.org/abs/2210.10330
Licensing matters: VCA is GPL-3.0, so it is research inspiration unless compatibility is deliberately reviewed.
Candidate physical mappings
None is selected.
Candidate A: minimal two-table mapping
environment_profiles
history_observations
Attempts, phases, settings, runtime, and content summaries are versioned structured payloads on the observation. This minimizes joins and migrations, but makes repeated measurements less relational/queryable and risks turning the observation into an opaque bag.
Candidate B: three-table core
environment_profiles
history_observations
encode_attempts
This gives the strongest discovered one-to-many data its own table while keeping execution, runtime, content summaries, and fixed phase totals on the observation.
Candidate C: four-table measurement model
environment_profiles
history_observations
encode_attempts
phase_measurements
This is attractive if arbitrary/repeated analysis/encode/remux/verification spans need independent querying. It is unnecessary if a small fixed set of timing columns answers the real product/model questions.
Add execution_profiles to B or C if the same canonical effective settings recur enough to justify deduplication and referential integrity. A fingerprint alone may be sufficient initially; normalization should not be justified by predicted accuracy because table count does not improve the model.
Separate hardware, platform, and toolchain logical fingerprints can still exist inside one environment row. They need independent tables only if reuse/cardinality/update behavior demonstrates a concrete benefit.
Canonical export direction
Internal tables do not define the public bundle. A possible versioned, streaming shape is:
manifest.json
observations.ndjson
environment_profiles.ndjson
encode_attempts.ndjson # if represented separately
phase_measurements.ndjson # if represented separately
local_sources.ndjson # optional, deliberately included
The manifest should record schema/collector versions, units, enumerations, row counts/hashes, feature stages, and export options. NDJSON is being considered as an interchange format, not selected as the application's transactional store.
The pathless statistical core retains hardware, platform, toolchain, execution, and content facts because those are prediction evidence. It excludes paths/names, host/user identifiers, serials, instance IDs, raw command lines, logs, and direct content identity.
Detailed technical profiles can be distinctive. The UI should describe a shared bundle as pathless/pseudonymous, not promise anonymity, and preview the exact profile fields included. Stable random contributor/system tokens, if adopted, make repeated contributions linkable and therefore require explicit disclosure/reset behavior.
Questions still open
Is toolchain part of one immutable environment_profile, or should observations reference hardware/platform and toolchain independently?
Does an execution profile recur enough to deserve a table, or are typed settings plus a fingerprint simpler?
Which phase timings materially improve user-visible estimates?
Which dynamic runtime probes are reliable enough on Windows and Linux to justify collection?
Can ab-av1 expose stable machine-readable per-sample timing/position data, or would this depend on unstable text parsing/upstream changes?
Which content descriptors improve held-out error beyond metadata and ab-av1 sample speed?
Can a cheap sampled pre-analysis scan improve Level 1 enough to repay its latency?
Should content summaries be selected typed columns, one versioned JSON payload, or both?
What privacy language and reset semantics are appropriate for pooled machine-aware datasets?
Closing this issue means the observation/collection/export contract has been selected and documented. It does not mean the estimator, pooled research system, optional content extractor, or storage implementation is complete.
Parent: #33
Related: #79, #89, #57, #77, #51
Status: open research, no schema decision
This issue records what V3 History may need to retain for:
It does not select:
Earlier drafts use seven logical headings to inventory possible facts. Those headings are not seven accepted tables. A later two-table sketch was also only a minimal physical mapping, not a settled reduction. The present working range is approximately three clearly motivated relational shapes, with a possible fourth child/profile table if a spike demonstrates value. This issue intentionally keeps the alternatives open.
Corrections to earlier discussion
One canonical export, not two reports
There should be one versioned canonical History bundle. The statistical observations and technical configuration are the same whether the bundle is kept personally or shared for research.
Readable local source names/paths, if the user explicitly includes them, belong in an optional attachment such as
local_sources.ndjson. Omitting that attachment produces the pathless research bundle. We should not maintain separate personal and research statistical schemas.Logical categories are not automatically tables
We need to distinguish:
A table is justified by independent identity, reuse, one-to-many cardinality, integrity constraints, or demonstrated query value—not by the existence of a conceptual noun.
In particular:
runtime_context_idis not currently justified; runtime facts describe one observation and can be inline unless repeated sampling requires child rows;Established domain requirements
These requirements are independent of the final physical mapping:
FileRecord,StandingVerdict, output, queue item, or current path.Prediction stages and leakage boundary
Every field needs an availability stage:
#57's Level 1 estimator may use only stage-1 facts. Sample performance can improve the post-analysis estimate, but using selected CRF, final size, or measured encode FPS to evaluate a pre-analysis model is target leakage.
Information inventory
The following is a field inventory, not a commitment to a particular table layout.
Observation identity and provenance
No research identity should be derived from a path, filename, content hash, timestamp, or machine hardware.
Source and work-volume facts
A useful base work-volume term is approximately frames × encoded pixels, but work volume is not content complexity.
Outcome and timing facts
Every timing needs a documented unit, monotonic-clock requirement, scope, and prediction-versus-measurement status.
Environment facts
The goal is a reusable performance context, not a device inventory.
CPU/hardware candidates:
Platform/toolchain candidates:
Do not collect or export hostnames, usernames, serial numbers, MAC addresses, account IDs, motherboard/RAM part numbers, device-instance IDs, bus addresses, device UUIDs, or raw command lines.
Execution facts
Prefer typed high-value fields plus a canonical allowlisted structured settings map and deterministic fingerprint. Do not store raw command lines because they can contain paths and do not provide a stable semantic representation. A fingerprint must account for tool/default-set revision.
Dynamic runtime facts
Potential best-effort quality controls:
These may explain contaminated wall times, but the expected prediction gain and collection reliability are unproven. They do not currently justify a standalone
runtime_contextstable.CRF-search/sample attempts
ab-av1 already performs representative sample encodes. Candidate attempt-level facts include:
Reducing the search to only its selected CRF and final prediction discards a direct content-plus-machine calibration signal. Because an observation has multiple attempts/samples, this is the clearest one-to-many relationship found so far.
Content complexity is required research, not an optional afterthought
A five-hour static black frame and a five-hour high-motion, noisy source can have identical duration, resolution, frame rate, codec, and nominal execution settings while imposing very different encoder work. Duration/resolution cohorts alone cannot represent that.
Candidate dimensions:
Every feature payload must identify:
Do not define an unversioned universal
complexity_score. SI/TI, DCT-energy features, entropy, motion, scene-change rate, grain/noise, and representative encode speed are different measurements and must not be silently treated as interchangeable.Three possible collection layers
The important distinction is that layer 2 cannot improve a prediction shown before analysis, while layer 3 can only do so if its overhead is worthwhile.
Cross-platform identity and relevance
OS-reported model strings are display evidence, not portable join keys.
For x86/x86-64, direct CPUID can provide structured vendor/family/model/stepping, CPU brand, feature, cache, topology, and hypervisor information. For GPU devices actually used, portable PCI vendor/device/revision identifiers are preferable to installation-specific instance IDs or bus addresses.
Do not collapse observations merely because a fingerprint matches. Retain typed facts and derive relevance hierarchically:
Thus a dual-boot Ryzen 5 5600X can contribute a shared hardware baseline while retaining distinct Windows and Linux observations. Different strings must not prevent cross-platform relevance, and equal CPU labels must not imply equal throughput.
Prior art and Rust collection options
These examples support separating reusable machine/environment description from individual measurements, but they do not dictate our SQL normalization:
https://opentelemetry.io/docs/specs/otel/resource/data-model/
https://opentelemetry.io/docs/specs/semconv/registry/attributes/host/
https://opentelemetry.io/docs/specs/semconv/system/process-metrics/
https://asv.readthedocs.io/en/stable/using.html#machine-information
https://github.com/phoronix-test-suite/phoronix-test-suite
Rust candidates to spike, not adopted dependencies:
sysinfoprovides cross-platform system, CPU, memory, process, and disk information. Its own guidance recommends refreshing only needed data, and unsupported/virtual environments can return empty values:https://docs.rs/sysinfo/latest/sysinfo/
raw-cpuidparses x86 CPUID directly without external dependencies and exposes vendor, features, caches, topology, brand, and hypervisor information:https://docs.rs/raw-cpuid/latest/raw_cpuid/
os_infoprovides typed OS type/version/edition/bitness:https://docs.rs/os_info/latest/os_info/
hwlocalityexposes rich hwloc topology/affinity/NUMA/cache information, but the Rust binding is currently1.0.0-alpha, links a C library, and a vendored build adds native build-tool requirements. It is useful prior art but may be excessive for the first collector:https://docs.rs/hwlocality/latest/hwlocality/
https://hwloc.readthedocs.io/
Platform references:
https://learn.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-system_cpu_set_information
https://www.kernel.org/doc/html/latest/admin-guide/cputopology.html
https://docs.kernel.org/arch/x86/cpuinfo.html
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/identifiers-for-pci-devices
https://docs.kernel.org/PCI/sysfs-pci.html
Content/encoding research basis
https://arxiv.org/abs/2401.16067
https://arxiv.org/abs/2210.10330
https://arxiv.org/abs/2304.12384
https://github.com/cd-athena/VCA
https://link.springer.com/article/10.1007/s10586-020-03207-x
https://ffmpeg.org/ffmpeg-filters.html
https://ffmpeg.org/pipermail/ffmpeg-cvslog/2023-March/136351.html
https://docs.rs/oximedia-analysis/latest/oximedia_analysis/complexity_metrics/index.html
Licensing matters: VCA is GPL-3.0, so it is research inspiration unless compatibility is deliberately reviewed.
Candidate physical mappings
None is selected.
Candidate A: minimal two-table mapping
environment_profileshistory_observationsAttempts, phases, settings, runtime, and content summaries are versioned structured payloads on the observation. This minimizes joins and migrations, but makes repeated measurements less relational/queryable and risks turning the observation into an opaque bag.
Candidate B: three-table core
environment_profileshistory_observationsencode_attemptsThis gives the strongest discovered one-to-many data its own table while keeping execution, runtime, content summaries, and fixed phase totals on the observation.
Candidate C: four-table measurement model
environment_profileshistory_observationsencode_attemptsphase_measurementsThis is attractive if arbitrary/repeated analysis/encode/remux/verification spans need independent querying. It is unnecessary if a small fixed set of timing columns answers the real product/model questions.
Candidate D: normalized reusable execution profile
Add
execution_profilesto B or C if the same canonical effective settings recur enough to justify deduplication and referential integrity. A fingerprint alone may be sufficient initially; normalization should not be justified by predicted accuracy because table count does not improve the model.Separate hardware, platform, and toolchain logical fingerprints can still exist inside one environment row. They need independent tables only if reuse/cardinality/update behavior demonstrates a concrete benefit.
Canonical export direction
Internal tables do not define the public bundle. A possible versioned, streaming shape is:
The manifest should record schema/collector versions, units, enumerations, row counts/hashes, feature stages, and export options. NDJSON is being considered as an interchange format, not selected as the application's transactional store.
The pathless statistical core retains hardware, platform, toolchain, execution, and content facts because those are prediction evidence. It excludes paths/names, host/user identifiers, serials, instance IDs, raw command lines, logs, and direct content identity.
Detailed technical profiles can be distinctive. The UI should describe a shared bundle as pathless/pseudonymous, not promise anonymity, and preview the exact profile fields included. Stable random contributor/system tokens, if adopted, make repeated contributions linkable and therefore require explicit disclosure/reset behavior.
Questions still open
environment_profile, or should observations reference hardware/platform and toolchain independently?Required spike before a decision
sysinfoplus targeted CPUID/platform probes rather than collecting every available system fact.Research completion criteria
Closing this issue means the observation/collection/export contract has been selected and documented. It does not mean the estimator, pooled research system, optional content extractor, or storage implementation is complete.