We shouldn't require track transferability

[guidou]

The current version of the API requires track transferability, but this shouldn't be necessary.
Currently, tracks are useless on workers except for this API, so we shouldn't add that as a requirement.

A way to keep the API worker first which has several benefits is to follow the postMessage-like approach of webrtc-encoded-transform.

Something (subject to discussion) like:

For MediaStreamTrackProcessor:

// main
navigator.mediaDevices.createTrackProcessor(myWorker, mytrack, myOptions, [myOptions]);`

// worker
ontrackprocessor = event => {
    let processor = event.processor;
   // Process frames using processor. `event.options` has the data sent via myOptions for extra configuration. 
}

For VideoTrackGenerator:

// main
let generatedTrack = navigator.mediaDevices.createVideoTrackGenerator(myWorker, myOptions, [myOptions]);`

// worker
ontvideorackgenerator = event => {
    let generator = event.generator;
   // generate frames for `generatedTrack`. `event.options` has the data sent via myOptions for extra configuration. 
}

[guidou]

cc @jan-ivar @youennf

[jan-ivar]

What problem is this solving? Transferring the track has other benefits like being able to apply constraints and read track stats and settings.

[guidou]

It solves the problem that you don't need track transferability to implement this API, which we consider a blocker, at least for the medium term.
Also, the benefits of transferring a track to a worker are very limited. In fact, I would argue that the only benefit would be using this API.
You can call applyConstraints and read stats/settings on Window, where the main application is.

This proposal uses a pattern that we are already using in encoded transform and should easily allow us to have interoperable implementations.

[youennf]

One use case for transferring media stream track is to create a track (via VideoTrackGenerator) and send it to sinks like RTCRtpSender or MediaRecorder.

[guidou]

The proposal is that createVideoTrackGenerator() is called from Window and returns a promise with a MediaStreamTrack on Window, where the RTCRtpSender or MediaRecorder are. The generator (which no longer has a track field) is created in the worker (the application gets it via an event, just like an RTCRtpScriptTransformer).

This removes the need to transfer the track. You only needed to transfer the track from the worker to window because the current spec creates the track on the worker, where it is largely useless, as all the track APIs are on Window.

[guidou]

Another benefit of this API surface is that it allows feature detection on main without creating a worker.

[jan-ivar]

This can be feature detected on main like this:

function isMstTransferable() {
  try {
    const [track] = document.createElement('canvas').captureStream().getVideoTracks();
    new MessageChannel().port1.postMessage(track, [track]);
    return true;
  } catch (e) {
    if (e.name != "DataCloneError") throw e;
    return false;
  }
}

[jan-ivar]

It solves the problem that you don't need track transferability to implement this API, which we consider a blocker, at least for the medium term.

@guidou why is transfer a blocker? What do you mean by medium term? Safari has already shipped this and it works.

If you explain the problem, perhaps their engineers can help?

[guidou]

It's a blocker for Chromium to ship it in the short term since Chromium doesn't implement track transferability and will not have it for quite some time.

I don't expect Chromium to have track transferability in the short term, so I guess we won't have an interoperable API for a long time.

[guidou]

This can be feature detected on main like this:

function isMstTransferable() {
  try {
    const [track] = document.createElement('canvas').captureStream().getVideoTracks();
    new MessageChannel().port1.postMessage(track, [track]);
    return true;
  } catch (e) {
    if (e.name != "DataCloneError") throw e;
    return false;
  }
}

This feature-detects track transferability, not mediacapture-transform.
With the current API you need to create a worker to feature-detect, which is costly and unergonomic.

[jan-ivar]

I doubt that's needed. As you said, "tracks are useless on workers except for this API". If MST transfer is detected, it seems reasonable to assume some purpose awaits these tracks in the worker.

This works in the only current implementation: "WebKit for Safari 18 beta adds support for MediaStreamTrack processing in a dedicated worker."

This seems like a property worth emulating. I've added a note to Firefox's implementation bug to do the same. Thanks for bringing attention to this!

[jan-ivar]

... Chromium doesn't implement track transferability and will not have it for quite some time.

If there's some difficulty or problem with the spec's transfer steps as specified, please bring it to our attention so we can address it.

[jan-ivar]

You can call applyConstraints and read stats/settings on Window, where the main application is.

Yes, but waiting on postMessage for these measurements hardly seems ideal. In the current spec, the worker transform can inspect real-time track stats counters like deliveredFrames, discardedFrames and totalFrames synchronously, and correlate them with the VideoFrame it is currently processing.

[guidou]

You can call applyConstraints and read stats/settings on Window, where the main application is.

Yes, but waiting on postMessage for these measurements hardly seems ideal.

That goes the other way too if you want access to the track on Window (which is the more common case today).

In the current spec, the worker transform can inspect real-time track stats counters like deliveredFrames, discardedFrames and totalFrames synchronously, and correlate them with the VideoFrame it is currently processing.

I'm not opposed to supporting transferability. I'm opposed to making it a requirement to use mediacapture-transform, as that will have the practical consequence of delaying interoperable implementations.

We already have a pattern for adding worker support without requiring transferability of tracks or streams. This doesn't mean applications are forbidden from transferring tracks on browsers that support it if they want to.
It just means that applications that don't need to transfer tracks to do processing (which are most if not all applications today) can more quickly have an interoperable API in practice.

[jan-ivar]

That goes the other way too if you want access to the track on Window ...

No, because tracks can be cloned. With transfer, stats are readily available in both places. So the problem of a transformer needing a roundtrip to main to read settings and applyConstraints, for lack of transfer, would be new with this proposal.