Skip to content

Zero-vote re-embed: prefer a non-incumbent speaker on weak margins - #948

Open
robertozoia wants to merge 1 commit into
FluidInference:mainfrom
robertozoia:verbatim/zero-vote-incumbents
Open

robertozoia wants to merge 1 commit into
FluidInference:mainfrom
robertozoia:verbatim/zero-vote-incumbents

Conversation

@robertozoia

Copy link
Copy Markdown

Problem

A zero-vote run (speech whose local slot got no embedding in any window) is re-embedded and assigned to the closest centroid "regardless of margin". For a ~1 s interjection the span embedding is often near-uninformative — e.g. cosines [0.066, −0.064] on a real two-person interview — so the run is handed back to the surrounding speaker, which is exactly the outcome the pass exists to avoid ("silently absorbing whole speaker turns into the surrounding speaker's segment").

Observation

By construction a zero-vote run is speech the segmentation model put in a different local slot from the neighbouring speech that did get votes in the same windows. So the clusters active just before/after the run are unlikely owners, and that structural evidence is stronger than a coin-flip cosine.

Change

ZeroVoteReembedder.incumbentClusters(around:perFrameClusters:frameDuration:lookaroundSeconds:) collects the clusters selected within 1.5 s on either side of the run; assignment(embedding:centroids:excluding:incumbentMargin:) only lets an incumbent win when its cosine beats the best non-incumbent by ≥ 0.25. With every cluster incumbent (run between two different speakers) or a strong match, behaviour is unchanged. The log line now includes incumbents [...].

Evidence

Real audio, ~80 min (four host+guest interviews, one four-person panel):

  • Holly Tyson / CHRO Conversations, 52.14–53.17 s "I'm thrilled to be here" (guest, between two host turns): cosines [0.066, −0.064] → was S1 (host), now S2 (guest). Correct.
  • Three other zero-vote runs (0.5–1.2 s): unchanged (two had a strong 0.66/0.70 match to a non-incumbent; one at end-of-file already non-incumbent).
  • Speaker counts unchanged on all five recordings.

Happy to add a unit test for assignment(excluding:) if you'd like it in.

A zero-vote run is, by construction, speech the segmentation model put
in a different local slot from the neighbouring speech that did get
votes in the same windows. Re-embedding a ~1 s span is often
near-uninformative (cosines 0.07 vs -0.06), and closest-centroid then
hands the run back to the surrounding speaker — e.g. a guest's
"I'm thrilled to be here" between two host turns.

Assignment now excludes the clusters active within 1.5 s on either side
of the run unless one of them beats the best other cluster by a cosine
margin of 0.25. Strong matches (e.g. 0.66 vs 0.12) are unaffected.
@Alex-Wengg

Copy link
Copy Markdown
Member
  1. The existing doc comment is now misplaced. The new functions were inserted between the old assignment(embedding:centroids:) doc comment and its function. That comment's - Returns: block now sits on top of incumbentClusters, and the old assignment has no doc.
  2. Some docs are now out of date. The type-level doc in ZeroVoteReembedder.swift ("assignment ignores any margin") and the doc on applyZeroVoteReembed ("regardless of margin") no longer describe the behaviour.
  3. Incumbents can come from placeholder labels:
    • Zero-vote frames shorter than minDurationSeconds keep the arbitrary tie-break to cluster 0, so S1 can be counted as an incumbent without any real evidence.
    • Zero-vote overlap frames can do the same, since they keep the tie-break too.
    • Runs are rewritten in place in order, so an earlier re-embedded run's own guess can become an incumbent for the next nearby run.
  4. Hardcoded constants. 1.5 and 0.25 should be fields on config.zeroVoteReembed next to minDurationSeconds, not literals at the call site.
  5. No unit tests. The repo requires them for new code. The author offered, and ZeroVoteReembedderTests.swift already exists, so tests for incumbentClusters and assignment(excluding:) fit there easily.
  6. The evidence is thin. It's one changed assignment across about 80 minutes of private audio. Since the pass is opt-in this isn't blocking, but an AMI DER run with the pass enabled, before and after, would make the 0.25 margin more than a hand-picked value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants