Skip to content

Use the offline mask blur ratio in the realtime blending path - #422

Open
cuhawk wants to merge 1 commit into
TMElyralab:mainfrom
cuhawk:realtime-blend-blur-consistency
Open

Use the offline mask blur ratio in the realtime blending path#422
cuhawk wants to merge 1 commit into
TMElyralab:mainfrom
cuhawk:realtime-blend-blur-consistency

Conversation

@cuhawk

@cuhawk cuhawk commented Aug 31, 2026

Copy link
Copy Markdown

Problem

get_image() (offline) and get_image_prepare_material() (realtime) build the blending mask the same way: face_seg over the expanded crop, crop to the face box, paste onto a black canvas, drop everything above upper_boundary_ratio, then Gaussian blur to soften the edge.

The blur is the only step where the two disagree. get_image() sizes the kernel at 0.05 of the crop width (blending.py:81) and get_image_prepare_material() at 0.1 (blending.py:134), so the realtime mask feathers over roughly twice the distance. Where the mask edge runs close to the mouth, the wider ramp mixes more of the original frame back over the generated pixels than the offline path would from the same inputs.

What the file history shows, because it cuts both ways

  • Before v1.5 both functions used 0.1 and agreed.
  • v1.5 (db20431) rewrote get_image only: expand 1.2 → 1.5, new mode and fp arguments, blur ratio 0.1 → 0.05. It left get_image_prepare_material on the older code. This commit created the divergence.
  • 39ccf69 ("feat: real-time infer", feat: real-time infer #286) then ported that rewrite into get_image_prepare_material: expand 1.2 → 1.5, added fp and mode, passed both through to face_seg. It edited the function line by line and left the 0.1 in place.

So this is not a value that only one commit overlooked; a later commit went through the same function and kept it. Nothing in the code or the commit messages argues for it either way. The reason it still reads as a leftover is that 39ccf69 carried over every other difference the v1.5 rewrite introduced here, and the blur ratio is the only one it did not.

If the wider feather is deliberate for the realtime path, please close this — a comment recording why would be worth more than the change.

Measured

On a real prepared avatar frame (704x896, face box [250,123,431,365], crop 362x362), running both versions of the function on the same frame with the same face_seg output:

kernel partial-blend pixels centre-column ramp
current (0.1) 37 px 16,428 64 px
this PR (0.05) 19 px 9,243 36 px

1.78x more partially-blended pixels on the current path; 12.5% of mask pixels change, max delta 67. The ratio is under 2x because the Gaussian tails round to 0/255.

Testing note

scripts/realtime_inference.py writes every computed mask to {avatar_path}/mask/*.png and pickles the crop boxes, and later runs read those back instead of recomputing them. get_image_prepare_material() is called only from prepare_material(), so this change is inert for an avatar that has already been prepared. Reproducing any difference needs a fresh avatar_id, or the existing avatar directory removed first.

get_image() (offline) and get_image_prepare_material() (realtime) build the
blending mask the same way: face_seg over the expanded crop, crop to the face
box, paste onto a black canvas, drop everything above upper_boundary_ratio,
then Gaussian blur to soften the edge. The blur is the only step where the two
disagree. get_image() sizes the kernel at 0.05 of the crop width and
get_image_prepare_material() at 0.1, so the realtime mask feathers over twice
the distance. With the default expand=1.5 a 149x200 face gives a 300 px crop,
and the two kernels come out at 15 px and 31 px. Where the mask edge runs close
to the mouth, the wider ramp mixes more of the original frame back over the
generated pixels than the offline path would from the same inputs.

What the file history shows, because it cuts both ways:

  - Before v1.5 both functions used 0.1 and agreed.
  - v1.5 (db20431) rewrote get_image only: expand 1.2 -> 1.5, new mode and fp
    arguments, blur ratio 0.1 -> 0.05. It left get_image_prepare_material
    untouched on the older code.
  - 39ccf69 ("feat: real-time infer", TMElyralab#286) then ported that rewrite into
    get_image_prepare_material: expand 1.2 -> 1.5, added fp and mode, passed
    both through to face_seg. It edited the function line by line and left the
    0.1 in place.

So this is not a value that only one commit overlooked; a later commit went
through the same function and kept it. Nothing in the code or the commit
messages argues for it either way. The reason it still reads as a leftover is
that 39ccf69 carried over every other difference the v1.5 rewrite had
introduced here, and the blur ratio is the only one it did not. If the wider
feather is deliberate for the realtime path, please close this: a comment
recording why would be worth more than the change.

Testing note: scripts/realtime_inference.py writes every computed mask to
{avatar_path}/mask/*.png and pickles the crop boxes, and later runs read those
back instead of recomputing them. get_image_prepare_material() is called only
from prepare_material(), so this change is inert for an avatar that has already
been prepared. Reproducing any difference needs a fresh avatar_id, or the
existing avatar directory removed first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant