Use the offline mask blur ratio in the realtime blending path - #422
Open
cuhawk wants to merge 1 commit into
Open
Use the offline mask blur ratio in the realtime blending path#422cuhawk wants to merge 1 commit into
cuhawk wants to merge 1 commit into
Conversation
get_image() (offline) and get_image_prepare_material() (realtime) build the blending mask the same way: face_seg over the expanded crop, crop to the face box, paste onto a black canvas, drop everything above upper_boundary_ratio, then Gaussian blur to soften the edge. The blur is the only step where the two disagree. get_image() sizes the kernel at 0.05 of the crop width and get_image_prepare_material() at 0.1, so the realtime mask feathers over twice the distance. With the default expand=1.5 a 149x200 face gives a 300 px crop, and the two kernels come out at 15 px and 31 px. Where the mask edge runs close to the mouth, the wider ramp mixes more of the original frame back over the generated pixels than the offline path would from the same inputs. What the file history shows, because it cuts both ways: - Before v1.5 both functions used 0.1 and agreed. - v1.5 (db20431) rewrote get_image only: expand 1.2 -> 1.5, new mode and fp arguments, blur ratio 0.1 -> 0.05. It left get_image_prepare_material untouched on the older code. - 39ccf69 ("feat: real-time infer", TMElyralab#286) then ported that rewrite into get_image_prepare_material: expand 1.2 -> 1.5, added fp and mode, passed both through to face_seg. It edited the function line by line and left the 0.1 in place. So this is not a value that only one commit overlooked; a later commit went through the same function and kept it. Nothing in the code or the commit messages argues for it either way. The reason it still reads as a leftover is that 39ccf69 carried over every other difference the v1.5 rewrite had introduced here, and the blur ratio is the only one it did not. If the wider feather is deliberate for the realtime path, please close this: a comment recording why would be worth more than the change. Testing note: scripts/realtime_inference.py writes every computed mask to {avatar_path}/mask/*.png and pickles the crop boxes, and later runs read those back instead of recomputing them. get_image_prepare_material() is called only from prepare_material(), so this change is inert for an avatar that has already been prepared. Reproducing any difference needs a fresh avatar_id, or the existing avatar directory removed first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get_image()(offline) andget_image_prepare_material()(realtime) build the blending mask the same way:face_segover the expanded crop, crop to the face box, paste onto a black canvas, drop everything aboveupper_boundary_ratio, then Gaussian blur to soften the edge.The blur is the only step where the two disagree.
get_image()sizes the kernel at0.05of the crop width (blending.py:81) andget_image_prepare_material()at0.1(blending.py:134), so the realtime mask feathers over roughly twice the distance. Where the mask edge runs close to the mouth, the wider ramp mixes more of the original frame back over the generated pixels than the offline path would from the same inputs.What the file history shows, because it cuts both ways
0.1and agreed.db20431) rewroteget_imageonly:expand1.2 → 1.5, newmodeandfparguments, blur ratio 0.1 → 0.05. It leftget_image_prepare_materialon the older code. This commit created the divergence.39ccf69("feat: real-time infer", feat: real-time infer #286) then ported that rewrite intoget_image_prepare_material:expand1.2 → 1.5, addedfpandmode, passed both through toface_seg. It edited the function line by line and left the0.1in place.So this is not a value that only one commit overlooked; a later commit went through the same function and kept it. Nothing in the code or the commit messages argues for it either way. The reason it still reads as a leftover is that
39ccf69carried over every other difference the v1.5 rewrite introduced here, and the blur ratio is the only one it did not.If the wider feather is deliberate for the realtime path, please close this — a comment recording why would be worth more than the change.
Measured
On a real prepared avatar frame (704x896, face box
[250,123,431,365], crop 362x362), running both versions of the function on the same frame with the sameface_segoutput:1.78x more partially-blended pixels on the current path; 12.5% of mask pixels change, max delta 67. The ratio is under 2x because the Gaussian tails round to 0/255.
Testing note
scripts/realtime_inference.pywrites every computed mask to{avatar_path}/mask/*.pngand pickles the crop boxes, and later runs read those back instead of recomputing them.get_image_prepare_material()is called only fromprepare_material(), so this change is inert for an avatar that has already been prepared. Reproducing any difference needs a freshavatar_id, or the existing avatar directory removed first.