Questions about Semantic Segmentation Loss

[HaoDot]

Dear @wingrune,

Thank you for sharing such brilliant work! I have a couple of questions regarding the Semantic Segmentation Loss:

1. Absence of the target object in the input view: I am curious about how often the target object actually appears in the input image. When using PPO, the trajectory is random, meaning the target object might not exist within the view for the entire trajectory. If this happens, does the Semantic Segmentation Loss still function effectively?
2. More advanced segmentation models: Table 4 shows that the influence of the segmentation mask is crucial. Have you considered trying stronger, foundational models like SAM or FastSAM to generate the binary masks?

Thank you for your time, and I look forward to your reply!

[wingrune]

Thank you for your interest and for these thoughtful questions!

1. Regarding target visibility, the target object is often absent from the current view, especially during the early part of an episode. We did not explicitly measure the exact fraction of such observations, since it varies across episodes and policies. Also, PPO trajectories are sampled from the current policy rather than being uniformly random, but unsuccessful trajectories may indeed never observe the target.
   The segmentation loss remains well-defined in this case. When the target is not visible, its ground-truth mask is all zeros, and this observation serves as a negative example. The Dice and binary cross-entropy terms encourage the auxiliary head to predict no foreground pixels. We do not filter out empty masks during training. Therefore, even a trajectory in which the target never appears still provides useful background supervision, although it naturally contains no positive localization examples. Positive masks are provided by other trajectories and observations, including expert-guided and successful rollouts.
   In addition, the semantic reward encourages the policy to reach observations with non-zero target masks. Its mask-area component rewards an increase in the visible area of the target between consecutive observations. Thus, while the auxiliary segmentation loss learns to distinguish target presence from absence, the semantic reward encourages the navigation policy to actively move toward viewpoints where the target becomes visible and occupies a larger part of the image.

2. Regarding stronger segmentation models, we experimented with a GroundingDINO + SAM pipeline. However, we did not observe a substantial improvement in navigation performance compared with YOLOE, while this pipeline was significantly slower. For this reason, we selected YOLOE as a better accuracy-efficiency trade-off for our experiments.
   Nevertheless, as Table 4 suggests, segmentation quality remains an important bottleneck. We believe that a more accurate open-vocabulary segmentation model could further improve navigation performance. Exploring such models, including newer efficient foundation segmentation approaches, is a promising direction for future work.