Fix PeriodicImpulse/PeriodicSequence watermark regression (#39026)#39465
Open
uditjainstjis wants to merge 1 commit into
Open
Fix PeriodicImpulse/PeriodicSequence watermark regression (#39026)#39465uditjainstjis wants to merge 1 commit into
uditjainstjis wants to merge 1 commit into
Conversation
The watermark reported by ImpulseSeqGenDoFn stalled at the last emitted element's timestamp when the SDF deferred to wait for the next fire time. For a side input firing every N seconds, downstream watermark age climbed to N then snapped back (saw-tooth), a regression introduced when process() was rewritten between 2.66.0 and 2.74.0. When deferring because the next scheduled output is in the future, advance the watermark to that next fire time: no element will be produced before it, so it is a valid watermark and restores the pre-2.74.0 behavior. The advance is guarded for pre-timestamped data, whose event times may be out of order, so we never declare future late events droppable. Adds unit tests that drive process() directly: one fails on the unpatched code (watermark stuck at last emitted) and passes after the fix; another asserts the watermark is not advanced past emitted timestamps for pre-timestamped data. Generated-by: Claude (Anthropic AI assistant)
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Contributor
|
Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #39026
Problem
After upgrading from 2.66.0 to 2.74.0, streaming jobs using a
PeriodicImpulseside input that fires at long intervals saw the job's watermark age follow a saw-tooth pattern, climbing up to the fire interval before snapping back. Users with watermark-age alerts were paged spuriously.Root cause
process()inImpulseSeqGenDoFnwas rewritten between 2.66.0 and 2.74.0. The old code, after emitting the element at indexk, recomputed the timestamp for indexk+1and set the watermark to that next fire time before deferring — so the watermark sat ~one interval ahead of the last emitted element and the reported age stayed flat.The rewritten code sets the watermark to the just-emitted element's timestamp and never advances it on the defer path. Between fires the watermark is therefore held at the last element's event time, so downstream watermark age grows to
fire_intervaland resets on each fire.Fix
When
process()defers because the next scheduled output is in the future, advance the watermark to that next fire time (monotonic-guarded). No element can be produced before it, so it is a valid watermark, and this restores the pre-2.74.0 behavior.The advance is gated on
not self._is_pre_timestamped: for pre-timestampeddata, event times may be intentionally out of order (a later element can carry an earlier timestamp, treated as a late event per thePeriodicImpulsedocstring), so we must not declare future late events droppable.Testing
Added two unit tests that drive
process()directly with aRestrictionTrackerView+ManualWatermarkEstimator:test_watermark_advances_to_next_fire_on_defer— fails on unpatchedmaster(watermark stuck at last emitted element) and passes with the fix. This is the regression reproduced as a unit test (the Dataflow-side symptom cannot be reproduced in a unit test).test_watermark_not_advanced_past_emitted_for_pre_timestamped— asserts the watermark is not advanced past emitted event times for pre-timestamped data.Full
periodicsequence_test.pypasses locally (26 passed, 1 skipped).yapf==0.43.0clean, pylint 10.00/10.CHANGES.mdwith noteworthy changes.