Skip to content

Chapter 6 example 6-3 watermark ts will overflow when job recovery from failure #5

Description

@dispensable

class PeriodicAssigner extends AssignerWithPeriodicWatermarks[SensorReading] {
// 1 min in ms
val bound: Long = 60 * 1000
// the maximum observed timestamp
var maxTs: Long = Long.MinValue
override def getCurrentWatermark: Watermark = {
new Watermark(maxTs - bound)
}
override def extractTimestamp(r: SensorReading, previousTS: Long): Long = {
// update maximum timestamp
maxTs = maxTs.max(r.timestamp)
// return record timestamp
r.timestamp
}
}

When the task recovery from the cluster failure, the maxTS is initialized with Long.MinValue and the task manager will call getCurrentWatermark method which will - 60 * 1000 and make the maxTS to 9223372036854715808. This invalid ts will block the stream work with the web Dashboard all green but actually they will never push the watermark forward. I know this is just a example but maybe a little warning about this example's job recovery behiver will make the reader less frustrated with their code?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions