|
class PeriodicAssigner extends AssignerWithPeriodicWatermarks[SensorReading] { |
|
|
|
// 1 min in ms |
|
val bound: Long = 60 * 1000 |
|
// the maximum observed timestamp |
|
var maxTs: Long = Long.MinValue |
|
|
|
override def getCurrentWatermark: Watermark = { |
|
new Watermark(maxTs - bound) |
|
} |
|
|
|
override def extractTimestamp(r: SensorReading, previousTS: Long): Long = { |
|
// update maximum timestamp |
|
maxTs = maxTs.max(r.timestamp) |
|
// return record timestamp |
|
r.timestamp |
|
} |
|
} |
When the task recovery from the cluster failure, the maxTS is initialized with Long.MinValue and the task manager will call getCurrentWatermark method which will - 60 * 1000 and make the maxTS to 9223372036854715808. This invalid ts will block the stream work with the web Dashboard all green but actually they will never push the watermark forward. I know this is just a example but maybe a little warning about this example's job recovery behiver will make the reader less frustrated with their code?
examples-scala/src/main/scala/io/github/streamingwithflink/chapter6/WatermarkGeneration.scala
Lines 45 to 62 in c188681
When the task recovery from the cluster failure, the
maxTSis initialized withLong.MinValueand the task manager will callgetCurrentWatermarkmethod which will- 60 * 1000and make the maxTS to9223372036854715808. This invalid ts will block the stream work with the web Dashboard all green but actually they will never push the watermark forward. I know this is just a example but maybe a little warning about this example's job recovery behiver will make the reader less frustrated with their code?