Stream Processing (Flink)

Many products are defined by streams that do not end: views, likes, payments, sensor readings, ad impressions, location pings, and login attempts. A nightly batch job can answer yesterday's question, but it cannot power a live fraud decision or a trending list that should react within seconds.

• Database hot spots: incrementing one row per event creates write contention on popular entities and melts the primary database.
• Delayed decisions: fraud, abuse, and operational alerts lose value if the system waits an hour to aggregate signals.
• Stateful logic: useful questions need memory, such as how many failures this user had in the last five minutes or which videos are rising fastest in this region.

A Flink job is a graph of operators. Source operators read from Kafka, transformation operators parse, filter, key, window, and aggregate, and sink operators write results to Redis, Elasticsearch, a database, or another Kafka topic. The powerful part is keyed state: Flink can keep independent state per key, such as per user, per video, or per merchant.

events = KafkaSource("video-events")

views_per_video = events
  .filter(event => event.type == "view")
  .keyBy(event => event.video_id)
  .window(TumblingEventTimeWindow.of(1 minute))
  .aggregate(count())

views_per_video.sinkTo(RedisSink("video:{id}:views:last_minute"))

State may live in memory or RocksDB on each worker, but it is not disposable cache. Flink periodically checkpoints that state so a failed worker can recover and continue from a consistent point.

Since streams are infinite, aggregations need boundaries. Awindow says which events belong to one calculation. The right window shape depends on the product question.

Why windows are not just SQL GROUP BY

A database query over a fixed table knows all rows are present. A stream window must decide when it is ready to produce a result while late events may still be on the way. That is where event time and watermarks matter.

Processing time is the clock on the machine when Flink sees the event. Event time is when the event actually happened, usually recorded by the producer or device. Event time is what users normally mean, but events arrive late, out of order, or after retries.

event stream arrival order:
  12:00:03  view(video_1)   event_time=12:00:01
  12:00:04  view(video_1)   event_time=12:00:02
  12:00:08  view(video_1)   event_time=12:00:00  # late

watermark = max_seen_event_time - allowed_lateness
watermark = 12:00:02 - 5 seconds = 11:59:57

A one-minute window can close when the watermark passes the window end.

• Use event time when correctness depends on when the user action happened, such as analytics, billing rollups, or fraud windows.
• Use processing time for operational signals where arrival time is the truth, such as current ingestion rate or worker health.
• Watermarksare Flink's progress estimate: "I believe I have seen almost everything up to this event time."

Flink achieves exactly-once state updates by periodically taking a distributed snapshot called a checkpoint. The checkpoint records operator state and source positions, such as Kafka offsets. On failure, Flink restores the last successful checkpoint and resumes input from the matching offsets, so state and input position move together.

checkpoint 41
  Kafka offsets: partition 0 -> 18420, partition 1 -> 9921
  keyed state: video_7 count=531, video_8 count=44
  sink transaction: pending batch chk-41

worker crashes

restore checkpoint 41
  rewind Kafka consumers to recorded offsets
  restore keyed state
  either commit or abort sink transaction for chk-41

Exactly-once is end-to-end only if the sink participates. Kafka transactional sinks can align with checkpoints. Databases often require idempotent upserts keyed by window ID, or a two-phase commit sink. Emails and external APIs are usually at-least-once unless you add explicit dedupe.

Stream processing shines when the answer must be continuously updated and can be expressed as state over events. It is not a replacement for every cron job or analytical warehouse query; it is a specialized tool for fresh, incremental computation.

• Trending: compute weighted views, likes, and shares in a sliding window so a feed can react to sudden momentum.
• Real-time metrics: aggregate request rates, latency histograms, payment approvals, or ad impressions into dashboards.
• Fraud detection: keep per-card, per-device, and per-user state for velocity checks, unusual geography, or failed-login bursts.
• Enrichment: join click events with campaign metadata or user segment state before writing analytics events.

• State growth: session windows, high-cardinality keys, and generous lateness can grow state without bound. Use TTLs and monitor checkpoint size.
• Skew: one hot key can overload a single task slot. You may need key salting, hierarchical aggregation, or a product-level limit.
• Backpressure: a slow sink causes upstream operators and Kafka consumers to lag. Watch checkpoint duration and consumer lag.
• Reprocessing changes history: deploying new logic and replaying old events can produce different results. Version outputs or backfill to a separate destination when correctness matters.