Stream Processing
Also known as:
Event Stream Processing
Real-Time Stream Processing
Streaming Computation
Definition
Stream processing is a long-running computation over an unbounded sequence of events. Instead of waiting for a file or a table to be complete, the job transforms, joins, and aggregates records as they arrive and emits updated results continuously. Time is part of the input: windows, watermarks, and event time decide when a result is ready. The events usually live in a log such as Kafka. The processor is a separate layer such as Kafka Streams, Apache Flink, or Spark Structured Streaming.
Popular reads
View All
How Google manages billions of lines of code in one monorepo
Sep 04, 2026
Transactional Outbox Pattern: Never Lose an Event Again
Apr 07, 2026
Git Flow vs GitHub Flow
Jun 05, 2026
Payment System Design: Ledger, Idempotency, and Settlement
Jul 18, 2026
Two-Phase Commit: The Protocol That Keeps Distributed Transactions Honest
Dec 06, 2025
Cursor Skills: How to Create and Use Agent Skills
Jun 23, 2026
Key Takeaways
- A stream never ends, so you cannot scan ‘all the rows’. You window, keep keyed state, and update results incrementally.
- The log (Kafka, Kinesis) stores and replays events. Stream processing is the computation on top of that log.
- Correct windows use event time and watermarks, not the wall clock of the worker.
- Restarts replay data. Treat duplicates as normal and make sinks idempotent, or use a transactional sink if you truly need end-to-end exactly-once.
How It Works
- Producers append events to a durable log. Each event carries a payload and usually an event-time timestamp.
- A processor reads the log, partitions work by key, and applies filters, maps, joins, and windowed aggregations.
- Stateful operators checkpoint state and offsets so a crash can rewind and continue.
- Results go to another topic, a database, a search index, or an alert system as soon as windows close.
Where It Is Used
- Fraud scoring and inventory checks that must decide before a checkout completes.
- Live product metrics, sessionization, and personalization features built from clickstreams.
- CDC pipelines that keep search indexes and caches in sync with an OLTP database.