Amazon Kinesis
Kinesis is a streaming data pipe that accepts continuously arriving events and lets multiple consumers read them in near real time. It is built around ongoing event flow rather than isolated work items.
▶Architecture Diagram
📊 Data FlowDashed line animations indicate the flow direction of data or requests
When click streams and sensor events arrive every second, batch loading or ordinary queues alone make it easy to lose ordering and real-time behavior. Without a layer that can keep receiving the flow continuously, downstream processors fall behind quickly.
In the past, logs were often collected into files for later batch processing, but as real-time response demands grew, stream platforms that flow data as soon as it arrives became necessary. This is why real-time data pipes like Kinesis became important.
Kinesis continuously receives data into streams for multiple consumers to read, and optionally exports it to S3 or Redshift through Firehose. It fits continuous data such as logs, clickstreams, audio and video, or IoT telemetry, and Lambda can be attached for real-time post-processing.
Kinesis and SQS are both used for async data flows, but the model is different. Kinesis is strong when multiple consumers need ordered, continuous streams, while SQS is closer to a queue that safely consumes individual task messages. If you need real-time processing over an event stream, look at Kinesis; if you need reliable delivery of discrete work items, look at SQS.
Well-suited for clickstreams, IoT events, log collection, real-time analytics pipelines, and stream post-processing. Overkill for simple queuing where tasks are queued and consumed without ordering guarantees.