As modern application architecture are rapidly moving from trivial monolithic to server-less microservices and event-based systems which makes the application more flexible, fault-tolerant, modular and easily maintainable apart from these benefits it also comes with the added overhead of managing the communication between various components this is where messaging systems come in to picture. Amazon Web Services (AWS) provides a well equipped fully managed messaging stack’s as services like SNS, SQS, Kinesis which comes handy in a very wide range of business scenarios. In this article, we will see details about each and the scenarios where they fit.
Table of Contents:
SNS is a push notification service that is used to send event notifications as messages to a number of recipients. It is a flexible, fast, fully, and cost-effective service to send push notifications to mobile device users, email recipients, HTTP endpoints and other distributed services, messages are grouped into a logical entity known as Topic to which various consumer subscribe and same copy of the message is sent to different message consumers. SNS is easy to use and manage and can be used in scenarios where a number of services/consumers need to be notified of an event.
SNS can be used to fanout mobile push notifications to iOS, Android, Fire OS, Windows, and Baidu-based devices it can also fanout Short Message Service( SMS) to a number of countries and fanout email messages. It comes with a limitation of 10 emails per second.
Some common use cases would be, send event notification for failure and other monitoring parameters, sending transactional email and SMS notifications, RSS feed, etc.
SQS is a fully managed distributed message queuing system. Consumers have to poll SQS and pull messages. It provides the most basic way to perform inter-process communication and message delivery is guaranteed. A queue, can be used to send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be always available.
SQS comes in two types of services STANDARD and FIFO. Standard messages are unordered while in FIFO message can be consumed from the queue in first-in-first-out order. As it is a fully managed service there is no overhead of managing message-oriented middleware and it can be scaled very easily on demand.
SQS also provides the feature to persist messages for a period of 1 minute to 14 days which makes sure the message is consumed even if the receiver service is temporarily busy or down.
SQS maintains a dead letter queue where those messages are stored that were not consumed for some reason.
Kinesis is based on Apache Kafka, it is a fully managed, real-time Data Streaming and analytic service used to handle the very large stream of data, big data from a large number of sources, for example, Netflix uses Kinesis to handle Terabytes of data each day from events occurring from millions of connected IoT devices in real-time. The real-time data may include video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications.
The Kinesis stream is subdivided into shards and each shard can process a stream of data in isolation of other shards. These subdivisions provide a degree of load-balancing in the stream. Kinesis is very easy to set up and scale and minimizes the overhead of setting and maintaining Kafka clusters.
Kinesis data streams can easily scale to hundreds of data sources and process gigabytes of data per second.
Amazon Kinesis Data Firehose is used to reliably load streaming data into data lakes, data stores, and analytics tools.
Another thing is Amazon Kinesis Data Analytics, which is used to analyze streaming data, gain actionable insights, and respond to business and customer needs in real-time. It reduces the complexity of the building, managing, and integrating streaming applications with other AWS services. It comes with SQL support and provides a SQL editor to query on the data stream.
By combining services SNS and SQS, we can achieve functionality similar to kinesis in a very simple and easy, scalable way. SNS provides real-time event notification push to the SQS queues and SQS retains the message and guarantees its delivery. This comes with a limitation as SNS can deliver messages to standard queues and not FIFO queues. Both SNS and SQS can be scaled automatically without any external instructions.
Table 1 shows the major differences in SNS, SQS, and Kinesis.
|Simple Notification Service (SNS)||Simple Queue Service
|Message access||Push||Pull||Data Stream|
|Persistence||None (Until retries exhaust)||One minute to 14 days||7 days|
|Message size||256 kb||256 KB (can be increased up to 2 GB with AWS S3)||1 MB|
|Cost||$0.50 per million (Mobile Push Notifications), $2.00 per 100,000 (email/email-JSON), $0.60 per million (HTTP/s)||$0.40 per million requests||$0.014 per million requests|
|Free tier||Free one million requests per month||Free one million requests per month||Not free|
We can see all three services provide an efficient way to handle event notifications, messaging between components, and real-time large data streams.
SNS manages event notification by push mechanisms in a fire and forgets manner which doesn’t guarantee delivery.
SQS provides support for messages to be consumed by the consumer through the use of queue and guarantees delivery by using the retention period.
Kinesis provides support for processing large data streams in real-time and it reduces the huge burden for setting and maintaining Kafka clusters.