Case Study on AWS SQS (Simple Queue Service)

What is AWS SQS ?

Amazon Simple Queue Service (Amazon SQS) is a fully managed queuing service that helps decouple applications, distributed systems, and microservices to increase fault tolerance. SQS queues come in two distinct types:

• Standard SQS queues are able to scale to enormous throughput with at-least-once delivery.
• FIFO queues are designed to guarantee that messages are processed exactly once in the exact order that they are received and have a default rate of 300 transactions per second.

As customers explore SQS FIFO queues, they often have questions about how the behavior works when messages arrive and are consumed. This post walks through some common situations to identify the exact behavior that you can expect. It also covers the behavior of message groups in depth and explains why message groups are key to understanding how FIFO queues work.

There are two different was that an SQS consumer can poll for messages. The first way is short polling. In this scenario, a consumer will look at the queue, read any messages that are in the queue, and then will return. If there are no messages in the queue, then the short-polling consumer will return immediately. The other way is to do long polling. In this scenario, the consumer will wait for a number of seconds for messages to appear in the queue.

Since AWS charges you for each SQS request, it may be more economical for your consumer to do long polling, since there will be fewer requests if messages are put into the queue on an infrequent basis.

Messages

A Message contains a payload, which can be any amount of data up to 256K bytes. A message can also contain custom attributes, which are name/value pairs. When a message is placed into a queue, it can have a non-zero Time-to-Live (TTL). If the message has sat in the queue without being consumed, and its TTL has expired, then the message is automatically deleted from the queue.

How SQS Queues Works

• SQS allows queues to be created, deleted and messages can be sent and received from it
• SQS queue retains messages for four days, by default.
• Queues can be configured to retain messages for 1 minute to 14 days after the message has been sent.
• SQS can delete a queue without notification if any action hasn’t been performed on it for 30 consecutive days.
• SQS allows the deletion of the queue with messages in it.

SQS Security and reliability

• SQS stores all message queues and messages within a single, highly-available AWS region with multiple redundant Availability Zones (AZs)
• SQS supports the HTTP over SSL (HTTPS) and Transport Layer Security (TLS) protocols.
• SQS supports Encryption at Rest. SSE encrypts messages as soon as SQS receives them and decrypts messages only when they are sent to an authorized consumer.
• SQS also supports resource-based permissions

Visibility timeout

• SQS does not delete the message once it is received by a consumer, because the system is distributed, there’s no guarantee that the consumer will actually receive the message (it’s possible the connection could break or the component could fail before receiving the message)
• Consumer should explicitly delete the message from the Queue once it is received and successfully processed
• As the message is still available on the Queue, other consumers would be able to receive and process and this needs to be prevented
• SQS handles the above behavior using Visibility timeout.
• SQS blocks the visibility of the message for the Visibility timeout period, which is the time during which SQS prevents other consuming components from receiving and processing that message
• Consumer should delete the message within the Visibility timeout. If the consumer fails to delete the message before the visibility timeout expires, the message is visible again for other consumers.
• Once Visible the message is available for other consumers to consume and can lead to duplicate messages.
• Visibility timeout considerations
• clock starts ticking once SQS returns the message
• should be large enough to take into account the processing time for each of the message
• default Visibility timeout for each Queue is 30 seconds and can be changed at the Queue level
• when receiving messages, a special visibility timeout for the returned messages can be set without changing the overall queue timeout using the receipt handle
• can be extended by the consumer, using ChangeMessageVisibility , if the consumer thinks it won’t be able to process the message within the current visibility timeout period. SQS restarts the timeout period using the new value
• a message’s Visibility timeout extension applies only to that particular receipt of the message and does not affect the timeout for the queue or later receipts of the message.
• SQS has an 120,000 limit for the number of inflight messages per queue i.e. message received but not yet deleted and any further messages would receive an error after reaching the limit

SQS Job Observer Pattern

1. Enqueue job requests as SQS messages.
2. Have the batch server dequeue and process messages from SQS.
3. Set up Auto Scaling to automatically increase or decrease the number of batch servers, using the number of SQS messages, with CloudWatch, as the trigger to do so.

There are two main use cases for considering messaging in most applications — light weight eventing and messaging between components, and processing large streams of data. If you are using AWS as your main platform, it makes sense to consider using AWS-native platforms to handle the messaging. For the case of simple peer-to-peer messaging, where the exact order of the messages is not mandated and a single application has to be notified, SQS is a good solution. It is easy to set up and manage, and it can be used to perform a simple event notification or be used in a request/response paradigm. For the case where a number of applications have to be notified of an event, SNS is a good solution, as it supports the concept of topics. SNS can be used to deliver notifications to a wide variety of notification mechanisms (email, SMS, Lamba, REST), and can interface with SQS queues for notifications to individual applications. If your application has a use-case where it needs a heavy amount of streaming, then Kinesis is an easy-to-use streaming platform for processing large streams of data. Kinesis Analytics can be used to perform Complex Event Processing, and to find interesting events within the streaming data. For what most applications need, Kinesis is preferable to Apache Kafka because it is easy to set up and administer, and you will avoid many of the operational complexities that usually plagues a deployment of Kafka.

The travel agency like redBus anticipates expanding the AWS solution to include Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) for monitoring, alerts, and intercommunication.

Hope you like it !!!

Thank you !!!