Amazon Computer Services
Amazon Simple Queue Service (Amazon SQS)
Amazon Amazon Simple Queue Service is a fast, reliable, scalable, and completely handled message queuing service. Amazon SQS makes it simple and costs effective to decouple the components of a cloud application. We can use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be continuously available.
With Amazon SQS, we can offload the administrative burden of operating and scaling a hugely possible messaging cluster while paying a low cost for only what we use. Using Amazon SQS, we can store application messages on reliable and scalable infrastructure, enabling us to move data between distributed components to perform different tasks as needed.
Why use SQS?
There are many reasons to use SQS that are as follows:
Decoupling application processes
When we start a project, we may not be able to predict what the future needs of the project would be. By segregating the data generation process, the data consumption process and implementing a message queue in between the processes, we can create a data-based interface. All the processes involved in the application can implement this data-based interface. In such a scenario, any process involved in the overall application workflow can be changed without disturbing other processes.
Since the application processes are decoupled with a message queue, we can quickly scale up the data generation and data consuming processes to control the rate at which the data is processed. Without changing the underlying code, and by simply adding more processing resources, we can scale up the whole process.
Guaranteed message delivery
Use of a message queue ensures that a message is delivered at least once and eventually processed by the processing application as long as the process continues reading the queue. Depending on the configuration of the queue, we can also ensure that a message is processed only once. Such a guarantee is possible by SQS, as when a process retrieves the message from the queue, it temporarily removes the message from the queue.
When the client informs the queue that it has finished processing the message, SQS deletes the message from the queue. If the client does not respond to the queue in a specific duration, SQS places the message back in the queue. This way, if a client does not process the message, it would be available for another client to handle.
Message order guarantee
In many scenarios, the order in which the data is processed is significant. SQS provides a mechanism to process the data in a predefined order. Asynchronous data processing: There are specific requirements wherein we do not need to process the data immediately. SQS allows us to handle the messages asynchronously. That means we can add a message on the queue but do not need to treat it quickly. We can add as many messages as required and process them later.
Building resilience in the system
If some processes in your application environment go down, our entire environment does not go down with it. Since SQS decouples the processes, if a client processing a message goes down, the message is re-added to the queue after a specific duration. This message is either processed by another client or the same client after it recovers. This way, SQS builds resilience in the system.
Building redundancy with SQS
Sometimes processes do fail while processing the data. In such scenarios, if the data is not persisted, it is lost forever. SQS takes care of such a data risk by continuing the data until it’s completely processed. There is something called the put-get-delete paradigm used by SQS. It requires a queue to explicitly state that the data it has, has finished processing the message is pulled from the queue. The message data is kept safe with the queue and gets deleted from the queue only after confirmation that it is processed.
Creating a Queue
Let us go through the following steps and create and configure our first SQS queue:
- Log in to the AWS Management Console and navigate to https://console.aws. amazon.com/sqs/.
- Click on a Create New Queue button, as shown in the following figure:
- In the subsequent page, type the name of the queue as shown in the following figure, ensuring that you are in the intended region. If required, we can change the region from the top-right corner of the screen. It is additionally note that Queue Name is case sensitive and can have a maximum of up to 80 characters. When we are creating a FIFO queue, Queue Name must end with .fifo suffix:
- By default, the SQS wizard has a Standard queue select. Depending on our requirement, you can choose Standard or FIFO, as shown in the following figure:
- If you want to create a queue with the default parameters, we can click on the Quick-Create Queue button. Alternatively, we can click on the Configure Queue button for configuring the queue parameters. We can enter appropriate values in the queue attributes, as shown in the following figure. These queue attributes are described in the previous table:
- On the same screen, optionally you can enable redrive policy as shown in the following figure. This is an optional step and is required only if we want to divert unprocessed messages to a DLQ. If we enable the redrive policy, we need to specify the name of an existing queue that can act as a DLQ. We also need to specify the maximum receive count. If a message is received back for the specified number of time, it is moved to the DLQ:
- In this step, we can optionally enable Server Side Encryption (SSE) as shown in the following figure. If we choose to allow SSE, we can either use the default Customer Master Key (CMK) from the AWS Key Management Service (KMS), or we can specify any other existing CMK we have in KMS.
If we select any different key than the default CMK, we need to specify the key ARN manually. We also need to specify a Data Key Reuse Period, which can range between 1 minute and 24 hours. For encrypting or decrypting data, SQS requires a data key which is provided by KMS. Once a key is obtained from KMS, it can be used by SQS for the time specified in a Data Key Reuse Period before again going to KMS for a new data key:
- After giving all the required input on the screen, we can click on the Create Queue button. Once the queue is created, we can see the queue in the SQS dashboard, as shown in the following figure: