SQS asynchronous tasks

SQS is a service (like RabbitMQ) that allows you to queue messages (aka "jobs"). It is a good fit for asynchronous tasks because it integrates natively with AWS Lambda.

Whenever a new message (job) is sent to SQS, Lambda is invoked with the message data. That means that there is no need to poll SQS, or run daemon/long-running processes to wait for messages. Lambda is invoked only when there are messages to process.

Handling SQS events

To handle SQS events (opens in a new tab), extend the SqsHandler class:

use Bref\Context\Context;
use Bref\Event\Sqs\SqsEvent;
use Bref\Event\Sqs\SqsHandler;
 
class MyHandler extends SqsHandler
{
    public function handleSqs(SqsEvent $event, Context $context): void
    {
        foreach ($event->getRecords() as $record) {
            // We can retrieve the message body of each record via `->getBody()`
            $body = $record->getBody();
 
            // do something
        }
    }
}

Then, create a Lambda function that listens to SQS events with the handler you created:

functions:
    # ...
    resizeImage:
        handler: App\MyHandler
        events:
            - sqs:
                arn: arn:aws:sqs:eu-west-1:111111111111:queue-name
                # process one message at a time
                batchSize: 1

Creating SQS queues

It is possible to deploy a preconfigured SQS queue in serverless.yml using the Queue feature of the Lift plugin (opens in a new tab). For example:

constructs:
    my-queue:
        type: queue
        worker:
            handler: handler.php

Partial Batch Response

While handling a batch of records, you can mark it as partially successful to reprocess only the failed records.

In your function declaration in serverless.yml, set functionResponseType to ReportBatchItemFailures to let your function return a partial success result if one or more messages in the batch have failed.

functions:
    worker:
        handler: handler.php
        events:
            - sqs:
                arn: arn:aws:sqs:eu-west-1:111111111111:queue-name
                batchSize: 10
                functionResponseType: ReportBatchItemFailures

In your PHP code, you can now use the markAsFailed method:

    public function handleSqs(SqsEvent $event, Context $context): void
    {
        foreach ($event->getRecords() as $record) {
            // do something
 
            // if something went wrong, mark the record as failed
            $this->markAsFailed($record);
        }
    }

SQS polling and AWS costs

AWS Lambda maintains a continuous "poller" for SQS that invokes our function when messages are available. This process, while eliminating the need for manual polling, still creates SQS requests, which can incur charges. A single SQS queue with Lambda will be free since it fits in the AWS free tier. However, if you plan on deploying a lot of SQS queues, you should be aware of potential costs (about $0.26/month per queue).

If you want to reduce the number of SQS requests (and thus potential costs), you can use the MaximumBatchingWindowInSeconds option (opens in a new tab). This setting delays message delivery to the Lambda function when batching, aiming to assemble the largest possible batch within the specified timeframe, counted as a single SQS API request. Note that this technique introduce a delay in message processing, so be aware of the trade-off.

Learn more

Learn more about all the options available for SQS events in serverless.yml in the Serverless Framework documentation (opens in a new tab).

You can also learn more about SQS, workers, scaling queues, and dealing with errors in Serverless Visually Explained (opens in a new tab).