Limitless scaling using Event Driven Architecture and AWS Lambda

Written by Michael Anckaert - Published on - Posted in Software Architecture, AWS

Event Driven Architecture has always been a wonderful paradigm for creating highly scalable and extensible architectures. In this short post I'll go over the advantages of using EDA together with AWS Lambda, a form of serverless compute.

Event Driven Architecture using AWS Lambda

Basics of Event Driven Architecture

I won't go over all the intricate details of an event driven architecture but I still want to highlight some important concepts and components.

In an Event Driven Architecture a single Producer creates messages that are pushed to a Message Channel. From there, many Consumers can read those messages and process them.

The key to scalability in an EDA is that the Message Channel is horizontally scalable and distributed. It can just as easily handle 1000 messages as it can handle 1.000.000 messages per seconds.

Just being able to receive hundreds of thousands of messages won't make your architecture highly scalable. If you use a single server as the consumer for the messages in your queue you'll be stuck with an immense backlog. Processing the messages received is as import as receiving them is.

Build an EDA with AWS Services

To build an Event Driven Architecture using AWS Services, you have a number of services available to you.

Producers

The Producers of your messages largely depend on your business case. In most cases I've seen a custom piece of software (think reservation system, transaction registration, etc) publishes a message to the Message Channel.

Message Channel

The heart of our EDA, the Message Channel is most commonly built using SNS (AWS Simple Notification Service) or SQS (Simple Queue Service).

AWS SNS is a Pub/Sub messaging service that is highly scalable, durable and fully managed.

A message that is sent to SNS is always sent to a specific topic. You can think of a topic as a sort of categorization of the type of messages. Consumers that subscribe to an SNS topic will receive all message sent to that topic.

Since SNS is a pub/sub mechanism, AWS SNS will try to deliver messages to the subscribers of a particular topic (push). To ensure message delivery, you will need to ensure high availability of your consumers, and ideally setup a dead letter queue to store any messages that could not be delivered.

Amazon SQS on the other hand is a fully managed message queuing service.

Messages sent to an SQS Queue need to be retrieved by a consumer, so think of SQS as a pull based system. Consumers come and fetch new messages to process. This of course means that SQS is a One to One message service, a single message is consumed once. This is opposite of SNS, which is a One to Mange message service, a single message can be consumed by multiple consumers.

Consumers

Consumers in an Event Driven Architecture will process the message (or messages) they receive (or have retrieved from a queue).

A powerful pattern is to use serverless technology to write your consumers. This allows your architecture to scale up very fast, while not consuming any resources if there are no messages to process.

Of course you are not limited to Serverless technologies to consumer messages in an Event Driven Architecture. Virtual machines running on EC2 or containerized applications on ECS / EKS are perfectly capable consumers. Using SNS you can publish messages to a HTTP/S endpoint for example.

Some interesting resources

Simplify your Pub/Sub Messaging with Amazon SNS Message Filtering