Limitless scaling using Event Driven Architecture and AWS Lambda

Posted in Software Architecture, AWS, published on

Event Driven Architecture has always been a wonderful paradigm for creating highly scalable and extensible architectures. In this short post I'll go over the advantages of using EDA together with AWS Lambda, a form of serverless compute.

Event Driven Architecture using AWS Lambda

Basics of Event Driven Architecture

I won't go over all the intricate details of an event driven architecture but I still want to highlight some important concepts and components.

In an Event Driven Architecture a single Producer creates messages that are pushed to a Message Channel. From there, many Consumers can read those messages and process them.

The key to scalability in an EDA is that the Message Channel is horizontally scalable and distributed. It can just as easily handle 1000 messages as it can handle 1.000.000 messages per seconds.

Just being able to receive hundreds of thousands of messages won't make your architecture highly scalable. If you use a single server as the consumer for the messages in your queue you'll be stuck with an immense backlog. Processing the messages received is as import as receiving them is.

Build an EDA with AWS Services

To build an Event Driven Architecture using AWS Services, you have a number of services available to you.

Producers

The Producers of your messages largely depend on your business case. In most cases I've seen a custom piece of software (think reservation system, transaction registration, etc) publishes a message to the Message Channel.

Message Channel

The heart of our EDA, the Message Channel is most commonly built using AWS Simple Notification Service. AWS SNS is a Pub/Sub messaging service that is highly scalable, durable and fully managed.

A message that is sent to SNS is always sent to a specific topic. You can think of a topic as a sort of categorization of the type of messages. Consumers that subscribe to an SNS topic will receive all message sent to that topic.

Since SNS is a pub/sub mechanism, AWS SNS will try to deliver messages to the subscribers of a particular topic (push). To ensure message delivery, you will need to ensure high availability of your consumers, and ideally setup a dead letter queue to store any messages that could not be delivered.

Consumers

I'm available to help you
Are you looking for someone to help build your next software architecture? I'm available for new projects as a freelance Solution Architect. Feel free to contact me and discuss what you need.

Some interesting resources

Simplify your Pub/Sub Messaging with Amazon SNS Message Filtering