Event Sourcing with SNS and SQS

Introduction

Event sourcing is a powerful architectural pattern that has transformed the way I build and scale distributed systems. Instead of persisting only the current state, I capture all changes as historical events. This provides a complete audit trail and the ability to reconstruct the state at any point in time—two advantages that can be incredibly useful in complex, evolving applications.

In this post, I’ll show you how to implement event sourcing using AWS services, particularly with Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service). I’ll combine modern technologies like NodeJS, NestJS, and TypeScript, and I’ll also demonstrate how it all can be aligned with Domain-Driven Design (DDD) and Command Query Responsibility Segregation (CQRS) principles.

Understanding Event Sourcing

At its core, event sourcing means I represent changes in the application state as a series of discrete events, rather than simply overwriting data in a database. Each event is an immutable fact that describes something that happened in the system. By storing every event, I can replay or rebuild application state at any point in history.

This approach brings a few key benefits:

A full audit log of everything that has ever happened in your system
The ability to replay events to debug issues, migrate data, or reconstruct corrupted read models
Enhanced scalability and performance when combined with CQRS, since read operations can be served by specialized read models

While event sourcing adds complexity—particularly in consistency and data modeling—these trade-offs are often worthwhile in systems that handle critical data and require robust auditing or are heavily read-oriented.

AWS Services for Event Sourcing

AWS provides a powerful suite of services to implement event sourcing in a scalable and cost-effective way. Two of my favorites for building a microservices architecture are SNS and SQS.

Amazon Simple Notification Service (SNS)

SNS is a fully managed pub/sub messaging service that I use to broadcast events to various subscribers. After a domain event is published, multiple microservices or consumers can receive a copy of that event, keeping the system loosely coupled. This helps me easily fan out new features or microservices without changing existing publishers.

Amazon Simple Queue Service (SQS)

SQS is a fully managed queue service that can buffer events published from SNS. Each microservice can have its own queue, pulling events at its own pace. This ensures reliable delivery, even if a consumer is temporarily offline. As the number of events grows, SQS can scale horizontally, and I can add more consumers for parallel processing. ^[3][2]

Implementing Event Sourcing with NodeJS and NestJS

I typically prefer NestJS (a progressive Node.js framework built with TypeScript) for building reliable server-side applications. Below is how I set up event publishing and consuming using SNS and SQS.

Setting Up the Project

First, I create a new NestJS project using the CLI, then install the necessary AWS SDK libraries:

npx @nestjs/cli new event-sourcing-project
cd event-sourcing-project
npm install @nestjs/config uuid dotenv @aws-sdk/client-sns @aws-sdk/client-sqs

Configuring AWS Credentials

I often use a .env file for credentials and configuration:

AWS_ACCESS_KEY_ID=<your-access-key-id>
AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
AWS_REGION=<your-region>
SNS_TOPIC_ARN=<your-sns-topic-arn>
SQS_QUEUE_URL=<your-sqs-queue-url>

Implementing SNS Publisher

In NestJS, I create a dedicated service for publishing domain events to SNS. This service is responsible for taking an event payload, converting it to JSON, and submitting it to the SNS topic:

import { PublishCommand, SNSClient } from '@aws-sdk/client-sns';
import { Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';

@Injectable()
export class SnsPublisherService {
  private snsClient: SNSClient;
  private topicArn: string;

  constructor(private configService: ConfigService) {
    this.snsClient = new SNSClient({
      region: this.configService.get('AWS_REGION'),
      credentials: {
        accessKeyId: this.configService.get('AWS_ACCESS_KEY_ID'),
        secretAccessKey: this.configService.get('AWS_SECRET_ACCESS_KEY'),
      },
    });
    this.topicArn = this.configService.get('SNS_TOPIC_ARN');
  }

  async publishEvent(event: any) {
    const command = new PublishCommand({
      TopicArn: this.topicArn,
      Message: JSON.stringify(event),
    });
    return this.snsClient.send(command);
  }
}

Whenever I call publishEvent here, it dispatches the message to my SNS topic, making it accessible to all subscribing services.

Implementing SQS Consumer

Next, I build a corresponding SQS consumer that checks the queue for new events and processes them. If you prefer serverless, you could also configure an AWS Lambda function to automatically trigger when a new message arrives in the queue.

import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } from '@aws-sdk/client-sqs';
import { Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';

@Injectable()
export class SqsConsumerService {
  private sqsClient: SQSClient;
  private queueUrl: string;

  constructor(private configService: ConfigService) {
    this.sqsClient = new SQSClient({
      region: this.configService.get('AWS_REGION'),
      credentials: {
        accessKeyId: this.configService.get('AWS_ACCESS_KEY_ID'),
        secretAccessKey: this.configService.get('AWS_SECRET_ACCESS_KEY'),
      },
    });
    this.queueUrl = this.configService.get('SQS_QUEUE_URL');
  }

  async receiveMessage() {
    const command = new ReceiveMessageCommand({
      QueueUrl: this.queueUrl,
      MaxNumberOfMessages: 1,
    });
    const response = await this.sqsClient.send(command);
    if (response.Messages && response.Messages.length > 0) {
      const message = response.Messages[0];
      await this.deleteMessage(message.ReceiptHandle);
      return JSON.parse(message.Body as string);
    }
    return null;
  }

  private async deleteMessage(receiptHandle?: string) {
    if (!receiptHandle) return;
    const command = new DeleteMessageCommand({
      QueueUrl: this.queueUrl,
      ReceiptHandle: receiptHandle,
    });
    await this.sqsClient.send(command);
  }
}

By separating publisher and consumer logic, I can independently scale or modify the write side (publishing events) and read side (consuming events).

Integrating with DDD and CQRS

Event sourcing works beautifully with Domain-Driven Design (DDD) and Command Query Responsibility Segregation (CQRS):

Domain Events: Each meaningful change in the domain is captured as an event. For instance, when a user is created or a payment is processed, that action spawns a domain event with a specific name (e.g.,UserCreated, PaymentCompleted).
Command Handlers: I encapsulate write operations in explicit commands (createUser, processPayment, etc.). If these commands succeed, domain events get published to SNS. This ensures my domain logic is explicit and traceable.
Event Handlers: Other microservices or consumers then react to these domain events and update their own read models or trigger side effects.
Read Models: Because reading from an event-sourced system can be slow if you only rely on replay for every query, you typically maintain separate read models that are updated asynchronously. This allows the read side to be optimized for queries, while the write side focuses on domain integrity.

Conclusion

By leveraging AWS services like SNS and SQS, along with NodeJS, NestJS, and TypeScript, I’ve found an elegant way to implement event sourcing and build scalable microservices. The event-driven approach, combined with the principles of DDD and CQRS, provides many benefits:

Preserved history and an auditable log of changes
Glitch-resistant scaling of reads and writes
The flexibility to replay and rebuild state anytime

Of course, adopting this approach adds complexity, especially around data consistency and eventual consistency. But as I see it, the payoff in clarity, scalability, and future-proofing is well worth it. As you embark on your event sourcing journey, remember that the key to success lies in understanding your domain, carefully designing events, and ensuring robust error handling and idempotency in your consumers. ^[1][3][2]