Introduction
A reasonably common pattern in AWS is subscribing your SQS queue to your SNS topic.
It is possible to directly use a Lambda to add to the SQS queue, but we will be using SNS for the other benefits it brings. For example, SNS could allow us to send out an immediate acknowledgement e-mail using SES, while also adding the message to a queue for processing later.
During this example exercise we will look at creating a simple Lambda, publishing the data to SNS and then into SQS for later processing.
Example Golang Lambda
First let’s create our Lambda that will publish messages to an SNS topic.
The SNS topic will be defined later as an environment variable call TopicArn.
The SNS message will be definted as a Json input event while calling the lambda, for example:
{
"SnsMessage": "My demo SNS message"
}
To get started create a file called main.go:
package main
import (
"context"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/sns"
"log"
"os"
)
type JsonInput struct {
SnsMessage string `json:"SnsMessage"`
}
func handler(ctx context.Context, event JsonInput) {
sess := session.New(&aws.Config{})
svc := sns.New(sess)
log.Println("Attempting to publish SNS message:", event.SnsMessage)
result, err := svc.Publish(&sns.PublishInput{
Message: aws.String(event.SnsMessage),
TopicArn: aws.String(os.Getenv("TopicArn")),
})
if err != nil {
log.Println("Unable to publish to SNS topic", err.Error())
log.Fatal(err.Error())
}
log.Println("SNS Message published. Message ID:", *result.MessageId)
}
func main() {
lambda.Start(handler)
}
Then we will run these CLI commands to create our .zip file:
# Initialise our golang project
go mod init example.com/demo
go get github.com/aws/aws-lambda-go/lambda
go get github.com/aws/aws-sdk-go/aws/session
go get github.com/aws/aws-sdk-go/service/sns
# If you are on a mac, let Go know you want linux
export GOOS=linux
export GOARCH=amd64
export CGO_ENABLED=0
# Build our lambda
go build -o hello
# Zip up our binary ready for terraform
zip -r function.zip hello
If this worked then you should now have a file created called function.zip that can be used to create your lambda in the next step.
Terraform
Now for the infrastructure! This terraform snippet will configure the Lambda, SNS and SQS.
Create a file called main.tf and add this content:
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
# Configure the AWS Provider
provider "aws" {
region = "eu-west-1"
}
# Create out lambda, using our locally sourced zip file
resource "aws_lambda_function" "demo_lambda_to_sns" {
function_name = "demo-lambda-to-sns"
role = aws_iam_role.demo_lambda_role.arn
package_type = "Zip"
handler = "hello"
runtime = "go1.x"
filename = "function.zip"
source_code_hash = filebase64sha256("function.zip")
depends_on = [
aws_iam_role.demo_lambda_role
]
environment {
variables = {
TopicArn = aws_sns_topic.demo_sns_topic.arn
}
}
tags = {
Name = "Demo Lambda to SNS"
}
}
# Lambda IAM Role
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}
locals {
account_id = data.aws_caller_identity.current.account_id
}
# Lambda IAM Role
resource "aws_iam_role" "demo_lambda_role" {
name = "demo-lambda-role"
assume_role_policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Action" : "sts:AssumeRole",
"Principal" : {
"Service" : "lambda.amazonaws.com"
},
"Effect" : "Allow"
}
]
})
inline_policy {
name = "demo-lambda-policies"
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : "logs:CreateLogGroup",
"Resource" : "arn:aws:logs:${data.aws_region.current.name}:${local.account_id}:*"
},
{
"Effect" : "Allow",
"Action" : [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource" : [
"arn:aws:logs:${data.aws_region.current.name}:${local.account_id}:log-group:/aws/lambda/*:*"
]
},
{
"Effect" : "Allow",
"Action" : "sns:Publish",
"Resource" : aws_sns_topic.demo_sns_topic.arn
}
]
})
}
}
# SQS
resource "aws_sqs_queue" "demo_queue" {
name = "demo-queue"
sqs_managed_sse_enabled = true
}
# SNS
resource "aws_sns_topic" "demo_sns_topic" {
name = "user-updates-topic"
}
# Subscribe SQS to SNS
resource "aws_sns_topic_subscription" "demo_queue_target" {
topic_arn = aws_sns_topic.demo_sns_topic.arn
protocol = "sqs"
endpoint = aws_sqs_queue.demo_queue.arn
}
# SQS Queue Policy
resource "aws_sqs_queue_policy" "demo_queue_policy" {
queue_url = aws_sqs_queue.demo_queue.id
policy = jsonencode({
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Sid": "First",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": aws_sqs_queue.demo_queue.arn,
"Condition": {
"ArnEquals": {
"aws:SourceArn": aws_sns_topic.demo_sns_topic.arn
}
}
}
]
})
}
NOTE: This example doesn’t use encryption on the SNS Topic, but if you care about Data Protection then there is a good chance you will want to add that too.
Deploy Terraform
First, run
terraform plan
Now, once we’re confident we can run:
terraform apply
And that’s it! You should now have a Lambda -> SNS Topic -> SQS Queue deployed.
Testing Lambda
This lambda requires one event input parameter, for example:
{
"SnsMessage": "Here is a sample message."
}
You can either execute this from the CLI, or find the lambda in the AWS console and create a test event:
Click “Test” and you should now see a success message!
Summary
And that’s it! The infrastructure is ready, and you should now be able to publish SNS messages from your Lambda. These messages will then flow through to your SQS queue ready to be processed later.
While it would be possible to write directly to SQS from the lambda, there are many benefits of using SNS as an intermediary step which we will explore in our next article.
Cleanup
If you were just experimenting, remember you can destroy the resources when you’re done by running:
terraform destroy
While these serverless components are relatively cheap, it is always good to keep your account clean to avoid any unnecessary billing.