Terraforming your Golang Webhook (Part 1)

As a DevOps engineer the last thing you should ever do is give yourself an extra server to manage.

SAAS platforms are exploding in popularity. Infact, my favourite part of BigCommerce is not having to maintain servers.

In this article we will look at how to host a serverless golang webhook by using Terraform. This will leave us to focus all our energy on adding value to the business while leaving all the grunt work to AWS.

The finished architecture will look like this:

Prerequisites

The usual prerequisites apply, you will need to configure golang, terraform and AWS Cli.

Setup Golang

Setup Terraform

Setup AWS Cli

Step 1) Install golang dependencies:

go mod init rhuaridh.co.uk/example
go get github.com/aws/aws-lambda-go/lambda

Step 2) Create main.go

To begin, we will create a very basic script that replies a 200 status code and "Hello World":

package main

import (
    "context"
    "github.com/aws/aws-lambda-go/lambda"
)

type MyEvent struct {
    Name string `json:"name"`
}

type Response struct {
    StatusCode int `json:"statusCode"`
    Headers map[string]string `json:"headers"`
    Body string `json:"body"`
}

func HandleRequest(ctx context.Context, name MyEvent) (Response, error) {
    response := Response{
        StatusCode: 200,
        Headers: map[string]string{"Content-Type": "application/json"},
        Body: "Hello world!",
    }

    return response, nil
}

func main() {
    lambda.Start(HandleRequest)
}

Step 3) Set up AWS user for Terraform

You will need to create an IAM role, which is out of scope for this. If you're using Linux, a helpful tip would be to create a second AWS profile and switch to it using:

export AWS_PROFILE=bootstrap-golang-webhooks

Step 4) Set up Terraform

Getting started with Terraform:

terraform init

You should see output similar to:

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.25.0...
- Installed hashicorp/aws v3.25.0 (signed by HashiCorp)

...

Step 4) Create s3 bucket to hold your code deployments

To pre-empt a chicken and an egg scenario, you must first create the bucket manually and deploy it once manually. Replacing the bucket name with your unique entry.

aws s3api create-bucket --bucket=serverless-example-golang-webhooks-YOURNAME --region=eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o main main.go
zip -r function.zip main
aws s3 cp function.zip s3://serverless-example-golang-webhooks-YOURNAME/function.zip

Step 5) Create lambda.tf

This file will contain our entire infrastructure

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

variable "bucket_name" {
  type        = string
  description = "Create a unique bucket name to you, for example: serverless-example-golang-webhooks-YOURNAME"
  default     = "serverless-example-golang-webhooks"
}

variable "aws_config" {
  type = map(string)
  default = {
    region = "eu-west-1"
    function_name = "ServerlessExample"
    description = "Terraform Serverless Application Example"
    stage_name = "test"
  }
}

provider "aws" {
  region = var.aws_config.region
}

resource "aws_s3_bucket" "serverless-example" {
  bucket = var.bucket_name
  acl    = "private"
}

resource "aws_lambda_function" "example" {
  function_name = var.aws_config.function_name

  s3_bucket = var.bucket_name
  s3_key    = "function.zip"

  # "main" is the filename within the zip file to run
  handler = "main"
  runtime = "go1.x"

  role = aws_iam_role.lambda_exec.arn
}

  # IAM role which dictates what other AWS services the Lambda function
  # may access.
resource "aws_iam_role" "lambda_exec" {
  name = "serverless_example_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF

}

resource "aws_api_gateway_rest_api" "example" {
  name        = var.aws_config.function_name
  description = var.aws_config.description
}

resource "aws_api_gateway_resource" "proxy" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  parent_id   = aws_api_gateway_rest_api.example.root_resource_id
  path_part   = "{proxy+}"
}

resource "aws_api_gateway_method" "proxy" {
  rest_api_id   = aws_api_gateway_rest_api.example.id
  resource_id   = aws_api_gateway_resource.proxy.id
  http_method   = "ANY"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "lambda" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  resource_id = aws_api_gateway_method.proxy.resource_id
  http_method = aws_api_gateway_method.proxy.http_method

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.example.invoke_arn
}

resource "aws_api_gateway_method" "proxy_root" {
  rest_api_id   = aws_api_gateway_rest_api.example.id
  resource_id   = aws_api_gateway_rest_api.example.root_resource_id
  http_method   = "ANY"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "lambda_root" {
  rest_api_id = aws_api_gateway_rest_api.example.id
  resource_id = aws_api_gateway_method.proxy_root.resource_id
  http_method = aws_api_gateway_method.proxy_root.http_method

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.example.invoke_arn
}

resource "aws_api_gateway_deployment" "example" {
  depends_on = [
    aws_api_gateway_integration.lambda,
    aws_api_gateway_integration.lambda_root,
  ]

  rest_api_id = aws_api_gateway_rest_api.example.id
  stage_name  = var.aws_config.stage_name
}

resource "aws_lambda_permission" "apigw" {
  statement_id  = "AllowAPIGatewayInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.example.function_name
  principal     = "apigateway.amazonaws.com"

  # The "/*/*" portion grants access from any method on any resource
  # within the API Gateway REST API.
  source_arn = "${aws_api_gateway_rest_api.example.execution_arn}/*/*"
}
output "base_url" {
  value = aws_api_gateway_deployment.example.invoke_url
}
output "bucket_name" {
  value = var.bucket_name
}

Bucket names need to be unique, so you can now edit lambda.tf and replace the default value with something unique to you

variable "bucket_name" {
  type = string
  description = "Create a unique bucket name to you, for example: serverless-example-golang-webhooks-YOURNAME"
  default = "serverless-example-golang-webhooks"
}

After this is done, you can now run this to build your infrastructure in AWS

terrafrom apply

One quirk to be aware of, due to API Gateway's staged deployment model, if you do need to make changes to the API Gateway configuration you must explicitly request that it be re-deployed by "tainting" the deployment resource:

terraform taint aws_api_gateway_deployment.example

Step 6) Build & Deploy

Now you have already deployed this once before manually, but to build and deploy this going forward you can run:

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o main main.go
zip -r function.zip main
aws s3 cp function.zip s3://$(terraform output -raw bucket_name)/function.zip
aws lambda update-function-code --function-name "ServerlessExample" --s3-bucket="$(terraform output -raw bucket_name)" --s3-key="function.zip" --region="eu-west-1"

To test the lambda from the CLI:

$ aws lambda invoke --region=us-east-1 --function-name=ServerlessExample output.txt
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

To test from a browser, execute this from the CLI to get the URL.

terraform output

BigCommerce/Shopify Webhooks

This architecture is perfect for small webhook builds. This gives a low cost, low maintenance and highly available infrastructure.

You can use this approach with any SAAS platform that supports webhooks. I use them often on BigCommerce and Shopify builds.

Here is a real world example of how I have used this in production:

And the best part? Being able to tell the client it will literally cost them pennies to run.

Click here to move onto the next part where we discuss how to add continous deployment to the project in Gitlab