Redshift, Lambda and SecretsManager
Let's look at a standard use-case. We have our Redshift instance, the Lambda that executes our queries and the SecretsManager we use to store our credentials.
Running these resources in a public subnet is straight forward. However, once we place them in a private subnet then we need to consider how to access our secret. SecretsManager exists within the AWS Network, but outside of our VPC.
Let's explore 3 ways to connect our Lambda and Redshift to SecretsManager from within a private subnet.
3 ways to connect to Secrets Manager
Option 1) Use a public subnet
While this is the easiest and cheapest approach, it's also the most insecure approach. Our database is left exposed and traffic can flow freely inside and outside of the VPC.
Option 2) Use a NAT Gateway
Once we move our Lambda into a private subnet then it loses access to the outside world. This means it can no longer access any of the AWS APIs like SecretsManager or RedshiftData.
We can open egress access by using a NAT Gateway to allow our lambda to connect out to the internet again. This also allows our resources to talk to SecretsManager again.
This can be a good option if you already have other code that needs to talk to the outside world. For instance if your Lambda is querying other third party APIs too.
A NAT Gateway is likely the most expensive option, starting at a little over $30/month + data costs. There are cases where this can work out cheaper which we will explore next.
Option 3) Use Endpoint Interfaces
Endpoints help keep everything locked down. They keep traffic to SecretsManager within the AWS Network, without opening up egress access to the wider internet.
There are two different endpoints we may need to configure:
- SecretsManager Endpoint
- RedshiftData Endpoint
The RedshiftData endpoint is optional. If the Lambda and Redshift are in the same subnet then you can execute database queries directly.
It's important to remember RedshiftData has its own endpoint. If you are using the API or SDK then you need to open this too.
Endpoint Vs NAT Gateway Pricing
Endpoint Gateways are free. So if you're connecting to DynamoDB or S3 then the right answer will always be to use the endpoint if you're inside a private subnet. Even if you're using a NAT Gateway then you can bring costs down by using the endpoint in addition.
Endpoint Interfaces cost about a 1/5th of the price of a NAT Gateway. So as a general rule of thumbs, if you have less than 5 endpoint interfaces then this will be a cheaper option.
Lastly, there are times when you just need a NAT Gateway. For instance if you have outbound traffic to a third party service outside of AWS.
Summary
Hopefully this gives a good overview of the different ways we can connect to resources when we're inside a private subnet. Please checkout this article here if you would like to see a hands on example of setting up a private subnet.