Move your AWS Lambda functions inside your VPC

Yeah, so I'm not great at reading documentation

I don't think anybody really enjoys reading documentation, most of just want to to "git 'er dun". In that spirit, skipping over paragraphs of florid tech writing and jumping straight to the code snippets is usually enough. Not in this case though

TL;DR;

  • Lambdas moved to your VPC must be in a private subnet (i.e. no internet gateway)
  • Lambdas must use a NAT gateway to have internet access
  • Your NAT gateway must be attached to a public subnet i.e. one that has a an internet gateway, not your private subnet where the Lambdas will live
  • Your private subnet must have a default route to the NAT gateway in the public subnet
  • NAT gateways cost money, $40+ / month at the time of writing.
  • Don't forget to fix your security groups and use endpoints if you can for AWS services

The overly florid explanation

I have been trying for some time to completely nail down public access to my AWS resources but one step just didn't seem to work: moving the Lambda functions at the heart of my web system out of the AWS VPC and into my own. The Lambda functions are not complicated (read from database, read/write to S3, get triggered by DynamoDB or S3, send email via SendGrid etc). However, each attempt at moving them inside my VPC so I could close down the limited public acess I had to the AuroraDB instance would fail. Either the Lambda's could not connect to the database, or they couldn't connect to S3, or they couldn't connect to SendGrid. Pretty clearly I was missing something obvious.

Internet gateway? NAT gateway? NAT instance?

The AWS instructions for doing this sort of thing are pretty clear but each time I tried to follow them, something screwed up. Mostly it was access to SendGrid. I have an external SendGrid subscription - if you subscribe via AWS then accessing it is much easier using an endpoint and you don't need actual internet access. Of course I do need internet access for my Lambdas so I'm stuck with trying to work out how it works.

The most promising (and unfortunately expensive) way to do this, as recommended by AWS, is to set up a NAT gateway. For some reason I can't fathom, your Lambda functions must be in a private subnet. In AWS terminology, a private subnet is one that doesn't have an internet gateway attached to it. Which leaves the conundrum that if your Lambda needs internet access, you need to give that private subnet some way out. In all of the naive attempts at this, I would duly create a NAT gateway and attach it to the private subnet and voila! Nothing worked.

The devil in the details

The trick is to put the NAT gateway on a public subnet (i.e. one that already has an internet gateway), then add a route from your private subnet to the NAT in the public subnet. It sounds obvious in retrospect. In the end, my private subnet has these routes:

I have two endpoints (one for S3, one for DynamoDB), two local routes (ipv4 and ipv6) and the NAT gateway for everything else. This allows internal communcation to the AuroraDB inside the VPC, access to S3/DynamoDB via AWS internal connections and the NAT gateway for talking to SendGrid.

Other considerations: security groups and endpoints

A discussion around security groups is probably out of scope here, but you do need to pay attention to them on AuroraDB to give access to your Lambda functions. Most AWS tutorials seem to use a single security group for everything and that's fine for small setups. The AWS endpoints are great for simple access from your private subnets to other AWS services. See here for a good explanation of how they work. If your Lambda doesn't need internet access, but does need access to AWS services, stick to the endpoints and don't use the NAT gateway and save some money.

It costs extra money

What price security? The NAT gateways have a really bad pricing model that I would normally struggle to justify. In my case I think it will add $40 or $50 to the monthly AWS bill just to have a NAT gateway idling there doing not much. There is apparently a "NAT instance" that is a lot cheaper, but AWS discourage it's use (or even finding it!) and would much prefer you use the NAT gateway. I could have powered this laptop with all the eye-rolling that ensued once I worked that out, but at least I'm resting slightly easier that the AuroraDB isn't quite so open any more.

Popular posts from this blog

Tailscale ate my network (and I love it)

Image
No matter where you go, there you are If you're like me, you travel occasionally. Access to your office from home, or from an airport lounge, or from a hotel room is paramount to being productive for most of us now. While I had a fairly simple Tailscale setup that got me into my test machines in the office, what I really wanted was to integrate my AWS production VPC into that system. It's surprisingly simple. Tailscale AWS instructions What do you want to do that for? In the "before tailscale" times, if I needed to test against the production AWS resources or connect dBeaver for database maintenance, I would edit the security group to add my IP address, do my testing, edit the security group to remove myself. This is as error prone as it sounds. I quite often forgot to remove my IP address from the allowed addresses, a major potential security risk when you are travelling. Tailscale has an extremely nifty way to get around it: if you run up a small ec2 ins...
Read more

Stupid stunts with WSL2 , Python3 and AWS ECS

Image
Hand rolled CMS system gets a previewer The ongoing process of updating the help system on the SmartShepherd inspired two new features: editing of the html in a temporary location and a small web server to check your work before pushing it to production. The surprise? That GitHub desktop will show you a side-by-side view if you change an image file. Can't believe I didn't notice that before. Can you hold my camel for me? Sometimes I get so far down a dead end I don't realise how many issues could be sorted simply by re-thinking something. The initial deployment of the SmartShepherd web server was to an Elastic Beanstalk setup on AWS, mostly because I had no idea what I was doing. Elastic Beanstalk isn't really one product, it's more of an elaborate script that creates systems for you. In my case, it was spinning up a small Windows EC2 instance, installing the .NET Core website and auto-configuring a load balancer to point to that instance. I mean, it's...
Read more

Time machine: Solaris 2.6 on QEMU

Image
You can never go home A big chunk of my professional career in the 1990s was spent with a Sparcstation on my desk. Before that the 68000 Sun2 and Sun3 floated past my sticky fingers but the Sparc machines quickly superseded those. I don't remember too much about which ones: pretty sure there was at least one IPX, one Sparcstation 5, a Sparcstation 10 and then the last one was most likely an Ultra 1 or maybe even an Ultra 5. It was a long time ago. Every now and again I think about how productive those machines were and do a little Ebay digging on how much one would be worth to have around. Except I don't have the room for more machines and the idea of trying to revive 25 year old hardware from that era is well beyond my electronic capabilities. Never fear, QEMU is here. Following these excellent instructions from Adafruit , you too can install Solaris 2.6 and have your very own emulated machine. Note that there is one instruction in there I am not sure you need. A...
Read more