In my honest (and truly humble) opinion, VPCs don't make much sense in a serverless architecture -- it's not that they don't add value, it's that the value the add isn't worth the complexity you incur.

After all, you can't log into a lambda function, there are no inward connections allowed. And it isn't a persistent environment, some functions may timeout after just 2-3 seconds. Sure, network level security is still worthy pursuit, but for serverless, tightly managing IAM roles and looking after your software supply chain for vulnerabilities would be better value for your money.

But if you've got a fleet of EC2s already deployed in a VPC, and your Lambda function needs access them. Then you have no choice but to deploy that function in a VPC as well. Or, if your org requires full network logging of all your workloads, then you'll also need VPC (and their flow logs) to comply with such requests.

Don't get me wrong, there is value in having your functions in a VPC, just probably not as much as you think.

Put that aside though, let's dive into the wonderful world of Lambda functions and VPCs

Working Example

First, imagine we deploy a simple VPC with 4 subnets.

1. A Public Subnet with a Nat Gateway inside it.
2. A Private Subnet which routes all traffic through that NAT Gateway
3. A Private Subnet without internet (only local routing)
4. A Private Subnet without internet but with a SSM VPCe inside it

Let's label these subnets (1), (2) ,(3) and (4) for simplicity.

Now we write some Lambda functions, and deploy each of them to each subnet. The functions have an attached security group that allows all outgoing connections, and similarly each subnet has very liberal NACLs that allow incoming and outgoing connections.

Then we create a gateway S3 VPC-endpoint (VPCe), and route subnet (4) to it.

Finally, we enable private DNS on the entire VPC. And then outside the subnet we create a bucket and an System Manager Parameter Store Parameter (AWS really need better terms for these things).

The final network looks like this:

Amazon KMS is one of the most integrated AWS services, but probably also the least understood. Most developers know about it, and what it can do, but never really fully realize the potential of the service. So here's a rundown of the innards of the KMS service.

What is KMS?

KMS (Key Management Service) is an AWS offering that allows us to create, manage and use cryptographic keys. Like everything else in AWS, it's highly available, provided via API, and charged on a per-use basis.

The system, which was introduced on the first day of the 2020 school session yesterday, takes only two seconds to scan a pupil’s face before his personal information, such as full name, pupil number and class, is stored into the school’s database

https://www.nst.com.my/news/nation/2020/01/552737/sk-taman-perling-1-uses-facial-recognition-scanners-mark-pupils

A school in Johor became the first in the country to introduce facial recognition scanners to take pupils attendance -- hopefully it'll be the last.

This year I resolve to support the media that I like, i.e start paying for content I've been consuming for free all this time.

I believe that if we want better media, we need to start paying for it, and it's not a matter of quality content, a free and fair media, is an absolute necessity for democracy to operate -- after all, people can't make informed choices if they aren't informed.

Last week, I launched a new pipeline for Klayers to build Python3.8 Lambda layers in addition to Python3.7. For this, I needed a separate pipeline because not only is it a new runtime, but under the hood this Lambda uses a new Operating System (Amazon Linux 2 vs. Amazon Linux 1)

So I took the opportunity to make things right from an account hierarchy perspective. Klayers for Python3.7 lived in it's own separate account from all my other hobby projects on AWS -- but I kept all stages in it (default, dev and production). [note:Default is an odd-name, but it ties to the Terraform nomenclature]. This afforded some flexibility, but the account felt bloated from the weight of the different deployments -- even though they existed in different regions.

It made no sense to have default and dev on the same account as production -- especially since accounts were free. Having entirely separate accounts for prod & non-prod incurred no cost, and came with the benefit of additional free-tiers and tidier accounts with fewer resources in them -- but the benefits don't stop there.

Had a blast at PyConSG 2019, really cool to be in the presence of so many pythonistas. Would definitely recommend, especially since python is one of the more broadly used languages (AI, Blockchain, RPA, etc).

My talk was on AWS Lambda (naturally!).

Slide deck can be found in this GitHub Repo. Let me know what you think either in the comments here or on Youtube.

Everyone knows that I'm a Lambda fanboy, and to be fair Lambda deserves all the praise it gets, it is **the** gold-standard for serverless functions. But yesterday, I gave Google Cloudrun a spin, and boy(!) is Lambda is going to get a run for its money.

Which is surprising given Google has traditionally lagged in this area -- isn't it quaint that we use words like 'traditional' in the serverless world!

But I digress.

The Lambda equivalent in the Google world, is Google cloud functions ... which is (generously speaking) what lambda was 2 years ago-- pretty boring. The only advantage I saw it having over Lambda, was the ability to build python packages natively in the requirements.txt file. But that incurred a build during deploy, which in turn had a limit.

And while, it did allow for a larger package size (double what AWS Lambda offers) it was severely more complex to understand. Just looking at it's limit and pricing models can make you dizzy.

In short, Google Cloud functions lacked the simplicity of Lambda, with little benefit for incurring all that additional complexity.

But Cloud Run is something else. It's still more complex than lambda, but here the trade-off seems worth it. So let's take a peek at Google's new serverless Golden Boy!

Containers vs. Functions

In Lambda the atomic unit of compute is the function, which for an interpreted language like Python is just plaintext code uploaded to AWS. But in Cloud Run the atomic unit is the container -- and that can be a container for just the one function, or the container for the entire app itself -- with all the routing logic embedded within it.

Now why would you need apps for the serverless world?! You ask indignantly. Aren't these all supposed to be function based?

Well actually lots of people have legacy code written at the application level, and re-writing an entire application takes a long time, and very rarely succeeds on the first try.

Lambda functions are awesome, but they only provide a single dimension to allocate resources - memorySize. The simplicity is refreshing, as lambda functions are complex enough -- but AWS really shouldn't have called it memorySize if it controls CPU as well.

Then again this is the company that gave us Systems Manager Session Manager, so the naming could have been worse (much worse!).

Anyway....I digress.

The memorySize of your lambda function, allocates both memory and CPU in proportion. i.e. twice as much memory gives you twice as much CPU.

The smallest lambda can start with minimum of 128MB of memory, which you can increment in steps of 64MB, all the way to 3008MB (just shy of 3GB).

So far, nothing special.

But, at 1792MB, something wonderful happens -- you get one full vCPU. This is Gospel truth in lambda-land, because AWS documentation says so. In short, a 1792MB lambda function gets 1 vCPU, and a 128MB lambda function gets ~7% of that. (since 128MB is roughly 7% of 1792MB).

Using maths, we realize that at 3008MB, our lambda function is allocated 167% of vCPU.

But what does that 167% vCPU mean?!

I can rationalize anything up to 100%, after all getting 50% vCPU simply means you get the CPU for 50% of the time, and that makes sense up to 100%, but after that things get a bit wonky.

After all, why does having 120% vCPU mean -- do you get 1 full core plus 20% of another? Or do you get 60% of two cores?

At the end of 2018, AWS introduced custom runtimes for Lambda functions, which provided customers a way to run applications written in languages not in the holy list of the 'Official AWS Lambda Runtimes' which include a plethora of languages. It has 3 versions of Python, 2 versions of Node, Ruby, Java, Go and .NET core (that's a lot of language support)

Security-wise, it's better to use an Official AWS Lambda runtime than it is to roll your own. After all, why take ownership for something AWS is already doing for you -- and for free!

But, as plentiful as the official runtime list is-- there're always edge-cases where you'd want to roll your own custom runtime to support applications written in languages AWS doesn't provide.

Maybe you absolutely have to use a Haskell component -- or you need to migrate a c++ implementation to lambda. In these cases, a custom runtime allows you to leverage the power of serverless functions even when their runtimes are not officially supported.

Bash Custom Runtime

Which brings us to the topic of today's post, the bash custom runtime.

For Klayers, I needed a way to update a github repo with a new json file every week -- which can be done in python, but no python package came close to the familiarity of git pull , git add and git commit.

So rather than try to monkey around a python-wrapper of git, I decided to use git directly -- from a shell script -- running in a lambda -- on the bash runtime.

So I pulled in the runtime a github repo I found, and used it for write a lambda function. Simple right? Well not entirely -- running regular shell scripts is easy, but there are some quirks you'll have to learn when you run them in a lambda function...

Not so fast there cowboy...

Firstly, the familiar home directory in ~/ is off-limits in a lambda function -- and I mean off-limits. There is absolutely no-way (that I know off), for you can add files into this directory. Wouldn't be a big isue, except this is where git looks for ssh keys and the known_hosts file.

Next, because lambda functions are ephemeral, you'll need a way to inject your SSH key into the function, so that it can communicate to GitHub your behalf.

Finally, because you've chosen to use the bash runtime, you're limited to the awscli utility, which while fully functional doesn't come with the usual tools as boto3 for python. It's a lot easier to loop and parse json in python than it is in bash -- fortunately, jq makes that less painful, and jq is included in the custom runtime :).

Enough talking let's build this

One of a great things about Lambda functions is that you can't SSH into it.

This sounds like a drawback, but actually it's a great security benefit -- you can't hack what you can't access. Although it's rare to see SSH used as an entry path for attackers these days, it's not uncommon to see organizations lose SSH keys every once in a while. So cutting down SSH access does limit the attack surface of the lambda -- plus the fact, that the lambda doesn't exist on a 24/7 server helps reduce that even further.

Your support engineers might still want to log onto a **server**, but in todays serverless paradigm, this is unnecessary. After all, logs no longer exists in /var/logs they're on cloudwatch, and there is no need to change passwords or purge files because the lambdas recycle themselves after a while anyway. Leave those lambda functions alone will ya!

As a developer, you might want to see what is **in** the lambda function itself -- like what binaries are available (and their versions), or what libraries and environment variables are set. For this, it's far more effective to just log onto a lambci docker container -- Amazon work very closely with lambci to ensure their container matches what's available in a Lambda environment. Just run any of the following

• docker run -ti lambci/lambda:build-python3.7 bash
• docker run -ti lambci/lambda:build-python3.6 bash

Lambci provide a corresponding docker container for all AWS runtimes, they even provide a build image for each runtime, that comes prepackaged with tools like bash, gcc-c++, git and zip. This is the best way to explore a lambda function in interactive mode, and build lambda layers on.

But sometimes you'll find yourself wanting to explore the actual lambda function you ran, like checking if the binary in the lambda layer was packaged correctly, or just seeing if a file was correctly downloaded into /tmp-- local deploy has it's limits, and that's what this post is for.