Lambda Terraform

In a previous post I wrote about how to deploy an application in a container. The gist of the post talked about how containers provide a layer of abstraction so that your applications can run on any machine. This is great for local development and large scale applications. However, in order for them to run in a production environment they more than likely are required to run on some sort of runtime or server which can provide some additional cost overhead.

So what if you don’t want to manage expensive infrastructure and just want to run the application without worrying about the underlying servers that run and pay as you go? Sounds like a FaaS.

AWS provides a FaaS solution called Lambda. Recently AWS added support for the Go runtime and is a great option to running low cost functions. Setting up a Lambda can be pretty straight forward if you follow their guides but after you’ve done the manual process more than once you quickly realize that it does not scale. To do effective development all of your infrastructure should be handled in a way that makes it repeatable so time isn’t wasted deploying these applications manually. It would be nice to define the infrastructure in code so you can easily and repeatedly deploy your application.

To do this we will write code to help us build our infrastructure. Throughout the DevOps movement there is a mantra that states, “infrastructure as code”. Why? Well it allows for you to iterate, debug, rebuild infrastructure from a defined file. This brings visibility to what the architecture should look like and also removes manual (and therefore error prone) processes.

Terraform

A great tool that helps you define “infrastructure as code” is Terraform. Terraform is an open source product started by HashiCorp that allows you to write simple commands to build architecture on several different cloud platforms. In this example we will be using AWS as our cloud provider but there are plenty of other modules for GCP and Azure.

I won’t go into details about what all Terraform can do but I suggest you take a look at Yevgeniy Brikman’s book: Terraform Up and Running.

Install

To install for your operating system please follow the steps on the website. For this example you will also need an AWS Account with aws cli installed.

Example Application Structure

Our application is going to run on AWS Lambda with an externally hosted Redis Instance. This is because AWS requires you to run databases (other than Dynamo) within a Virtual Private Cloud (VPC) for security reasons. I felt that the scale of writing an application like that would be too confusing so I thought I would focus on just the Lambda Function in this post and use an external option for the database.

We will be using our Gira Project which has been built to run in both a server and a FaaS. Please review the code in the main.go file to see any sort of changes. This function will require you to pass in the Redis URL and password as environment variables to connect to our external Redis cluster.

Also we are going to plan to have this run in two environments (QA and Production) and so we will have two deployed instances when we are done. However we will share the same database. This again isn’t ideal but we will be doing it for simplicity’s sake.

All code can be found here

Build Script

First we will create a build script that can be used to build and upload the binary to S3 to be used by our Terraform application. AWS Lambda allows Zip files to be uploaded and then unpacked as part of its deployment process. We will compile our Go code into a binary, then pack it in a zip file, and then upload it to S3 as the version number of the Lambda we are hoping to deploy.

#!/usr/bin/env bash
TAG=$(git describe --abbrev=0 --tags)

GOOS=linux go build -o main main.go
zip main.zip main

aws s3 cp main.zip s3://hex-lambda/$TAG/main.zip

What we are doing here is reading the most recent tag from our git history and uploading to a directory of that name on S3. This will be the version number in the module that you’ll see later.

Module

First we are going to build a module. A module is reusable piece of code that defines some architecture that will be reused. The best way to think about this is a collection of objects that need created specifically to run a product and will need to be deployed the same way across different environments. In Object Oriented Terms think of it kind of like a class. You don’t want it to be too big in case you need to change small bits but don’t want it to be too small because it will be complex.

Let’s start by creating a directory for all of this code to live in. First we’ll create a parent directory called terraform in it we’ll have modules, qa, and prod directories to separate our code. The modules directory will house the modules we need for our application to be built, and the other directories will actually implement the modules and deploy them depending on the variables we define. In the modules directory we’ll create another directory called api-lambda since we’ll be defining how we want our Lambda function to be deployed.

Before writing the module let’s define some variables that will be used in our module. The variables allow us to customize the module by providing certain input to define what we are deploying. In this case we want to capture a logical name group for the function, where we can find the function source code, the version to deploy, what environment to deploy to, and finally any environmental variables the function may need. We’ll put all of this in a file called variables.tf in our api-lambda directory.

Deployments

By building the module we allow ourselves to have a reusable piece of code that can be applied to multiple environments. This ability to repeat allows for a certain level of stability as you go through your different environments just like your regular code. Each environment should have its own folder and the main.tf files should really be invocations of your module. This allows you to test your infrastructure and make changes to the module before it gets to production.

Conclusion

Our goal is to be able to manage and deploy our applications in a way that is repeatable and maintainable. If we were to build all of these resources manually we would have to make a step by step guide. However, if it is maintained as code we can deploy similar applications or make modifications to our existing resource in a way that we can revert if there is a problem or enhance if we need some additional functionality.

Note: This code was adapted from the example on Terraform’s site which does not use Go nor does it allow for logging and routing to be done by the application.

Avatar
Joel Holmes
Software Developer