Automating Lambda modules deployment with GitLab CI

Photo by Sai Kiran Anagani on Unsplash

While working with terraform lambda modules, I had a hard time finding out the best repository architecture to automate my lambdas deployment. I couldn’t find any article that I could use as a guideline, that’s why I’m writing this article.

While working on big projects, project organization is a must-have. In this story, you will be presented to one way to organize your repositories so that it facilitates the deployment procedure, making it much more scalable.

In this article, we'll be building the following repositories architecture:

• Lambda-Module: Repository containing the Terraform Lambda module.
• Lambda-Infra: Repository containing Terraform code for deployment into AWS.
• Lambda-Code: Repository containing the lambda code for deployment using Gitlab-CI.

Lambda Module

As defined by HashiCorp:

A module is a container for multiple resources that are used together.

Using terraform modules solve problems like organize configuration, encapsulate configuration, re-use configuration and it provides consistency and ensure best practice.

We'll be using an AWS lambda module that can be found in here. I won't go into details of how to build a lambda module, since it isn’t the main goal of this article.

Lambda-Infra

Once you have your module ready, you can set up a repository that will centralize all of your lambdas infrastructure. In the example below, each .tf represents a different lambda.

.
├── README.md
├── lambda_alarm.tf
├── lambda_general.tf
├── lambda_sms.tf

With you lambda-module ready, you can use it to build your lambda infrastructure.

As it can be seen on the code below, we are storing our lambda zip file in a s3 bucket that has versioning activated. This is a crucial step, since it will be the version id of our zip file in s3 that will trigger the lambda re-deployment, in case of any change in the code.

In the example above, lambda-sms.zip is our lambda source code that will be defined below.

Lambda Code

In order to automate the lambda deployment, it will be necessary to zip your lambda code and put it into your s3 bucket.

First, let’s check our lambda code example.

.
├── README.md
└── .gitlab-ci.yml
└── lambda
    ├── config.py
    ├── database.py
    ├── handler.py
    ├── package
    ├── queries
    │   └── save_data.sql
    └── requirements.txt

Our CI will be responsible for:

• Running Flake8 in order to check for syntax error and linting conventions.
• Installing the lambda dependencies defined at requirements.txt
• Zipping our code
• Moving it to AWS s3 bucket using AWS CLI

Following The Principle of Least Privileges in AWS, you should create an IAM user that has only the Put-Object permission:

You are going to store your AWS CLI credentials in the CICD settings for your GitLab project:

Screenshot of GitLab CICD setting

With everything set up, when you push your code to your Master branch your lambda code will be zipped and stored in your S3 bucket.

Conclusion

Let’s go through all the deployment procedure:

• Create your lambda code and push it to your GitLab repository.
• Your code will then be zipped and stored in your versioned s3 Bucket.
• Run the terraform in lambda-infra code.
• Terraform will check for changes in the version id of your lambda source code.
• If it detects that a new .zip file has been uploaded to s3, it will then re-deploy your lambda.

This architecture made my life much easier, giving my code much more scalability. I hope it helps you as it helped me!

References

Modules Overview

Upload Files From Gitlab To S3 Automatically