AWS Lambda Monorepo

What is a Monorepo

A monorepo is a single repository which stores all the code for multiple independent projects. It can be used to create a single source of truth and enable code reuse between the different functions which are provided.

Why would you use a monorepo?

Once of the problems during the development of individual microservices comes from copy and pasting of code (lazy developer === happy developer). During the development of microservices you will create modules which provide functionality which can be reused, such as data formatting functions or wrappers for external libraries.

Another problem which can occcur in microservices is the management of external dependencies. Having to manage compatible versions between dependencies can become dependency hell. For example, I tend to use axios when developing microservices. If this library requires an update due to a vulnerability then multiple updates, releases and deployments would have to happy to update every service which included this library.

Benefits of a Monorepo

There are a number of benefits to using a monorepo:

Easily reuse code between functions
Improve management of external libraries
Code refactoring made easier
Team collaboration can be improved

Limitations and Disadvantages

As with everything there are some limitations or disadvantages:

Increased complexity of the repository
Deployments (Especially Bitbucket) have limitations
There is a learning curve to how to setup and manage a repo

How to Create a JavaScript Monorepo for AWS Lambda

This is not a definitive how-to guide, this is the process of me learning how to setup a monorepo for AWS Lambda’s which use Node. The information here is what I have learnt from implementing this technique. Some of the steps are self explanatory but where I feel there were decisions made I will try to explain why I made those choices.

The project uses the following:

Node v14.16.0
SAM (AWS Serverless Application Model)
CloudFormation Nested Stacks
Bitbucket Pipelines
Axios

Create a project folder:

mkdir node-monorepo-lambda
cd node-monorepo-lambda
git init

Create the root package.json file:

npm init

Add multiple Lambda projects to separate folders

mkdir -p lambda-a/src
mkdir -p lambda-b/src

Initialise workspace folders to enable shared dependencies

To enable the monorepo to share dependencies it will use npm workspaces.

NPM define workspaces as “Workspaces is a generic term that refers to the set of features in the npm cli that provides support to managing multiple packages from your local files system from within a singular top-level, root package.”

The following commands will create a package.json file in each lambda project which is then referenced by the workspace name in the root package.json file.

npm init -w lambda-a/src
npm init -w lambda-b/src

Create a root CloudFormation Template which will create a nested stack

To deploy the monorepo the decision was made to have one AWS CloudFormation stack. This template uses the SAM transform function to tell AWS to convert from Serverless to Cloudformation

Each Microservice Lambda function would have it’s own CloudFormation template file which will be referenced from the following root template. This enables the functions to be decoupled and allows them to manage their own resources.

The root stack could always reference infrastructure resources which may be required across the whole organisation.

touch template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'

Description: SAM app which has multiple Lambdas which are in a monorepo

Resources:
  LambdaFunctionA:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: lambda-a/template.yaml
  LambdaFunctionB:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: lambda-b/template.yaml

Create AWS SAM Templates for each function

This snippet is taken and modified from the AWS SAM ‘hello-world’ template. It deploys a Lambda function using Node v14 and an API Gateway which has a GET endpoint at '/hello'

The output of the template will be the fully qualified URL for the endpoint which will return a simple JSON structure.

touch lambda-a/template.yaml
touch lambda-b/template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'

Description: SAM app which has Lambda and an API Gateway

Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.lambdaHandler
      Runtime: nodejs14.x
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

Outputs:
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"

Create Lambda Handler Files

These handler files are reference in the SAM Template. Any calls to the endpoint will trigger this function. A call is made to the 'http://checkip.amazonaws.com/' endpoint which returns the callers IP address. This data is passed to the shared package createresponse which will return a formatted response block which can then be passed back to the API Gateway.

touch lambda-a/src/app.js
touch lambda-b/src/app.js

const axios = require('axios');
const { createResponse } = require('createresponse');
const url = 'http://checkip.amazonaws.com/';
let response;

exports.lambdaHandler = async (event, context) => {
    try {
      const ret = await axios(url);
      response = createResponse(200, {
        message: "hello world",
        location: ret.data.trim(),
      })
    } catch (err) {
        console.log(err);
        return err;
    }

    return response
};

Install Axios as a Dependency of each Lambda

The function code above references axios as a dependency, to install this as a dependency of each function the following command with a -w flag targets the workspace.

npm install axios -w lambda-a
npm install axios -w lambda-b

Using a shared module in both Lambdas

As both Lambda functions can use the same createResponse function, we can add a packages folder which contains shared modules. These are added to the monorepo in the same way that the functions are created as workspaces. The use of the packages folder is used to separate these shared modules from the lambda functions.

mkdir -p packages/create-response
npm init -w packages/create-response
touch packages/create-response/index.js

Add the following code which is a helper function to create a response object, this is a very basic function which takes a status and body argument. An object is returned to the caller.

function createResponse(status, body) {
  return {
    statusCode: status,
    body: JSON.stringify(body),
  };
}

module.exports = {
  createResponse,
};

Add the package as a dependency of each of the Lambda’s by adding the createresponse module referenced by its file path.

"dependencies": {
  "axios": "^0.26.1",
  "createresponse": "file:../../packages/create-response"
}

Monorepo Complete

You now have a working monorepo with shared dependencies and a shared module. Once you have installed these functions, if you check in the root node_modules folder you will see that the two lambda functions and the shared createreponse module are symlinked.

You can add devDependencies to the root package.json so that they can be shared amongst each of the functions. Each Lambda should have a test suite written which will share libraries such as mocha, chai or jest. Depending on your chosen testing framework. This means that these dependencies are then managed in one place where they can be kept up to date.

Using Bitbucket Pipelines to Deploy Lambda’s to AWS

Using CI/CD pipelines to deploy code is generally the best practice to get your code from your local environment into the cloud. AWS SAM provides a helpful CLI (Command Line Interface) to deploy SAM projects into your AWS environments.

Bitbucket is a Git repository management solution which offers pipelines. These pipelines provide containerised enviroments which can be triggered upon commits to deploy your solution to the cloud.

One quirk when deploying Lamba’s to AWS using SAM is that the dependencies that are deployed are not always what your are expecting. If you were to use the sam deploy command when you have all your devDependencies installed then they also get deployed which can result in some pretty large functions.

During the initial deployments of the monorepo, the shared packages were failing when using the sam build command. The command installs the dependencies and generates an .aws-build folder, as the symlinks are not generated correctly the deployments would fail or be missing the shared module.

To get round this issue, the following shell script was created which finds all the folders with a /src/ folder. It then loops over those directories and installs the production npm dependencies.

The pipeline uses an AWS image which included the SAM CLI. Before using the CLI the aws configure command is used to set the user authentication for the deployment. Once configured the npm build script is triggered before a sam deploy command is used to deploy the stack to the AWS Cloud.

#!/bin/bash

CURRENT_DIR=$(pwd)

FILES=`find ./*/src -type d`
for dir in $FILES
do
  cd $dir && npm install --production
  cd $CURRENT_DIR
done

image: public.ecr.aws/sam/build-nodejs14.x

pipelines:
  branches:
    master:
      - step:
          name: Build and Package
          script:
            - aws configure set aws_access_key_id $AWS_ACCESS_KEY
            - aws configure set aws_secret_access_key $AWS_SECRET_KEY
            - aws configure set default.region $AWS_DEFAULT_REGION
            - ./lambda-npm.sh 
            - >
              sam deploy --no-confirm-changeset --no-fail-on-empty-changeset 
              --stack-name lambda-monorepo 
              --s3-bucket $DEFAULT_S3_BUCKET 
              --region $AWS_DEFAULT_REGION 
              --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

Thank you for taking the time to read this blog post, if you have any suggestions of thoughts then please reach out. I’ll be happy to discuss any of the decisions. Hopefully as I discover more about npm workspaces and monorepos I will be able share that knowledge here.

AWS Lambda Monorepo – Node.js