In this blog post, I show you how to organize a repository, so that developing and maintaining applications (AWS Lambda) and infrastructure (AWS CDK) can be done in a modular way. This increases extensibility, maintainability and reduces the risk of dependency hell.

This setup even allows you to transpile (build), test, and deploy all modules within the repo with simple commands from the repo's root directory!

TL;DR: Use this template repository to set up the monorepo. Whenever you change a dependency in a package.json file, or whenever you change source code within a custom library, just run from the repo's root:

yarn install
yarn workspaces run build

And you're good to test / deploy!

If you want to know the details, keep on reading...

Intro

In this blog post, I describe the implementation and some technical details of the monorepo setup and structure. The first part is about the folder structure and how dependencies are being resolved by making use of Yarn workspaces. The second part is about configuring Jest for testing. Jest is entirely optional and is not required to make the repo successfully deploy lambdas or any infrastructure.

Example links within this blog link to files in my template repository on Github. These links are an easy way to directly inspect how some parts of the configuration were implemented.

Table of Contents

• Minimal setup

  • Main structure

  • Lambda Package Structure

  • Custom Libraries

    • Why use Custom Libraries?

    • Custom library structure

    • How to Version Custom Libraries

  • Putting it all together

    • Yarn workspaces

    • Reference Custom Libraries in package.json

    • Typescript Peculiarities

• Adding Tests

  • Project-wide tests

  • Package-specific unit tests

• Some technical details

  • Prerequisites

  • How do workspaces resolve dependencies?

Minimal setup

Main structure

Below is the general structure of the monorepo depicted. I want to highlight 3 folders in this folder tree:

.
├── cdk
│   ├── bin
│   │   ├── ...
│   ├── lib
│   ├── main.ts
├── lambdas
│   ├── lambda-a
│   ├── lambda-b
│   ├── shared
│   │   ├── custom-lib-a
│   │   ├── custom-lib-b
├── cdk.json
├── test
│   ├── ...
├── package.json
├── tsconfig.json
├── jest.config.ts
├── yarn.lock

• cdk: all infra defined as CDK code is placed here. It adheres to the typical folder structure if cdk init was to be run.

• lambdas: all Lambda's runtime code will reside here. Every folder within this folder contains all code and configuration to define a single lambda. (Apart from infra and deployment-related characterstics, which is defined with CDK code)

  • The shared folder does not represent a lambda. This folder contains custom libraries that can be used as runtime dependencies within any lambda. This is further described in the section Custom Libraries.

• test: project-wide tests will be placed in this folder. Examples are integration tests, infrastructure-related tests, etc. Note that unit tests for runtime logic of lambdas or custom libraries should not be placed here. Those should be located within the lambda and custom library packages respectively.

Lambda Package Structure

A Lambda's runtime logic and dependencies are contained within the lambda-package folder (example). This folder contains a package.json, tsconfig.json, jest.config.ts, and yarn.lock file as depicted below. The yarn.lock file is used by the NodeJSFunction construct (example) to bundle the required dependencies.

lambda-folder
├── src
|   ├── ...
├── test
├── package.json
├── tsconfig.json
├── jest.config.ts
├── yarn.lock

Custom Libraries

I define a 'custom library' within this blog's context as follows: 'A library contained as a package which provides functionality to multiple lambdas within this monorepo'.

Why use Custom Libraries?

Let's say we have multiple lambdas that interact with AWS SQS. You will likely end up writing some abstractions for interacting with SQS in the form of functions or classes. These abstractions are probably useful for every lambda that interacts with SQS. Hence, we want to be able to import that code from within those lambda packages.

This could be achieved by creating a plane typescript module (ts file) located somewhere in the repo, containing these abstractions. Let's call that kind of files helper-modules. This approach leads to the following question; where are you going to put the package.json containing the dependencies of this file? There are broadly speaking two options:

The wrong way: Define the dependencies at the root level of the repo (project's root package.json). This approach has the following consequences:

• When you keep adding helper modules, new external dependencies of these modules will be added to the root of the repo. The number of dependencies in the package.json file will grow over time.

• The lambdas will need to reference this root package.json as well, leading to:

  • Lambdas are bundled with possibly much more external libraries than needed

  • Dependency conflicts arise because of different versions of an external library being needed by separate lambdas

    • Custom library A needs External library Z version 1, while custom library B needs external library Z version 2.
      Hello dependency hell :(

The right (and a bit more involved) way: Define all helper functionality in custom packages within your monorepo. These packages will define their dependencies in a local package.json within the scope of the package, contrary to putting them all in the root package.json. Limit the scope of a single custom library by keeping the number of external dependencies that the custom library needs small. This approach has the following benefits:

• The helper functionality is packaged together with its external dependencies:

  • Each lambda can be bundled with just the custom libraries that it needs; thereby only bundling external dependencies of those custom libraries, contrary to bundling all repo's external libraries.

• A custom library can contain its own unit tests, reducing complexity and scope creeps of otherwise centrally defined unit tests

• It becomes very easy to publish these custom packages to a package-repository

  • If your team ends up implementing some very useful libraries, they can be shared with other teams.

  • Take the entire custom library, and put it in a new repo. Now this custom library becomes an external library for your monorepo. But there was no refactoring needed of the code within the library at all!

• The chance of dependency conflicts is lowered, increasing the extensibility and maintainability of your project!

Custom library structure

Following the right approach, all custom libraries should follow the structure shown below (example).

. custom-libary-name
├── src
|   ├── index.ts
|   ├── ...
├── test
|   ├── unit.test.ts
├── package.json
├── tsconfig.json
├── jest.config.ts
├── yarn.lock
├── dist # will be generated by running `yarn workspaces run build`

• The index.ts file contains export statements of any typescript object that is meant to be used within code that consumes this custom library

• The test folder contains unit tests for this custom library

• The package.json contains the name of this custom library, and external dependencies that this custom library uses. Besides:

  • It contains a script that instructs this package to transpile typescript into javascript code (to the dist folder) when yarn workspaces run build is run from the root of the repo.

• The tsconfig.json file contains typescript-specific configuration

  • Optionally: References to other custom libraries within this repo. This is needed to make VSCode (or another IDE) work correctly, and be able to give typing hints etc.

• The jest.config.ts file (optional) if you include unit tests in this package

How to Version Custom Libraries

It's very simple: We don't need versioning!

As long as these custom libraries are only used within this monorepo.

Whenever you change a custom library or some package that consumes this library, you run all tests. So if your changes break something, you just fix the code until it no longer breaks. Hence, no versioning is needed. Whenever APIs of packages change, the entire impact of these changes is known and resolved before the commit is made.

Putting it all together

Yarn workspaces

We make this all work together by making use of Yarn Workspaces.

We simply enable workspaces by defining paths to folders within the root package.json (example). You can make use of wildcards in these paths to denote that each subfolder of a certain folder is a workspace.

Yarn will resolve the dependencies for all workspaces when you run yarn install. It will install every dependency in one of the node_modules folders within this repo. Check How do workspaces resolve dependencies? for details on this process.

Reference Custom Libraries in package.json

Since we defined the custom libraries as 'real' packages, we should add them to package.json files (example) to be able to use them at runtime. We do that by using the Custom Library's package name (so not its path!). The version has to be a wildcard (*).

Typescript Peculiarities

Typescript adds quite some additional configuration (and complexity, sadly) to make this all work correctly:

• tsconfig.json files (example) need:

  • a paths property (under compilerOptions) that maps the custom libraries that are used within this config's package. E.g. "custom-library-name": ["../path-to-custom-library"]

  • a references property that points to custom libraries that are used within this config's package

Adding Tests

For testing, we make use of the Jest library.

We will use the Projects feature of Jest, which allows us to run all tests in the entire repo with a few commands from the root of the repo.

A Jest Project is characterized by a folder that contains <testname>.test.ts files and a jest.config.ts (example) file.

A Jest Project's rootDir property should point to the repo's root jest.config.ts file. Every project should also contain a displayName property. This name is used to specify in commands which Jest projects to run and which to ignore.

The repo's root jest.config.ts (example) file should point to all Jest projects that you want to be evaluated when running jest commands from the repo's root dir.

When this all has been set up, the following Jest commands can be run from the repo's root:

npx jest # runs all projects with the configurations from jest.config.ts files
npx jest --selectProjects <jest project name> # only runs these projects
npx jest --ignoreProjects <jest project name> # runs all Jest projects except these

Project-wide tests

Some tests verify structures / behaviour that cannot be limited to the scope a single lambda or library. These tests can be placed in the <root-folder>/test folder. Just make subfolders for any aspect you want to group tests on, e.g. 'integration', 'infra', etc.

Package-specific unit tests

Both the custom libraries and the Lambdas should define their unit tests within their package's test directory. This makes sure that the test-dependencies and the unit tests are located in the same package space (Example).

Some technical details

Prerequisites

To install libraries and transpile your Custom Libraries:

• Yarn

• tsc (typescript compiler)

To be able to deploy the template repo:

• CDK cli installed

• AWS account

  • Bootstrapped with CDK toolkit

How do workspaces resolve dependencies?

If all project's workspace packages allow a single version of an external library to be used, Yarn will only install that version of the external library in the project root node_modules folder. However, if there are conflicting versions required of that external library by different Workspace packages in this repo, Yarn will have to resolve these differences. It will do this in the following way.

Version requirements of an external dependency in two custom libraries:

custom-lib-a --> sqs-lib version == 1.0
custom-lib-b --> sqs-lib version == 2.0

This will be resolved by Yarn with installations as shown below:

root/node_modules --> sqs-lib version 1.0
custom-lib-b/node_modules --> sqs-lib version 2.0

However, if a new custom library is introduced, and shares the same version with another custom library, Yarn will resolve the installation so that a minimum number of installations is required in node_modules folders.

If we change the requirements stated above to:

custom-lib-a --> sqs-lib version == 1.0
custom-lib-b --> sqs-lib version == 2.0
custom-lib-c --> sqs-lib version == 2.0

Yarn will change the installations resulting in:

root/node_modules --> sqs-lib version 2.0
custom-lib-a/node_modules --> sqs-lib version 1.0

In this way, only 2 installations of the sqs-lib are needed, while if the version of sqs-lib in the root node_modules folder was kept the same, sqs-lib version 1 would have to be installed in two custom library node_modules folders. Resulting in more installations than necessary.