ProgrammingSoup

Node.js API Project Architecture (with Docker, Tests, and CI/CD)

by Nicklas Envall

In this article, we'll set up a REST API by using the unopinionated web application framework Express. The theme of our API will be artists.

It's a very practical guide intended to expose you to a great number of technologies and keywords, in a concise manner. It's not meant to give you the exact path to a successful architecture, it's about giving you perspective and some of the keys to a successful architecture for your application.

The nodejs-api-project-architecture repository contains the code for each part, so fork it if you get stuck and continue from there.

Part 1. Setting the foundation:

  1. Setting the base
  2. Setting up Docker
  3. Transpilation with Babel
  4. Static testing with ESLint
  5. Testing Framework Jest
  6. Environment variables
  7. Continuous integration (CI)
  8. Continuous delivery (CD)

Part 2. Building the app:

  1. Data Access Object (DAO)
  2. Data Transfer Object (DTO)
  3. Services & Business logic
  4. Controller & Mocking
  5. Middleware
  6. Express Routes

Part 3. Persisting data:

  1. Persisting data with MongoDB
  2. Your team uses PostgreSQL
  3. Starting the server
  4. Review

We'll also cover static testing, unit testing, integration testing, HTTP testing (E2E). So, the project will be well-tested, and we'll be using a TDD approach for the most part.

Part 1. Setting the foundation

Now we'll start by setting up everything before we start coding our application.

1.1 Setting the base

Start with creating a new directory, and set up a project:

mkdir my-express-app
cd my-express-app
npm init -y

As a result, we should now have a manifest file called package.json.

So, continue by installing express:

npm install express

Create /src/server.js, which will act as the project's entry point.

Then, paste the following code into it.

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Listening at http://localhost:${port}`)
})

Ensure everything works by running node src/server.js.

1.2 Setting up Docker

Docker helps us create, deploy, and manage containers. If you want to brush up on your Docker knowledge, then you can read about the Docker Basics here. You can also install docker here. But here's an extract of key concepts:

  • Containers: are isolated instances on a host where we can run our applications.
  • Images: are templates that create containers.
  • Volumes: allows us to persist and share data between containers.

Dockerfile

We'll start by creating a Dockerfile, which contains instructions on how to build our image. It uses the following format:

# Comment
INSTRUCTION arguments

Create Dockerfile and add the following:

FROM node:17-alpine
 
WORKDIR /node

COPY package*.json ./

RUN npm install && npm cache clean --force

WORKDIR /node/app

These instructions (image) help us create a new container with the right dependencies installed. Let's inspect each instruction:

  1. FROM <image>: sets the base image. In this case, node:17-alpine is our base image.
  2. WORKDIR </path/to/workdir>: creates (if non-existent) and sets the working directory. In this case, we create and set /node.
  3. COPY <src> <dest>: copies files or folders from <src> with regards to the build context of your container's path at <dest>. In this case, we're taking package*.json, so we can access our dependencies inside the container.
  4. RUN <command>: will create a new layer and execute the provided commands. In this case, we're installing our dependencies.
  5. WORKDIR </path/to/workdir>: same as step 2, but now we set the working directory /node/app for our app.

Docker Compose

Docker Compose is a tool used for defining, running, and managing a multi-container application. It allows us to run multiple containers with ease. In this article, we'll use it for development, testing, and our CI workflow.

The tool allows us to define services in a YAML-file called docker-compose.yml. Then, we can manage all those defined services with the docker-compose CLI.

Now, create docker-compose.yml:

version: '3.7'

services:
  server:
    build: .
    ports:
      - "3000:3000"
    volumes:
      - "./:/node/app"
      - "exclude:/node/app/node_modules" # prevent host from overwriting node_modules
    command: "npm start"

volumes:
  exclude:

Continue by updating your scripts object in package.json so it matches the following:

"scripts": {
  "start": "node src/server.js"
}

Then run:

docker-compose up server

In summary, with our Dockerfile we are first creating a directory into which we copy our manifest file and install our dependencies. Then we create yet another directory which we later in our docker-compose.yml file we use to bind mount our app's source code into - so our changes are reflected in the container as well during development. We install our dependencies one step above our source code so that our local npm_modules do not overwrite the one inside our container, we also use an empty volume "/node/app/node_modules" to ensure it always goes one step above, i.e. from /node/app to /node.

1.3 Transpilation with Babel

Node.js does support ES modules via the .mjs file extension. But, I presume you still want to use the latest JavaScript, so we'll use babel.

Babel doesn't do anything by default, so we'll need to use plugins. Plugins are just small programs that let Babel know how to transform your code. But keeping track of multiple plugins is a hassle so we can use presets. Presets are a set of carefully picked out plugins.

Now, we'll add three packages to our project:

  • @babel/core contains the core functionality.
  • @babel/cli allows initiation of transpilation via the terminal.
  • @babel/preset-env is a preset for using the latest JavaScript.

Install them by running the following:

npm install --save-dev @babel/cli @babel/core @babel/preset-env

To tell Babel what plugins and presets to use, we need a configuration file.

So, we create a .babelrc file and add the following:

{
  "presets": [
    [
      "@babel/preset-env",
      {
        "targets": {
          "node": "current"
        }
      }
    ]
  ],
  "plugins": []
}

With the targets object, we inform babel what environment our project supports. In this case, we notify it to compile against the node version process.versions.node by using "node": "current".

Next, let us stop using Node's module loader function (require) in server.js and update it to:

import express from 'express'

const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!')
});

app.listen(port, () => {
  console.log(`Listening at http://localhost:${port}`)
});

Update your scripts object in package.json so it matches the following:

"scripts": {
  "build": "babel src -d dist",
  "start": "npm run build && node dist/server.js"
}

Finally, you can run docker-compose build server (our container needs the babel packages) and docker-compose up server to make sure it all works.

1.4 Static testing with ESLint

With static testing, we analyse our code without running it. The benefits are that we may find possible syntax errors, bugs, code styling errors, and optimization possibilities.

Linters are tools used for static analysis, and ESLint is the linter that we'll use. So, install it by running:

npm install --save-dev eslint

We'll also need a configuration file for ESLint. So, run:

npx eslint --init

As a result, you need to answer some questions. I answered them in the following manner:

√ How would you like to use ESLint? · style
√ What type of modules does your project use? · esm
√ Which framework does your project use? · none
√ Does your project use TypeScript? · No / Yes
√ Where does your code run? · node
√ How would you like to define a style for your project? · guide
√ Which style guide do you want to follow? · airbnb
√ What format do you want your config file to be in? · JSON
Checking peerDependencies of eslint-config-airbnb-base@latest
The config that you've selected requires the following dependencies:

eslint-config-airbnb-base@latest eslint@^5.16.0 || ^6.8.0 || ^7.2.0 eslint-plugin-import@^2.22.1
√ Would you like to install them now with npm? · No / Yes

You should now have a generated ESLint config file (mine's called .eslintrc.json).

Continue by adding "root": true so the file looks like:

{
    "env": {
        "es2021": true,
        "node": true
    },
    "extends": [
        "airbnb-base"
    ],
    "parserOptions": {
        "ecmaVersion": 12,
        "sourceType": "module"
    },
    "rules": {
    },
    "root": true
}

Then, update your scripts object in package.json so it matches the following:

"scripts": {
  "test": "eslint src/**/*.js",
  "build": "babel src -d dist",
  "start": "npm run build && node dist/server.js"
},

Finally, run docker-compose build server (our container needs the eslint package), and then docker-compose run server npm test, and you should see something like:

/node/app/src/server.js
   1:30  error    Missing semicolon             semi
   3:22  error    Missing semicolon             semi
   4:18  error    Missing semicolon             semi
   7:27  error    Missing semicolon             semi
  11:3   warning  Unexpected console statement  no-console
  11:67  error    Missing semicolon             semi

✖ 6 problems (5 errors, 1 warning)
  5 errors and 0 warnings potentially fixable with the `--fix` option.

I use semicolons, so to correct the errors, we'll add semicolons.

Some do prefer semicolons and some don't. If you don't then you can read how to configure eslint here.

Update your server.js so it matches the following:

import express from 'express';

const app = express();
const port = 3000;

app.get('/', (req, res) => {
  res.send('Hello World!');
});

app.listen(port, () => {
  console.log(`Listening at http://localhost:${port}`);
});

If you run docker-compose run server npm test again, all the errors should be gone.

1.5 Testing Framework Jest

Previously your developing process might've involved using nodemon and continuously refreshing the browser. But, we'll be using a TDD approach, so we won't be using nodemon.

Start with installing the testing framework Jest.

npm install --save-dev jest

Then update your package.json:

"scripts": {
  "test": "eslint src/**/*.js & jest --passWithNoTests",
  "test:watch": "jest --watch",
  "build": "babel src -d dist --ignore src/**/*.test.js",
  "start": "npm run build && node dist/server.js"
},

Now, with Jest we don't need to worry about transpilation. Because, it comes with babel-jest out of the box, which loads our Babel configuration and transforms files like .js, .jsx, ts, and .tsx.

Create a test file /src/hello.test.js:

const getSum = (arr) => arr.reduce((sum, curr) => sum + curr);

describe('#getSum', () => {
  it('should return the sum of the array', () => {
    expect(getSum([1, 2, 3])).toBe(6);
  });
});

Then rebuild your docker image and run docker-compose run server npm test, and it should work.

Note: Have a look at https://github.com/facebook/jest/issues/3126 if you run into an error looking like, "ReferenceError: regeneratorRuntime is not defined."

But, your eslint will tell you something like this:

/node/app/src/hello.test.js
  3:1  error  'describe' is not defined  no-undef
  4:3  error  'it' is not defined        no-undef
  5:5  error  'expect' is not defined    no-undef

So add "jest": true to "env": { ... } in eslintrc.json so that it knows we're using Jest. Run your tests again, and when everything looks fine, you can go ahead and delete /src/hello.test.js, because we now know it works.

1.6 Environment variables

Environment variables are variables that are set outside the program while also being accessible to the program. Thus we may have different configurations for different environments.

Inside our Node.js process, we can access the variables via the process.env object, by doing process.env.{name}. There are different ways to set environment variables, and we'll start by creating a .env file (ensure that you add it to .gitignore):

PORT=3000

By installing the dotenv package we'll be able to load the variables from the .env file into process.env:

npm install dotenv

Now, all our environment variables (PORT) are available in the .env. But, for maintainability reasons let us create a /src/config.js file that will map all the process.env.{name} variables to a module:

import { config } from 'dotenv';

config();

export default Object.freeze({
  port: process.env.PORT,
});

It'll work, but an .env file is not the answer to everything:

  • What happens when we want to deploy our app to the cloud?
  • .env might accidentally be committed and pushed to the repo.
  • Might promote developers to send the .env file unencrypted to coworkers.

Also, don't forget to update your server.js:

import express from 'express';
import config from './config';

const app = express();

app.get('/', (req, res) => {
  res.send('Hello World!');
});

app.listen(config.port, () => {
  console.log(`Listening at http://localhost:${config.port}`);
});

Preparing for the cloud

We must also think about how to start our app without the .env file, especially since we've coupled our code to the dotenv library:

import { config } from 'dotenv';

config();

So, we remove the dotenv specific code and library.

dotenv also has an option like "node -r dotenv/config dist/server.js" have a look if applicable for you.

Then, add the following to our server service in docker-compose.yml:

environment:
  - PORT=${PORT}

Then update config.js:

const config = Object.freeze({
  port: process.env.PORT,
});

export default config;

1.7 Continuous integration (CI)

Continuous integration (CI) is a practice where automated builds and tests are used in order to enable regular code changes in a central repository. Every time a developer pushes code changes up to a shared version control repository it'll automatically be built and tested. Amongst many other things, it helps avoid difficult and time-consuming merge conflicts, while also making it easier to identify bugs earlier and quicker.

There are many Continuous Integration tools available, some being:

  • Jenkins
  • TeamCity
  • TravisCI
  • Github Actions
  • Azure-pipelines

In this article, we'll use Travis. Travis acts as a continuous integration platform by cloning our repository into a virtual environment, where our application is tested and built. With a .travis.yml file we define our build process. A build consists of stages, and within a stage jobs are executed in parallel, while the stages themselves are run sequentially. A job entails our repository being cloned and put into a virtual environment where we can then carry out tasks, like installing packages, compiling code, or running our tests.

Make sure you connect your GitHub repo (https://docs.travis-ci.com/user/tutorial/), and then create .travis.yml and place it in the root of our project:

script:
  - docker-compose run server npm test
  - docker-compose run server npm run build
services: 
  - docker

You choose the language, and the version of node, by just having node you inform Travis to use the latest version. You can customize much more if you want, in our example we use Docker. Read more here, https://docs.travis-ci.com/user/languages/javascript-with-nodejs/.

1.8 Continuous Delivery (CD)

Continuous Delivery (CD) is a superset of Continuous integration that automates the release process to a specified environment. After the build stage, the automated release process starts and you only have to push a button to deploy the changes. We also have Continuous deployment which does the same, but bypasses the button and deploys automatically as well.

We'll use Heroku to host our application and Travis to automate the release process. Make sure you've connected Travis to your GitHub repo before continuing, and then go to https://www.heroku.com/free and set up an account. Now do these steps:

  1. New > Create new app > name "my-express-app" > Create app
  2. Deployment method: Connect to GitHub
  3. App connected to GitHub: Choose repo > Connect
  4. Automatic deploys: choose "wait for CI to pass" > Enable Automatic Deploys

Then go to the settings of your Heroku app and set the environment variables needed at "Config Vars." Also, set NPM_CONFIG_PRODUCTION to false because else our build will fail, look at https://devcenter.heroku.com/articles/nodejs-support#skip-pruning as to why that is.

Install Heroku CLI and Travis CLIs and encrypt your keys. Read more here https://docs.travis-ci.com/user/deployment/heroku/. Then, your travis.yml should look like this:

deploy: 
  api_key: 
    secure: { your encrypted key }
script: 
  - "docker-compose run server npm test"
  - "docker-compose run server npm run build"
services: 
  - docker

Now do:

git commit -am "your message"
git push

And, Travis will automatically deploy our code to Heroku if the build is successful.

2. Building the app

Hasty decisions when coding often makes it more expensive and slower to update our code as time goes by. So, in this part, we'll acquaint ourselves with commonly used patterns that intend to mitigate that issue while giving us some fundamental structure to our project.

Of course, there are disadvantages and advantages with pretty much everything. I encourage you to try to see both when going through this part. Ask yourself, does it make sense for your project?

2.1 Data Access Object (DAO)

Data access objects (DAO) abstract and encapsulate code for accessing data from a persistence mechanism. As a result, our business logic does not get coupled with our data access logic. A generic implementation of a DAO would provide CRUD operations.

In return, our business objects gets a more straightforward interface to work with. We may also update the underlying approach to persisting data (like MongoDB, MySQL, or the file system) in our DAO without affecting the business logic.

Therefore, create a /src/daos/ folder and add a createArtistDAO.test.js file inside it. Because we'll start by writing some test cases:

import createArtistDAO from './createArtistDAO';

const artists = new Map();
const artist = createArtistDAO(artists);

afterEach(() => artists.clear());
beforeEach(() => {
  artists.set(1, { id: 1, name: 'Jason Mraz' });
  artists.set(2, { id: 2, name: 'Veronica Maggio' });
});

describe('#getAll', () => {
  it('should return all artists', async () => {
    const fetchedArtists = await artist.getAll();
    expect(fetchedArtists).toMatchObject([
      {
        id: 1,
        name: 'Jason Mraz',
      },
      {
        id: 2,
        name: 'Veronica Maggio',
      },
    ]);
  });
});

describe('#getById', () => {
  it('should return correct artist - Jason Mraz', async () => {
    const fetchedArtist = await artist.getById(1);
    expect(fetchedArtist).toMatchObject({
      id: 1,
      name: 'Jason Mraz',
    });
  });

  it('should return correct artist - Veronica Maggio', async () => {
    const fetchedArtist = await artist.getById(2);
    expect(fetchedArtist).toMatchObject({
      id: 2,
      name: 'Veronica Maggio',
    });
  });

  it('should throw an error when artist not found', async () => {
    await expect(artist.getById(3))
      .rejects
      .toThrow('Artist not found!');
  });
});

describe('#create', () => {
  it('should create a new artist', async () => {
    await artist.create({
      id: 3,
      name: 'John Mayer',
    });
    const allArtists = await artist.getAll();

    expect(allArtists).toMatchObject([
      {
        id: 1,
        name: 'Jason Mraz',
      },
      {
        id: 2,
        name: 'Veronica Maggio',
      },
      {
        id: 3,
        name: 'John Mayer',
      },
    ]);
  });

  it('should throw an error when artist name is lower than 3', async () => {
    await expect(artist.getById(3))
      .rejects
      .toThrow('Artist not found!');
  });
});

Run your tests, and you'll see them fail, which makes sense because we haven't implemented anything. But the tests are now set up, so let us proceed by implementing our DAO for artists.

Create /src/daos/createArtistDAO.js:

const createArtistDAO = (artists) => Object.freeze({
  getAll: async () => [...artists.values()],
  getById: async (id) => {
    const artist = artists.get(id);

    if (artist) {
      return artist;
    }

    throw new Error('Artist not found!');
  },
  create: async ({ name, team }) => {
    if (name.length < 2) {
      throw new Error('Name requires more than 2 characters!');
    }

    const id = artists.size + 1;
    artists.set(id, { id, name, team });
  },
});

export default createArtistDAO;

In this example, artists is assumed to be an in-memory implementation with Map. So, we've succeeded with creating an abstraction layer, which will allow us to easily swap approaches to persisting data, which we'll do with a database later.

You might wonder why I'm using Dependency injection and just not importing modules, like:

import { artists } from './db';

The reason is that I don't want hardcoded dependencies (within reason). Because we are creating a relationship when we load another module via import. So things like mocking can quickly become a struggle when the "database" used by the module is the same stateful instance for all tests. Instead, our tests may now their own artists instance as long as they abide by the interface of Map. Also, if we used a shared artists, we would be prone to race conditions because Jest runs the test in parallel. So, you'll be using dependency injection extensively throughout this guide.

Finally, run your tests, and they should all pass now.

2.2 Data Transfer Object (DTO)

Data Transfer Objects (DTO) filters out any undesirable properties. We'll use a DTO when passing our artist data around. Because:

  • We want our DAO to always return data according to the artistDTO interface.
  • We want our service to be able to assume it's always getting an artistDTO from the controller.
  • We want our controllers able to filter out unwanted data before sending it to the service.

So, create /src/dtos/createArtistDTO.js:

export default (data) => ({
  name: data.name,
  id: data.id,
  team: data.team,
});

A note if you're using a TypeScript interface, is that it's not sufficient. Because TypeScript offers no runtime type checking, instead, we have the createArtistDTO function to mitigate that issue. But that is not to say that TypeScript would be a good addition in itself!

2.3 Services & Business logic

Services encapsulate our business logic and provide a simple interface for our controllers, middleware, or other services to call.

We'll create a service that gives us a place to define the business logic related to artists. So, create a /src/services/ folder and add createArtistService.js to it:

const createArtistService = ({ artistDAO, teamService }) => {
  const getAll = async () => artistDAO.getAll();

  const getById = async (id) => artistDAO.getById(id);

  const createArtist = async (artistDTO) => artistDAO.create({
    name: artistDTO.name,
    team: await teamService.getTeam(),
  });

  return Object.freeze({
    getAll,
    getById,
    createArtist,
  });
};

export default createArtistService;

According to the fictional marketing team that I just made up, we must generate a team for the artist. They have an external API that we must call.

So we create a new /src/services/teamService.js module that we'll later inject into our artistService.

const getTeam = async () => {
  const fetchedTeam = 'team name'; // fictional api call
  return fetchedTeam;
};

export default Object.freeze({
  getTeam,
});

2.4 Controller & Mocking

The controller is the link between a user and your application. It deals with the HTTP request and response while delegating most of the work to services.

Create a /src/controllers folder, and a createArtistControllers.test.js file:

import createArtistDAO from '../daos/createArtistDAO';
import teamService from '../services/teamService';
import createArtistService from '../services/createArtistService';
import createArtistControllers from './createArtistControllers';

const artists = new Map();
const artistControllers = createArtistControllers({
  artistService: createArtistService({
    artistDAO: createArtistDAO(artists),
    teamService,
  }),
});

afterEach(() => artists.clear());
beforeEach(() => {
  artists.set(1, { id: 1, name: 'Jason Mraz' });
  artists.set(2, { id: 2, name: 'Veronica Maggio' });
});

describe('#getArtists', () => {
  it('should return all artists', async () => {
    const req = {};
    const res = { json: jest.fn() };

    await artistControllers.getArtists(req, res);
    expect(res.json.mock.calls[0][0]).toMatchObject([
      { id: 1, name: 'Jason Mraz' },
      { id: 2, name: 'Veronica Maggio' },
    ]);
  });
});

As we see, our controller(s) will be taking a req and res object. We've also separated our controllers from the express routes so we are mocking both the req and res objects.

We also mock the properties json and status with jest.fn (only mock what you need). The jest.fn function creates another function that keeps track of how many times it's called. Additionally, we can go deeper by using mockFn.mock.calls, which is an array that contains the arguments of the call made to the mock function.

This is a so-called spy that records the usage of a function.

Run your tests, and they'll fail. So create createArtistControllers.js and implement the following:

import createArtistDTO from '../dtos/createArtistDTO';

const createArtistsControllers = ({ artistService }) => {
  const getArtists = async (req, res) => {
    const artists = await artistService.getAll();
    return res.json(artists);
  };

  const createArtist = async (req, res) => {
    const artistDTO = createArtistDTO(req.body);

    try {
      await artistService.createArtist(artistDTO);
      return res.status(200).send();
    } catch {
      return res.status(500).send();
    }
  };

  return Object.freeze({
    getArtists,
    createArtist,
  });
};

export default createArtistsControllers;

The attentive reader will notice that we do not send the entire request object to our services. Instead, by using our createArtistDTO function, we filter out any undesirable properties, and the reason for that is so that our services do not get coupled with the Express framework.

Finally, run the tests, and they should pass.

2.5 Middleware

Middleware in Express is a function that takes three arguments, request, response, and next. A request flows through a pipeline of middlewares, one at a time, before reaching your controller.

Important to note is that:

  • Middleware can modify the request and pass it along to the next middleware by calling next.
  • The pipeline will terminate the request if you don't call next.
  • You should eventually send a response to the client with something like res.send or res.json.

Now we'll create a middleware that ensures request.body contains the required fields for createArtist before calling it. With that middleware, we will block artists from being added when the data is invalid.

Create the folder /src/middleware and add validateArtistMiddleware.test.js:

import validateArtistMiddleware, { isValidArtist } from './validateArtistMiddleware';

describe('#isValidArtist', () => {
  it('should return true when body has name property', () => {
    const body = { name: 'Céline Dion' };
    expect(isValidArtist(body)).toEqual(true);
  });

  it('should return false when body is missing name property', () => {
    const body = { age: 35 };
    expect(isValidArtist(body)).toEqual(false);
  });
});

describe('#validateArtistMiddleware', () => {
  it('should call res.status(400) when missing name', () => {
    const req = {
      body: {
        age: 35,
      },
    };
    const res = { status: jest.fn(() => res), send: jest.fn() };
    const next = jest.fn();

    validateArtistMiddleware(req, res, next);

    expect(res.status.mock.calls[0][0]).toEqual(400);
    expect(res.send.mock.calls[0][0]).toEqual('Bad Request - parameters are missing');
    expect(next.mock.calls.length).toEqual(0);
  });

  it('should call next() when name property is set', () => {
    const req = {
      body: {
        name: 'Jason Mraz',
      },
    };
    const res = { };
    const next = jest.fn();

    validateArtistMiddleware(req, res, next);

    expect(next.mock.calls.length).toEqual(1);
  });
});

By having separate tests for middleware and not solely relying on E2E tests, we shorten the feedback loop when developing.

Now implement it in validateArtistMiddleware.js:

export const isValidArtist = (body) => {
  if (!body.name) {
    return false;
  }

  return true;
};

const validateArtistMiddleware = (req, res, next) => {
  if (isValidArtist(req.body)) {
    return next();
  }

  return res.status(400).send('Bad Request - parameters are missing');
};

export default validateArtistMiddleware;

2.6 Express Routes

Our controllers are coupled to the Express framework since they're using the req and res objects - so why bother separating them from the routes?

The reasoning, in this case, is that we want to be able to test our middlewares and controllers separately. But admittedly, we still want to ensure everything works together, and that's why we'll be setting up HTTP API testing (E2E) for our routes in this section.

We'll use SuperTest for HTTP API testing. So install it along side body-parser (for req.body):

npm install --save-dev supertest
npm install body-parser

Continue by creating the folder /src/routes and adding artistRoutes.test.js to it with the following code:

import express from 'express';
import request from 'supertest';
import bodyParser from 'body-parser';
import artistRouter from './artistRoutes';

const app = express();
app.use(bodyParser.json());
app.use(artistRouter);

describe('POST /artists/new', () => {
  it('should return 200', async () => {
    const res = await request(app)
      .post('/artists/new')
      .set('Accept', 'application/json')
      .send({ name: 'John' });

    expect(res.status).toBe(200);
  });

  it('should return 400 code - because name is missing', async () => {
    const res = await request(app)
      .post('/artists/new')
      .set('Accept', 'application/json')
      .send({ name: undefined, age: '40' });

    expect(res.status).toBe(400);
  });
});

Then, create a folder /src/routes and add artistRoutes.js to it:

import express from 'express';
import createArtistDAO from '../daos/createArtistDAO';
import teamService from '../services/teamService';
import createArtistService from '../services/createArtistService';
import createArtistControllers from '../controllers/createArtistControllers';
import validateArtistMiddleware from '../middleware/validateArtistMiddleware';

const router = express.Router();
const artistControllers = createArtistControllers({
  artistService: createArtistService({
    artistDAO: createArtistDAO(new Map()),
    teamService,
  }),
});

router.post('/artists/new', validateArtistMiddleware, artistControllers.createArtist);

export default router;

In this section, we are testing routes separately. However, this approach does come with disadvantages. Because perhaps your server.js app contains middleware or configuration that you might want to include in your test cases, so bear that in mind.

Part 3. Persisting data

A database is structured data that most likely gets controlled via a database management system (DBMS), more colloquially put a database system. A database system provides an API to access the database. They essentially:

  • Store data (disk, sometimes even RAM).
  • Make their data accessible via something like SQL or MQL.

DBMS has different models. A database model decides the structure of the database, how data is stored, organized, and accessed. The two most famous models are:

  • Relational model: requires a defined schema. It sorts data into tables (referred to as relations), each consisting of rows and columns. A single data object can spread across several tables.
  • Document model: does not require a defined schema. It sorts data into collections (equivalent to tables). And a collection contains documents that do not enforce data types.

Examples of relational database management systems are MySQL, PostgreSQL, Microsoft SQL Server, and Oracle Database. Furthermore, examples of nonrelational database management systems include MongoDB and Couchbase.

Object Mapping

Object-relational mapping (ORM) entails at its core converting tables into classes, rows into objects, and columns into properties. As you can imagine, this comes with benefits and disadvantages, which we won't cover. Instead, we'll:

  • Use Mongoose, which is an ODM.
  • Use Sequelize, which is an ORM.

ORMs are for SQL databases, while so-called object document mapping (ODM) is for document-oriented databases.

3.1 Persisting data with MongoDB

MongoDB is a document database, and it provides a driver for Node called mongodb. We'll be using it, but via the ODM mongoose.

So, start by installing mongoose:

npm install mongoose

Create /src/common/db/db.js and paste the following code into it:

import mongoose from 'mongoose';

const mongooseOptions = {
  useNewUrlParser: true,
  useUnifiedTopology: true,
};

const connect = async (mongoUri) => mongoose.connect(mongoUri, mongooseOptions);

export default {
  connect,
};

Then, import mongoose and define the schema directly in the DAO. From my viewpoint, there's no point in creating an artist.schema.js file because that logic gets coupled anyway. However, we still use DI so we can mock the input.

Update the /src/daos/createArtistDAO.js to:

import mongoose from 'mongoose';
import createArtistDTO from '../dtos/createArtistDTO';

const artistSchema = new mongoose.Schema({
  name: { type: String, required: true },
  id: { type: Number, required: true },
  team: { type: String },
});

export const artistModel = mongoose.model('artist', artistSchema);

export default (ArtistModel) => {
  const getAll = async () => (await ArtistModel.find()).map(createArtistDTO);

  const getById = async (id) => {
    const artist = await ArtistModel.findOne({ id });

    if (artist) {
      return createArtistDTO(artist);
    }

    throw new Error('Artist not found!');
  };

  const create = async ({ name, team }) => {
    if (name.length < 2) {
      throw new Error('Name requires more than 2 characters!');
    }

    const id = (await ArtistModel.countDocuments()) + 1;

    const artistDoc = new ArtistModel({ id, name, team });
    await artistDoc.save();
  };

  return Object.freeze({
    getAll,
    getById,
    create,
  });
};

Run your tests, and they'll fail. They fail because we're missing a MongoDB server. Luckily, the mongodb-memory-server package starts a mongod process that stores data in-memory. Which, in return, allows us to avoid having to mock our own objects.

So continue by installing @shelf/jest-mongodb and mongodb-memory-server:

npm install --save-dev @shelf/jest-mongodb mongodb-memory-server

Then, add the following to jest.config.json:

{
  "preset": "@shelf/jest-mongodb"
}

It also doesn't work with alpine read more here, so I'll update the top of our Dockerfile to:

FROM node:17-buster

Now, the package gives you two important global variables, global.__MONGO_URI__ and global.__MONGO_DB_NAME__. So, we create /src/common/db/dbTestHelper.js where we'll set up functions that make our tests cleaner.

/* eslint no-underscore-dangle: 0 */
import mongoose from 'mongoose';
import db from './db';

const close = async () => {
  await mongoose.connection.dropDatabase();
  await mongoose.connection.close();
};

const connect = async () => {
  await db.connect(global.__MONGO_URI__);
};

const clearCollection = async (colName) => {
  await mongoose.connection.collection(colName).deleteMany();
};

const addToCollection = async (colName, data) => {
  await mongoose.connection.db.collection(colName).insertMany(data);
};

export default {
  close,
  connect,
  clearCollection,
  addToCollection,
};

Then update your test:

import createArtistDAO, { artistModel } from './createArtistDAO';
import dbTestHelper from '../common/db/dbTestHelper';

const artist = createArtistDAO(artistModel);

afterAll(dbTestHelper.close);
beforeAll(dbTestHelper.connect);
afterEach(() => dbTestHelper.clearCollection('artists'));
beforeEach(() => dbTestHelper.addToCollection('artists', [
  {
    id: 1,
    name: 'Jason Mraz',
  },
  {
    id: 2,
    name: 'Veronica Maggio',
  },
]));

describe('#getAll', () => {
  it('should return all artists', async () => {
    const fetchedArtists = await artist.getAll();
    expect(fetchedArtists).toMatchObject([
      {
        id: 1,
        name: 'Jason Mraz',
      },
      {
        id: 2,
        name: 'Veronica Maggio',
      },
    ]);
  });
});

describe('#getById', () => {
  it('should return correct artist - Jason Mraz', async () => {
    const fetchedArtist = await artist.getById(1);
    expect(fetchedArtist).toMatchObject({
      id: 1,
      name: 'Jason Mraz',
    });
  });

  it('should return correct artist - Veronica Maggio', async () => {
    const fetchedArtist = await artist.getById(2);
    expect(fetchedArtist).toMatchObject({
      id: 2,
      name: 'Veronica Maggio',
    });
  });

  it('should throw an error when artist not found', async () => {
    await expect(artist.getById(3))
      .rejects
      .toThrow('Artist not found!');
  });
});

describe('#create', () => {
  it('should create a new artist', async () => {
    await artist.create({
      id: 3,
      name: 'John Mayer',
    });
    const fetchedArtists = await artist.getAll();
    expect(fetchedArtists).toMatchObject([
      {
        id: 1,
        name: 'Jason Mraz',
      },
      {
        id: 2,
        name: 'Veronica Maggio',
      },
      {
        id: 3,
        name: 'John Mayer',
      },
    ]);
  });

  it('should throw an error when artist name is lower than 3', async () => {
    await expect(artist.getById(3))
      .rejects
      .toThrow('Artist not found!');
  });
});

But the problem with this approach is that you'll likely run into race conditions when you add more tests and run all your tests in parallel. So, have a look at https://github.com/shelfio/jest-mongodb/issues/236 before you resort to jest --runInBand. I did so, and with the acquired information, I now ensure each test file gets a unique database. For example, inside jest-mongodb-config.js:

module.exports = {
  mongodbMemoryServerOptions: {
    binary: {
      version: '5.0.3', // note: it caused issues when 4.0.3
      skipMD5: true,
    },
    instance: {},
    autoStart: false,
  },
};

And inside /src/common/db/dbTestHelper.js:

const connect = async () => {
  const mongoUri = global.__MONGO_DB_NAME__
    ? `${global.__MONGO_URI__.split('/').slice(0, -1).join('/')}/${global.__MONGO_DB_NAME__}`
    : global.__MONGO_URI__;

  await db.connect(mongoUri);
};

Lastly, I'll leave it as an exercise to update your createArtistControllers.test.js and artistRoutes.test.js files.

3.2 Your team uses PostgreSQL

The organization that you work for now wants to replace MongoDB with PostgreSQL.

Your heart stops for a moment.

But you realize that a refactor might not be that terrible after all. You only have to update /src/daos/createArtistDAO.js since you never coupled the database details to the controllers, services, and so on.

But what is PostgreSQL?

PostgreSQL is a relational database management system (RDBMS). We use it to manage data stored in relations which we call tables.

  • A table is a database object that contains rows of data.
  • A row consists of columns.
  • A column has a data type.

PostgreSQL is open-source and derived from a package called POSTGRES. The project got developed at the University of California at Berkeley Computer Science Department. The project started in 1986 and got led by esteemed Professor Michael Stonebraker. Then, in 1995, two students from Berkely (Andrew Yu and Jolly Chen) replaced the QUEL with SQL, and named that version Postgres95. Later, Postgres95 got a more appropriate name PostgreSQL to reflect that it uses SQL.

Refactoring with sequelize

We'll replace Mongoose with sequelize. Sequelize is an ORM commonly used for Postgres, MySQL, MariaDB, SQLite, and Microsoft SQL Server.

Install the following packages:

npm install --save sequelize pg pg-hstore

Create /src/db.js:

import { Sequelize } from 'sequelize';

const db = new Sequelize('postgres://user:[email protected]:5432/dbname');

export default db;

How do we synchronize the database with sequelize? Well, with sequelize we note the following:

  • With db.define, you define a model which informs sequelize about the table.
  • Artist.sync({ force: true }) creates a table, but beforehand drops it, if it already exists.
  • Artist.sync() creates a table, if it does not exist.
  • You can use migrations instead of sync, https://sequelize.org/master/manual/migrations.html.

Continue by updating the createArtistDAO.js file:

import { DataTypes } from 'sequelize';
import db from '../db';

export const Artist = db.define('Artist', {
  id: {
    type: DataTypes.INTEGER,
    allowNull: false,
    primaryKey: true,
  },
  name: {
    type: DataTypes.STRING, // VARCHAR(255)
    allowNull: false,
  },
  team: {
    type: DataTypes.STRING,

  },
});

export default (ArtistModel) => {
  const getAll = async () => ArtistModel.findAll();

  const getById = async (id) => {
    const artist = await ArtistModel.findOne({ where: { id } });

    if (artist) {
      return artist;
    }

    throw new Error('Artist not found!');
  };

  const create = async ({ name, team }) => {
    if (name.length < 2) {
      throw new Error('Name requires more than 2 characters!');
    }

    const id = (await ArtistModel.count()) + 1;

    await ArtistModel.create({ id, name, team });
  };

  return Object.freeze({
    getAll,
    getById,
    create,
  });
};

Also, update your config.js file:

const config = Object.freeze({
  port: process.env.PORT,
  dbName: process.env.DB_NAME,
  dbUser: process.env.DB_USER,
  dbPassword: process.env.DB_PASSWORD,
  dbHost: process.env.DB_HOST,
  dbPort: process.env.DB_PORT,
});

export default config;

Then, update /src/db.js to:

import { Sequelize } from 'sequelize';
import config from './config';

const db = new Sequelize(config.dbName, config.dbUser, config.dbPassword, {
  host: config.dbHost,
  port: config.dbPort,
  dialect: 'postgres',
  logging: false,
});

export default db;

Now, we also have to update our tests, and there are many alternatives. Each approach comes with its advantages and disadvantages:

  1. Use mocks.
  2. Use an in-memory database.
  3. Spin up a temporary database (probably with Docker).
  4. Test against a database specific for test or dev.

I reflected on using sequelize-mock, but it seems not to be maintained anymore. Thus, I'll set up my own PostgreSQL server instead, made for testing with Docker.

Start by adding two new services, where one of them spins up a temporary DB with the postgres docker image:

  test:
    build: .
    environment:
      DB_NAME: ${DB_NAME}
      DB_USER: ${DB_USER}
      DB_PASSWORD: ${DB_PASSWORD}
      DB_HOST: ${DB_HOST}
      DB_PORT: ${DB_PORT}
    volumes:
      - "./:/node/app"
      - "exclude:/node/app/node_modules"
    depends_on:
      - db-test

  db-test:
    image: postgres:latest
    container_name: db-test
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_HOST_AUTH_METHOD: trust
    ports: 
      - ${DB_PORT}:${DB_PORT}

Update your .env to:

# Server
PORT=3000

# Database
DB_NAME=db-name
DB_USER=username
DB_PASSWORD=password
DB_HOST=db-test
DB_PORT=5432

Update the top of your createArtistDAO.test.js file to:

import createArtistDOA, { Artist } from './createArtistDAO';
import db from '../db';

const artist = createArtistDOA(Artist);

afterAll(async () => {
  await db.close();
});
beforeEach(async () => {
  await db.sync({ force: true });
  await Artist.bulkCreate([
    {
      id: 1,
      name: 'Jason Mraz',
    },
    {
      id: 2,
      name: 'Veronica Maggio',
    },
  ]);
});

Lastly, update your package.json to use --runInBand and run docker-compose run test npm test (sadly, this approach requires the tests to be sequentially executed). Then, you can update the other tests accordingly.

Update Travis.yml

Add your environment variables to Travis. These aren't secret (only used for testing), so I'll add them without encryption - but you should encrypt yours if they need to be secret.

deploy: 
  api_key: 
    secure: { your encrypted key }
env: 
  global: 
    - DB_NAME=db-name
    - DB_USER=username
    - DB_PASSWORD=password
    - DB_HOST=db-test
    - DB_PORT=5432
script: 
  - "docker-compose run test npm test"
services: 
  - docker

Note that the global variables get used by our docker-compose.yml.

3.3 Starting the server

We haven't started our app since part 1. We have only been using the feedback of tests, which was more enjoyable than continuously refreshing the browser. Do you agree?

Anyway, let's finally start our server. Go back and use the MongoDB set up from this branch. Then, we'll use the free cloud-based database that MongoDB Atlas provides. Read more about getting started with MongoDB Atlas here if you're interested.

Here's the updated server.js file:

import express from 'express';
import bodyParser from 'body-parser';
import artistRouter from './routes/artistRoutes';
import db from './common/db/db';
import config from './config';

db.connect(config.mongoUri);

const app = express();

app.use(bodyParser.json());
app.use(artistRouter);

app.get('/', (req, res) => {
  res.send('Hello World!');
});

app.listen(config.port, () => {
  console.log(`Listening at http://localhost:${config.port}`);
});

Then update the configuration parts to make it work. You can inspect this commit if you get stuck.

3.4 Review

As stated in the beginning, this article's intent was to give you perspective. With that said, here are some of my thoughts on the project:

  • TypeScript would have been a great addition, though I wanted to keep the article more accessible.
  • Having to rebuild your image every time you add a new package takes time - but should/will you really install packages that often?
  • Security is important, yet we did not cover it here. So, I encourage anyone who will use this structure to allocate more focus on it.
  • When using a TDD approach, you probably won't create as many test cases as we did all at once - however, I liked to structure the article that way.
  • You probably won't be changing your database that often, and Mongoose is in a way acting as a DAO, so you could skip the DAO and use Mongoose directly in your services.
  • The routes could have used dependency injection as well.

I also would like to give you some recommendations for further reading that I found useful when setting up the project environment:

  1. https://github.com/BretFisher/node-docker-good-defaults
  2. https://medium.com/the-node-js-collection/making-your-node-js-work-everywhere-with-environment-variables-2da8cdf6e786