Node.js Configuration Management for Applications in Real-World

Managing configs for your apps can be a tricky thing to do in the big, bad world of microservices—especially when balancing different upstream APIs alongside different configs for dev, staging, QA, and production environments. If you look up at npmjs.org, there are different config loaders and managers available as modules, but rarely do they follow a well-defined or unified universal approach.

That’s why we started following an approach using a runtime env variable-based config management for our Express.js and Node.js apps, which we have been using at scale for the past 3–4 years.

Let’s assume our software is deployed in a few environments, e.g., “QA,” “staging,” and “production.”

What are the pitfalls of NodeJS configuration management?

It’s very common to have config-qa.json, config-staging.json, and config-prod.json in the repository, but we recommend avoiding this approach.

The cons are the following:

1. Access control. Not all developers who have access to the code need to have access to production. Further, such code cannot be published as open source due to the presence of the configs in the repository with parameters for accessing your infrastructure.

2. The set of environments can be expanded: various dev-boxes, local environments, and so on. It will be hard to maintain. You will constantly have to add new files to the repository and maintain many copies of the configs.

3. By creating builds in the form of Docker containers, they will be tied to the environment, and this violates the main idea of ​​using Docker builds. It would be better to test the build for QA and then roll out the same build in prod and not build a new one. Building a new one increases the risk of a different configuration (for example, one of the dependent packages has been updated).

“12 factors” and why it’s not enough

According to the Twelve-factor methodology, the configuration should be passed in the environment variables (https://12factor.net/ru/config).

Holding the configuration data in the env variables is pretty flexible. You can run your app in a Docker container, on your host machine, or in the Lambda environment. Furthermore, you could reuse the same source code and images in all environments.

But at the same time, passing environment variables when starting a process is not very convenient. There is a popular library (dotenv) that allows you to save environment variables in the .env file, but you should not commit this file to the repository. Usually, in the repository, you keep a file with an example configuration (for example, “.env.sample,” which you need to copy to the .env file if you deploy without Docker). You can use dotenv-defaults, a library that allows you to have a default .env.defaults file that you have already committed to the repository.

But in reality, this is not enough. You will have rather complex configs with nested structure, and you often don’t want to define all the values ​​in the environment variables. There are too many of them, and not all of them change depending on the environment (for example, the link to your product’s Twitter account should be in the config, but it makes no sense to pass it in environment variables). If you are familiar with Ansible, you know that you can have a config template and substitute variables in it depending on the environment and build the final config, but remember that you cannot pack this config into a build. In this regard, we store the config template in the repository and template it with environment variables. And if there are many options in the config, then it makes sense to automatically validate it after templating. It will greatly improve the life of the developers.

How do we do Node.js app configuration management?

We’ve created libraryconfme in order to solve the problem of managing configs (We also use it in our Node.js start app that is available as an open source project on Github: (https://github.com/WebbyLab/webbylab-starter-app-for-nodejs)

It’s a small library, but it solves all the problems described above; it is a small wrapper around dotenv-defaults and LIVR (for validation)

How does confme work under the hood?

“confme” loads the config and templates it with environment variables. The library uses dotenv-defaults to get environment variables, so you can create a “.env.defaults” file to store the default values of your environment variables. If you have placeholders in the config for which there are no environment variables, then “confme” will throw an exception.

You can also describe the config scheme, and it will be checked for compliance with the scheme automatically after templating.

How to use confme?

const confme = require("confme");
const path   = require(“path”);

const config = confme(path.join(__dirname, "config.json"));

Configuration file example:

{
  "listenPort": "{{PORT}}",
  "apiPath": "https://{{DOMAIN}}:{{PORT}}/api/v1",
  "staticUrl": "https://{{DOMAIN}}:{{PORT}}/static",
  "mainPage": "https://{{DOMAIN}}:{{PORT}}",
  "mail": {
    "from": "MyApp",
    "transport": "SMTP",
    "auth": {
      "user": "{{SMTP_USER}}",
      "pass": "{{SMTP_PASS}}"
    }
  }
}

DOMAIN, PORT, SMTP_PASS, SMTP_PASS must be passed in environment variables in any way.

We recommend doing the following:

☑️ Define default values ​​for environment variables in the “.env.defaults” file. They should be suitable for local development. In this case, the developer does not need to configure anything after cloning the repository.

☑️ If you deploy without containers and lambdas, you can create a “.env” file during deployment and override the default variables for your environment. Or you can pass it at startup if you have only a few parameters (though there is a risk of forgetting about them when restarting or debugging production): DOMAIN = myapp.com PORT = 80 node app.js.

☑️ If you are building a Docker container or packing Lambdas, then just pass environment variables in a way that suits your environment (there is always a mechanism for this).

“confme” supports config validation with LIVR out of the box. Uses LIVR JS and livr-extra-rules (additional rules set)

You can pass the path to the file with the config schema as the second argument:

const confme = require("confme");
const path   = require(“path”);
 
const config = confme(
  path.join(__dirname, "config.json"),
  path.join(__dirname, "config-schema.json")
);

Schema example:

{
  "listenPort": ["required", "positive_integer"],
  "apiPath": ["required", "url"],
  "staticUrl": ["required", "url"],
  "mainPage": ["required", "url"],
  "mail": ["required", {"nested_object": {
    "from": ["required", "string"],
    "transport": ["required", {"one_of": ["SMTP", "SENDMAIL"] }],
    "auth": {"nested_object": {
      "user": ["required", "string"],
      "pass": ["required", "string"]
    }}
  }}]
}

As we have described, the confme approach provides a few important benefits for Node.js configuration management:

1. Improved reliability: fail at the service start if any of the required env variables is not defined
2. Keep the environment-dependent configuration outside of the repository/Docker container.
3. Provide a structured configuration for the config with validation.