Create your own NPM private feed with Azure DevOps

In this blog post I’m going to talk about a technical solution where you can share software components between different applications. This is only relevant if you have multiple applications and want to share parts between them.

There is also a Dutch version of this blogpost

TL;DR; (Resume)

In this blog, we link a private Node Package Manager (NPM) feed to use during development, in a container, and in the Continuous Integration (CI) pipelines. I do this to have versions of components. I use the tools Azure DevOps, NPM, Nodejs and Docker. To ensure that the authentication, we use an Oath2 flow where just before the installation (preinstall) retrieve tokens. There is a Github repository where a working example is given

Project wishes

For a project I am working on, there is a wish for separate NPM registries. NPM stands for Node Package Manager. NPM is used to reuse javascript based projects online. The public npmjs.com feed is used for this. If you want to share code between a limited set of users or teams you can publish private user (project) or organisation-scoped packages to the npm registry. User (project) scoped means it will only be shared within the same Azure DevOps project and if you choose organisation-scoped it will be shared across the organization. These private NPM registries have the advantage that front-end applications can exchange self-developed software components.

What is Azure DevOps

Azure DevOps is a developer services tool that allows teams to plan work, collaborate on code development, and build and deploy applications. Azure DevOps supports a collaborative culture and set of processes that bring developers, project managers, and contributors together to develop software. We use the pipeline functionalities the most. That is the central compilation of software and release to environments.

Azure DevOps Artifacts

In addition to the pipelines, there are also Artifacts. Azure Artifacts allows developers to share and use packages from different feeds and public registries. Packages can be shared within the same team, organization and even publicly. Azure Artifacts supports multiple package types, such as NuGet, npm, Python, Maven, and Azure Universal Packages.

For example, a feed looks like this:

Overview of a private npm feed | Photo 1

Docker containers

For the project we use Docker containers to run the software.
A container is a way to keep software and all software dependencies together and to easily exchange them between different environments. A Docker container image is a lightweight, self-contained, executable software package that contains everything needed to run an application: code, runtime, system tools, system libraries, and settings.

Azure’s Windows-only solution

The official tool to handle authentication to your private NPM feed is vsts-npm-auth only this one only works on Windows. Since we work in a combination of Mac, Windows and Docker in our project, this is not a workable solution in our situation.

Personal access tokens (PATs)

Unfortunately, PATs don’t work well with retrieving NPM packages from the Azure Artifacts feed. I get 401 errors on this and can’t get any further.

An alternative: better-vsts-npm-auth

I looked further and came across the better-vsts-npm-auth project. This tool does an oauth2 authentication towards Azure DevOps and ensures that you as a user have a valid token to retrieve npm packages

In practice: Steps required in advance

You need the following steps to generate a refresh token yourself.

1. Before you start: Make sure your development machine has a recent version of nodejs installed
2. Make sure you have access to the Azure DevOps Artifacts section: https://dev.azure.com/YOUR_ORG/_packaging

I have collected the code snippets below and put them in a separate repository. It can be found and downloaded on Github.

Create a new feed in Azure Devops

Within Azure DevOps this falls under Artifacts click on “+ Create Feed”

You can choose from two different scopes here

1. Project, so only the current project
2. Organization: within the current Azure DevOps organization

Add a new feed | photo 2

Install the global package better-vsts-npm-auth

On your local machine you install better-vsts-npm-authas a global package so that it can be called from different locations

Do this with the following command:

npm i -g better-vsts-npm-auth

Setup for front-end library

Go to your front-end library folder and make sure it contains at least two things.

1. project .npmrcfile
   The project npmrc contains the locations of your own private feeds prefixed with a package name. Below I give an example of how I apply this.
2. package.jsonfile
   This contains the names of all dependencies that are used.

1. Library Project .npmrcfile

@qdraw-components:registry=https://pkgs.dev.azure.com/qdraw/_packaging/demo/npm/registry/
always-auth=true

2. Library Project package.json file

{
    "name": "@qdraw-components/components",
    "version": "0.0.1-rc.1",
    "description": "Test front-end components",
    "scripts": {
    },
    "author": "",
    "license": "ISC",
    "devDependencies": {
        "typescript": "^4.5.5"
    },
    "dependencies": {
    },
    "main": "dist/cjs/index.js",
    "module": "dist/esm/index.js",
    "files": [
        "dist"
    ],
    "types": "dist/index.d.ts"
}

Authentication from your development machine

The authentication then takes place via the Azure DevOps using the recently installed tool. I do this within the front-end library folder that also contains the above package.json and npmrc file.

better-vsts-npm-auth

better-vsts-npm-auth | photo 3

If I copy the above URL into the browser I get the following screen. Press Accept to continue

Stateless VSTS NPM OAuth | photo 4

When you click on Accept you will see the Success screen. This screen contains a command with a unique refresh token.

Copy this command and run it in your terminal | photo 5

We do two things with the token: Run the command:

better-vsts-npm-auth config set refresh_token eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laW

Copy the token part without: better-vsts-npm-auth config set refresh_token. Set this as environment variable DEMO_NPM_REFRESH_TOKEN, how to do that is described below. We will need this variable later in Docker.

1. On Mac OS, add the following to your .zprofile or .zshrc:

nano ~/.zprofile
export DEMO_NPM_REFRESH_TOKEN="LongTokenHere"

These settings are only active after restarting your terminal window

2. On Windows, add the following:

• Press the following key combination: Windows + R
• Type: `sysdm.cpl`
• Click on the Advanced tab
• Slightly above OK & Cancel click on Environment Variables…
• Under System Variable: Press New
• DEMO_NPM_REFRESH_TOKEN
• And the token (without the better-vsts prefix)
• Press Ok to confirm

These settings are only active after restarting your Powershell window

Check if you are logged in in the correct environment

Verify that the app: “Stateless VSTS NPM OAuth” is present at: https://dev.azure.com/YOUR_ORG/_usersSettings/authorizations

If the app isn’t there, you’re probably connected to the wrong organization. If so, try copying and pasting the url into a browser window that is in private or incognito mode.

Check if the app is in the list | photo 7

In case of authentication problems

If there are problems with the authentication, remove the following files from your home folder. On Mac OS an example /Users/dioncould be this and on Windows C:\Users\Dion.

1. “.vstsnpmauthrc”
2. “.npmrc”
3. The environment variable “DEMO_NPM_REFRESH_TOKEN”

Both files contain credentials and can be safely deleted. Then try again. Below I show that the two files are present

Your home folder, you can delete these files to recreate | photo 8

NPM Publish from front-end library folder

From the front-end library folder we do a publish to Azure DevOps. When publishing, make sure you have a unique incrementing version number in the package.json each time.

NPM Publish | photo 6

Now this package is in the private npm feed and it looks like this.

Newly created feed | photo 9

Unique prefix when you have multiple private feeds

In our case we use multiple npm feeds for different parts. In the example above, I’m using the prefix: “@qdraw-components” and a forward slash to end this prefix. It is important to construct the name this way, because then you can indicate this in the project npmrc file. If you have a feed next to it for icons, it can’t have the same prefix. So the prefix must be unique and start with an at sign.

Docker Application setup

In the application where the front-end library is used, we do the following setup: The same. npmrc file is located in the application folder.

The project .npmrc file

@qdraw-components:registry=https://pkgs.dev.azure.com/qdraw/_packaging/demo/npm/registry/
always-auth=true

In addition, this project has its own package.json

package.json

{
  "name": "use_app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "preinstall": "node scripts/preinstall.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@qdraw-components/components": "^0.0.1-rc.1"
  }
}

In the preinstall, an access token is retrieved based on a refresh token so that your private packages can be retrieved and installed

The scripts/preinstall.js file can be found on Github

In the Dockerfile, the scripts/cleanauth.js file is called to remove the credentials from the container.

scripts/cleanauth.js

#!/usr/bin/env node
 
// remove tokens from docker image
 
const fs = require('fs')
const path = require('path');
 
function UserProfileFolder() {
    let userProfileFolder = "~";
    if (process.env.CSIDL_PROFILE) {
        userProfileFolder = process.env.CSIDL_PROFILE;
    }
    if (process.env.HOME) {
      userProfileFolder = process.env.HOME;
    }
    return userProfileFolder;
}
 
const userProfileFolder = UserProfileFolder();
 
const vstsNpmauthRcFilePath = path.join(userProfileFolder, '.vstsnpmauthrc');
if (fs.existsSync(vstsNpmauthRcFilePath)) {
    fs.unlinkSync(vstsNpmauthRcFilePath)
}
 
const userNpmRcFilePath = path.join(userProfileFolder, '.npmrc');
if (fs.existsSync(userNpmRcFilePath)) {
    fs.unlinkSync(userNpmRcFilePath)
}

This project has the following Dockerfile

Dockerfile

# Install dependencies only when needed
FROM node:16-alpine AS deps
ARG YOURPROJECT_NPM_REFRESH_TOKEN=default
ARG AZURE_AUTH_TOKEN=default
 
RUN apk add --no-cache libc6-compat
WORKDIR /app
 
ENV YOURPROJECT_NPM_REFRESH_TOKEN=$YOURPROJECT_NPM_REFRESH_TOKEN
ENV AZURE_AUTH_TOKEN=$AZURE_AUTH_TOKEN
 
COPY use_app/scripts/preinstall.js ./scripts/preinstall.js
COPY use_app/scripts/cleanauth.js ./scripts/cleanauth.js
COPY use_app/package.json ./
 
RUN node scripts/preinstall.js
RUN npm ci --prefer-offline
 
RUN node scripts/cleanauth.js

This Dockerfile can be called in two ways so that it becomes a docker container.

1. Docker build command

cd root_of_solution
docker build -f use_app/Dockerfile . --no-cache --build-arg DEMO_NPM_REFRESH_TOKEN=${DEMO_NPM_REFRESH_TOKEN}

2. Or as Docker compose

docker-compose.yml

version: '3.4'
 
services:
  demo.use.app:
    image: ${DOCKER_REGISTRY-}demouseapp
    build:
      context: .
      dockerfile: use_app/Dockerfile
      args:
        - DEMO_NPM_REFRESH_TOKEN=${DEMO_NPM_REFRESH_TOKEN}
    environment:
      - DEMO=true
    ports:
      - "19443:9443"

Docker runs on a local environment

Docker | photo 10

Azure DevOps Pipeline

To configure Azure DevOps you use yaml files, which contain all the steps that the build pipeline will perform.
See for this: azure/azure-pipelines.yml and azure/templates/build-docker.yml

Azure Devops Build pipeline, 403 | photo 11

If you use an organisation-scoped packages from the Azure Artifact you will get the following error:

“npm ERR! 403 403 Forbidden — GET https://pkgs.dev.azure.com/YOURENV/_packaging/demo/npm/registry/@qdraw-components%2fcomponents — User d9af02c4-b000–4fd4–9754–68706813d7ca”

To fix this add the build user to the private feed

Follow the arrow | photo 12

In the screenshot below you can see how I do that.

Click Add users/group | photo 13

Build successfully

Azure Devops is running well | photo 14

Conclusion

It was an interesting search for the possibilities of Azure DevOps and in particular how they have put this part together. Because there was so little to be found on the internet, I researched a lot myself and in this blog I tried to write out my knowledge as well as possible.

And the package has been successfully delivered…

Parcel deliverer | photo 15

Error scenario: Connected to the wrong tenant

If you see the following message in the preinstall.js: You are probably connected to the WRONG tenant Then go to the switcher in Azure Devops https://app.vsaex.visualstudio.com/me and select the correct company on the left. Then go back to oauth provider, this is in most cases: stateless-vsts-oauth.azurewebsites.net to get a new refresh token. Place these again as environment variables and restart the window to try again afterwards

Optional step: Self-host the stateless-vsts-oauth provider

If you want to host the stateless-vsts-oauth yourself, for example not to depend on: stateless-vsts-oauth.azurewebsites.net then the code can be found at: github.com/zumwald/stateless-vsts-oauth For this you need to register an App on Azure Devops. Go to Azure Devops Register oauth app Enter your company name and contact details here. At Application Information you enter the address where the app is hosted at Application website. This address must be publicly available. Furthermore, the Authorization callback URL is the same address with the addition /oauth-callback. The Authorized scopes are Packaging (read and write). You don’t have to tick anything else. Click Create Application to continue. You will need the App ID and the Client Secret in the following steps. To access this overview later, go to https://app.vsaex.visualstudio.com/me

Use the following format to set the environment variables

CLIENT_ID : YOUR_CLIENT_ID_GUID
CLIENT_SECRET : eyJ0eXAi....
PORT : 8080
WEBSITE_HOSTNAME : devopsauth.example.com

The App ID is also used in the preinstall.js and the address where the oauth provider is hosted. It is important that you also change this to this