Running shell scripts with Node

Since this site has been around for quite a while now in various incarnations, I’ve been thinking for a while about how to audit all the things I’ve linked to and make sure the links exist (and also that I’m not introducing breaking changes as I develop it). Fortunately before I ended up trying to write something myself, I came across Hyperlink, which does the job. (In the process I discovered a lot of 404s, some bad #fragment links, and a distressing number of redirects from http => https, or /path to /path/ or vice versa without any apparent consistency.) The basic usage is well documented in the readme and the intro post, but it took me a bit of experimentation to get it running the way I wanted.

The basic command I landed on is something like this:

$ hyperlink http://localhost:3000 -r --skip http://localhost:3000/client.js | tap-spot

That is, run hyperlink recursively against my local server, skipping a particular URL that fails (it’s an example link in a tutorial), then pipe the output into a test reporter.

I had a few requirements:

1. it should be easily run with just one or two commands - I don’t want to have to memorise the list of urls to --skip
2. it should run with different configurations (e.g. run against local build and run against production site)
3. it should not involve hard-coding long cryptic commands
4. it must not increase my install/build time

Hardcoded NPM scripts

The obvious solution to me was to add some scripts into the package.json and hard code the different configs in different scripts. To start with we can install the required packages globally - not ideal, but better than wasting CPU time and bandwidth for something I only want to run locally.

{
  "scripts" {
    "test-links": "hyperlink http://localhost:3000 -r --skip http://localhost:3000/client.js | tap-spot",
    "test-links:prod": "hyperlink https://kylemacquarrie.co.uk -r --skip http://localhost:3000 --skip http://localhost:3000/client.js | tap-spot"
  }
}

This works, but is a bit magicky, hard to maintain, and clutters the package.json. It clearly fails on point three.

Spawn

My next thought was to try and use Node’s child_process.spawn, similar to the ps ax | grep ssh example from the docs:

// test-links/index.mjs
import { spawn } from 'node:child_process'

// pass the options to hyperlink
const hyperlink = spawn('hyperlink', [
  'http://localhost:3000',
  '-r',
  '--skip',
  'http://localhost:3000/client.js',
])

const tapSpot = spawn('tap-spot', [])

// pipe hyperlink output into tap-spot
hyperlink.stdout.on('data', (data) => {
  tapSpot.stdin.write(data)
})

hyperlink.on('close', (code) => {
  tapSpot.stdin.end()
})

// log out tap-spot's output
tapSpot.stdout.on('data', (data) => {
  console.log(data.toString())
})

Running node test-links from an npm script does work, and we could use an environment variable (e.g. NODE_ENV) and add some logic to change the variables based on it. Unfortunately, this ruins tap-spot’s nice output formatting, and no amount of tinkering with the stdio option seems to help.

Hardcoded shell scripts

It occurred to me that I could do the same kind of thing that Homebrew uses for their install, which is roughly “get a shell script from somewhere and run it”. We can do something similar, by putting the commands in test-links/dev.sh and test-links/prod.sh, e.g.:

# test-links/dev.sh
hyperlink http://localhost:3000 -r --skip http://localhost:3000/client.js --skip | tap-spot

and updating our scripts in package.json:

"scripts": {
  "test-links:dev": "bash test-links/dev.sh",
  "test-links:prod": "bash test-links/prod.sh"
}

That works, and it’s tidied up our scripts nicely, but we’re back to maintaining two separate commands as life’s too short for me to learn how to write proper bash scripts with logic and stuff.

Using Node to generate shell scripts

A language I do know how to write logic and stuff with is JavaScript, maybe I can use that? I’ll also move it into a scripts folder to avoid naming clashes.

// scripts/test-links/index.mjs
const isProduction = process.env.NODE_ENV === 'production'

const url = isProduction
  ? 'https://kylemacquarrie.co.uk'
  : 'http://localhost:3000'

// build up the list of `--skip ${url}` commands
const skipUrls = [
  'http://localhost:3000/client.js',
  isProduction && 'http://localhost:3000',
]
  .filter(Boolean)
  .reduce((acc, curr) => `${acc}--skip ${curr} `, '')

// use npx for node modules we don't have as dependencies in the project
process.stdout.write(`npx hyperlink ${url} -r ${skipUrls} | npx tap-spot`)

This lets us update our scripts to match:

"scripts": {
  "test-links": "node scripts/test-links | bash",
  "test-links:prod": "NODE_ENV=production npm run test-links"
}

Using process.stdout.write() means we can just pipe it straight into bash, and it runs and keeps tap-spot’s nice formatting.

I’ve used npx to run the commands so we’re not relying on having them globally. It might be better in the long run to give test-links its own package.json and explicitly install them there, separately from the main site, but this should do for now, and I think it meets all the requirements set out earlier.

← All articles