kylemacquarrie.co.uk

Styling a Description List with CSS Grid

Thu, 15 Jun 2023 00:00:00 GMT

import DescriptionList from './_DescriptionList.tsx'

An underrated HTML element is the <dl> description list, ideal for doing key: value style lists of things when a one-dimensional <ul> doesn't quite cut it.

It consists of a description list <dl> tag, which contains <dt> (description term) and <dd> (description details), most often in matched pairs which is what we'll deal with here.

When styling a list like this, you often want some kind of visual layout that looks something like a table; however, since there's no wrapper element for each key/value pair (like a <tr> table row element), this has historically not been so simple. Fortunately, with CSS Grid this becomes much simpler. (Technically you can use a <div> to wrap each key/value group, but now we don't have to!)

Here's the HTML we'll be styling:

<dl>
  <dt>description title</dt>
  <dd>description data</dd>
  <dt>short</dt>
  <dd>short data</dd>
  <dt>A long title that takes up more space</dt>
  <dd>short</dd>
</dl>

And some basic styling:

dl,
dt,
dd {
  border: 1px dotted currentColor;
}

dt,
dd {
  padding: 0.25em;
}

dt::after {
  /* gives us a title: description format  */
  content: ':';
}

Default layout

Without any extra styling, everything has display: block by default. The : we added as a pseudo-element and the browser default indentation on the description details are the only visual indication of how the items relate to each other. So far, so brutalist.

Old Style Floats

We can tidy that up a bit using floats to give us a table row type layout, but the <dt>s all have different sizes, so the alignment of items looks a bit random; it's not easy to scan the list quickly. You also have to do some kind of clearfix or risk other bits of the layout going a bit mad.

You could tidy it up a bit by fixing the width of the <dt> but it's always going to be a compromise, especially if you don't know what data you're going to be dealing with ahead of time.

dl {
  overflow: hidden; /* clearfix hack */
}

dd {
  margin: 0; /* remove browser default margin */
}

dt,
dd {
  float: left;
}

dt {
  clear: both;
}

Modern CSS Grid

With CSS grid, we can get our table-style layout to fully align the titles, without having to hard code a width, using less code, and with less chance of doing something unexpected. auto 1fr sets the first column to fit the content and the second to use all the remaining space in the row. You get responsive behaviour out of the box, without having to special case anything.

dl {
  display: grid;
  grid-template-columns: auto 1fr;
}

dd {
  margin: 0; /* remove browser default margin */
}

Being able to nail that layout, without requiring any extra wrapper elements, with just a few lines of CSS would have been unimaginable not very long ago. I've not had much cause or opportunity to really dig into CSS Grid properly yet but just this simple use case has really piqued my interest.

Optimising webfonts with Glyphhanger

Sun, 16 Apr 2023 00:00:00 GMT

So Webfonts are cool, but you know what else is cool? Not sending loads of bytes over the wire that aren't required to display a web page. I remember hearing about Glyphhanger many years ago but I could never figure out how to get it to work in a reasonable way. Seeing Simon Dann posting about doing it with Eleventy reminded me of it, and I finally managed to get it set up in a reasonable(?) fashion.

There are some interesting tradeoffs around how and when to run this process - I've optimised for fast builds, at the expense of having to run the subsetting script manually and commit the output. If I forget and a couple of characters end up in the wrong font, it's not the end of the world.

Installing in WSL

I'm using WSL2 in Windows 11, which generally works very well but has occasional quirks. The Glyphhanger documentation only really covers MacOS, so here's how I went about getting it working. I wouldn't try this under native Windows, but if I was setting this up on MacOS I'd recommend using Homebrew there too.

Install Puppeteer dependencies with apt

(This step should be Linux/WSL only, I expect this to work out the box on MacOS.) Glyphhanger uses Puppeteer under the hood (there is a JSDOM option (--jsdom) but it still seems to require Puppeteer to be installed - possibly a bug?). Since Puppeteer is driving a headless Chrome instance, you need to be able to install and Chrome. So based on the troubleshooting guide, install the required dependencies:

sudo apt install libgtk-3-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2

Install fonttools with brew

Glyphhanger also uses fonttools, which is a Python package. I would rather saw off my own hand than try and navigate the minefield of getting Python working by hand, but fortunately some kind soul thought to put it on Homebrew, which works on Linux (including WSL) as well.

brew install fonttools

I cannot reiterate this enough. Getting a Python environment set up is the closest to a nervous breakdown a computer has ever brought me. Just use Homebrew.

Getting URLs

Before we can subset the font, we need to get a list of all the characters we need to include in the subset. If you only have a couple of pages you can pass them directly on the CLI:

glyphhanger http://kylemacquarrie.co.uk http://kylemacquarrie.co.uk/blog --subset=./path/to/fonts/*.ttf --output=./path/to/fonts/subset --formats=woff2

If you have a larger site, you can pass a single URL (e.g. your home page) and --spider, which will recursively follow all relative links it finds. Unfortunately this is slightly fragile - on my site, it follows the link to the RSS feed then blows up because Puppeteer can't execute anything on that page; there are other issues reported on Github relating to href:tel phone number links and the like. The current filthy hack I'm using is to parse the sitemap and pass a list of every URL in. I think I prefer this to having to comment out the RSS link, or adding special code into the application to handle this (e.g. hide the RSS link based on an environment variable or similar), but it's not ideal and won't scale well if you have more or less than one sitemap.

It's also worth noting you have to use TTF files as the input - trying to subset an existing WOFF2 won't work.

Subset

Here's the script I'm using to build up and run the CLI command.

// scripts/subset/index.mjs
import { readFile } from 'node:fs/promises'
import { resolve } from 'node:path'

// read the sitemap XML
const sitemap = await readFile(resolve('./dist/sitemap-0.xml')).then((data) =>
  data.toString()
)

// split into lines like <url><loc>https://kylemacquarrie.co.uk/blog/2/</loc></url>
const lines = sitemap.split('<url>')

// discard the first line which is all XML metadata
lines.shift()

// for each line, strip everything except the URL
const urls = lines
  .map((line) =>
    line
      .replace('<loc>', '')
      .replace('</urlset>', '')
      .replace('</loc></url>', '')
      // the sitemap builds with the production url but we're running this locally
      .replace('https://kylemacquarrie.co.uk', 'http://localhost:3000')
  )
  // make it into one big string
  .join(' ')

const fontPath = `./src/assets/fonts`

// build the command
const command = [
  'glyphhanger',
  urls,
  `--subset=${fontPath}/*.ttf`,
  `--output=${fontPath}/subset`,
  '--formats=woff2',
].join(' ')

// write it to stdout so we can pipe it into bash in the next step
process.stdout.write(command)

Then run it with node scripts/subset/index.mjs | bash or add it as an npm script

{ "scripts": { "subset": "node scripts/subset/index.mjs | bash" } }

and run it with npm run subset.

Results

The total size of the woff2 files went from 65kb to 28kb. I did have to adjust some line-heights as the conversion that FontSquirrel uses is obviously a bit different, but nothing too major.

Next Steps

That's a pretty decent saving. We could go further and use the --family option to only use characters that use a specific font, but we'd have to run the script once for each font we use which gets a bit tedious. It might be worth it for the smallest possible size though.

You could automate this as part of your build process, but as with my previous scripts, I don't really want to add additional dependencies that will slow down the build (especially if it might involve debugging a Python installation on a machine I don't own) so for now I'm just running it manually against a local version - Astro's default scripts make it easy to check the production build locally by doing npm run build && npm run preview.

It should be possible to refactor the script to cache the unicode range between runs as well, and only regenerate the fonts when a character has been added/removed. Running it against the site's markdown content might be better too.

Running shell scripts with Node

Sat, 25 Feb 2023 00:00:00 GMT

Since this site has been around for quite a while now in various incarnations, I've been thinking for a while about how to audit all the things I've linked to and make sure the links exist (and also that I'm not introducing breaking changes as I develop it). Fortunately before I ended up trying to write something myself, I came across Hyperlink, which does the job. (In the process I discovered a lot of 404s, some bad #fragment links, and a distressing number of redirects from http => https, or /path to /path/ or vice versa without any apparent consistency.) The basic usage is well documented in the readme and the intro post, but it took me a bit of experimentation to get it running the way I wanted.

The basic command I landed on is something like this:

$ hyperlink http://localhost:3000 -r --skip http://localhost:3000/client.js | tap-spot

That is, run hyperlink recursively against my local server, skipping a particular URL that fails (it's an example link in a tutorial), then pipe the output into a test reporter.

I had a few requirements:

it should be easily run with just one or two commands - I don't want to have to memorise the list of urls to --skip
it should run with different configurations (e.g. run against local build and run against production site)
it should not involve hard-coding long cryptic commands
it must not increase my install/build time

Hardcoded NPM scripts

The obvious solution to me was to add some scripts into the package.json and hard code the different configs in different scripts. To start with we can install the required packages globally - not ideal, but better than wasting CPU time and bandwidth for something I only want to run locally.

{
  "scripts" {
    "test-links": "hyperlink http://localhost:3000 -r --skip http://localhost:3000/client.js | tap-spot",
    "test-links:prod": "hyperlink https://kylemacquarrie.co.uk -r --skip http://localhost:3000 --skip http://localhost:3000/client.js | tap-spot"
  }
}

This works, but is a bit magicky, hard to maintain, and clutters the package.json. It clearly fails on point three.

Spawn

My next thought was to try and use Node's child_process.spawn, similar to the ps ax | grep ssh example from the docs:

// test-links/index.mjs
import { spawn } from 'node:child_process'

// pass the options to hyperlink
const hyperlink = spawn('hyperlink', [
  'http://localhost:3000',
  '-r',
  '--skip',
  'http://localhost:3000/client.js',
])

const tapSpot = spawn('tap-spot', [])

// pipe hyperlink output into tap-spot
hyperlink.stdout.on('data', (data) => {
  tapSpot.stdin.write(data)
})

hyperlink.on('close', (code) => {
  tapSpot.stdin.end()
})

// log out tap-spot's output
tapSpot.stdout.on('data', (data) => {
  console.log(data.toString())
})

Running node test-links from an npm script does work, and we could use an environment variable (e.g. NODE_ENV) and add some logic to change the variables based on it. Unfortunately, this ruins tap-spot's nice output formatting, and no amount of tinkering with the stdio option seems to help.

Hardcoded shell scripts

It occurred to me that I could do the same kind of thing that Homebrew uses for their install, which is roughly "get a shell script from somewhere and run it". We can do something similar, by putting the commands in test-links/dev.sh and test-links/prod.sh, e.g.:

# test-links/dev.sh
hyperlink http://localhost:3000 -r --skip http://localhost:3000/client.js --skip | tap-spot

and updating our scripts in package.json:

"scripts": {
  "test-links:dev": "bash test-links/dev.sh",
  "test-links:prod": "bash test-links/prod.sh"
}

That works, and it's tidied up our scripts nicely, but we're back to maintaining two separate commands as life's too short for me to learn how to write proper bash scripts with logic and stuff.

Using Node to generate shell scripts

A language I do know how to write logic and stuff with is JavaScript, maybe I can use that? I'll also move it into a scripts folder to avoid naming clashes.

// scripts/test-links/index.mjs
const isProduction = process.env.NODE_ENV === 'production'

const url = isProduction
  ? 'https://kylemacquarrie.co.uk'
  : 'http://localhost:3000'

// build up the list of `--skip ${url}` commands
const skipUrls = [
  'http://localhost:3000/client.js',
  isProduction && 'http://localhost:3000',
]
  .filter(Boolean)
  .reduce((acc, curr) => `${acc}--skip ${curr} `, '')

// use npx for node modules we don't have as dependencies in the project
process.stdout.write(`npx hyperlink ${url} -r ${skipUrls} | npx tap-spot`)

This lets us update our scripts to match:

"scripts": {
  "test-links": "node scripts/test-links | bash",
  "test-links:prod": "NODE_ENV=production npm run test-links"
}

Using process.stdout.write() means we can just pipe it straight into bash, and it runs and keeps tap-spot's nice formatting.

I've used npx to run the commands so we're not relying on having them globally. It might be better in the long run to give test-links its own package.json and explicitly install them there, separately from the main site, but this should do for now, and I think it meets all the requirements set out earlier.

VS Code tip: Match all lines starting with a certain string

Sat, 31 Dec 2022 00:00:00 GMT

Sometimes you want to batch-edit files to remove certain lines, where you know what the line starts with but not how it ends. In my case I had a legacy slug value in the frontmatter of a bunch of Markdown files, like this:

title: A Cool Post
slug: a-cool-post
description: A brief summary of the cool post

I didn't want to have to remove that by hand in 60+ files, so I figured there had to be a way to use the regex mode in VS Code's find & replace to do it. I am far from fluent in regular expressions, so it took me a little while to work this out, and I'm posting it here to refer back to in future, if nothing else.

Using e.g. slug: (.*)\n will match every line beginning with slug: , up to and including the new line character.

slug: match the exact string (amend to whatever string you're searching for)
(.*) match any number of characters, including whitespace
\n match a new line character (i.e. the end of the line)

Then you can replace with an empty string to remove the line.

If you omit the \n and do the same, it leaves an empty line, which I really wanted to avoid.

You can test-drive this on regex101.com.

Porting Content Between Site Versions

Wed, 21 Dec 2022 00:00:00 GMT

Over the years I've had a number of different personal websites (not even counting the number of abortive half-built efforts that got abandoned along the way), but each one has started from the same basic content as the previous iteration. Here are some of the scripts and tricks I've used for migrating content between different systems. This involved a large degree of fucking around && finding out, so let's consolidate it from all the repos on my machine into one place for future reference.

Caveats

Many of the implementation details in these scripts are hyper-specific to my use case, but the principles should generalise. I think it's useful to show how I've solved some of these problems by writing one-off scripts, not everything needs to be a clean reusable abstraction. It's fine to hack together a quick script to get the job done, even if it doesn't handle errors or edge cases correctly. My Ruby, in particular, is very much hack level.

1a: Stacey to ActiveRecord

At first I had a Stacey site for my portfolio, with a separate Wordpress site for my blog (see next section). Stacey uses a YAML file with a content key that you put markdown or HTML in for each post (similar to the markdown + YAML frontmatter format that things like Eleventy use.), e.g.

title: Post Title
date: 2022
description: Quick description of the post
tag: tag1, tag2
content: |
  Some markdown or HTML content

I was porting to a Padrino site using ActiveRecord as an ORM, so using a rake task seemed like a reasonable solution. The Post model would store HTML in its content field, and could also have and belong to many tags. Once the Post and Tag models are defined, and the Stacey content copied to /app/data/portfolio, we can add an import.rake task to loop over the folder and add a new post to the database for each item, as well as any tags required.

require 'yaml'
require 'kramdown'

# define the import task
task :import => :environment do
  portfolio_import
end

def portfolio_import
  Dir.foreach('./app/data/portfolio') do |item|
    next if item == '.' or item == '..'
    # load the yml file
    p = YAML.load_file("./app/data/portfolio/#{item}/project.yml")
    # get the content field
    raw_content = p['content']
    # Everything was inside ULs in the content, so we could tell if
    # the content was markdown or HTML by checking the first character
    if raw_content.first == '-'
    # it's Markdown, convert to HTML
      content = Kramdown::Document.new(raw_content, entity_output: :as_char).to_html
    else
      # just use the HTML as is
      content = raw_content
    end
    # get the list of tags, split on , and filter any empty items
    tags = p['tag'].downcase.gsub(',', ' ').split(' ').reject(&:blank?)

    # build an active record collection of tags
    tag_collection = []
    tags.each do |tag|
      # find or create the tag, and add it to the collection
      t = Tag.find_or_create_by(title: tag)
      tag_collection << t
    end
    # create a new Post
    post = Post.new(
      title: p['title'],
      slug: item.to_s.split('.').last,
      publish_date: (Date.new(p['date']) rescue nil),
      content: content,
      post_type: 'project',
      status: 'publish',
      tags: tag_collection
    )
    # save it
    post.save
  end
end

Running bundle exec padrino rake import would run the task. Job half done.

1b: WordPress XML to ActiveRecord

Next we want to take a WordPress XML export and do the same process. Turns out nokogiri is useful for something other than making life impossible for people trying to use Ruby on Windows.

# add to the earlier requires
require 'nokogiri'

# add to the import task
task :import => :environment do
  portfolio_import
  blog_import
end

def blog_import
  # import and parse the xml
  archive = File.open("./app/data/archive.xml") { |f| Nokogiri::XML(f) }
  # get each item (page or post)
  archive.css('item').each do |item|
    # we only want posts
    if (item.css('wp|post_type').first.content rescue nil) == 'post'
      # parse the post's tag names into an array of strings
      tags = []
      item.css('category').each do |tag|
        if tag.attributes['domain'].to_s == 'post_tag'
          tags << tag.attributes['nicename']
        end
      end

      # build an active record collection of tags
      tag_collection = []
      tags.each do |tag|
        t = Tag.find_or_create_by(title: tag.value)
        tag_collection << t
      end

      # create a new Post and populate with data from the XML.
      post = Post.new(
        title: (item.css('title').first.content rescue nil),
        slug: (item.css('title').first.content.parameterize.dasherize rescue nil),
        publish_date: (Date.parse(item.css('pubDate').first.content) rescue nil),
        # WP stores the content as HTML
        content: (item.css('content|encoded').first.content rescue nil),
        # published/draft
        status: (item.css('wp|status').first.content rescue nil),
        post_type: 'blog',
        tags: tag_collection
      )
      # save it to the DB
      post.save
    end
  end
end

Now running bundle exec padrino rake import will import the blog entries into the database.

2: Active Record to Markdown files

Fast forward a few years, and I've decided I'd like to move to a static site. At various times this was going to be in Next.js, Eleventy and eventually Astro, all of which can build pages from Markdown files. So we want to export all the posts from the database into correctly formatted Markdown, and also grab their associated poster images from the Asset model (which was using paperclip under the hood for file uploading).

This is largely the same process as the import, but in reverse

require 'kramdown'
require 'open-uri'

task :export => :environment do
  posts_export
  assets_export
end

def assets_export
  # for each Asset in the database
  Asset.all.each do |asset|
    # create a local file with the right name
    File.open("images/#{asset.file.url.split('/').last}", 'wb') do |fo|
      # read the file from the server and write to the local file
      fo.write open("https://kylemacquarrie.co.uk#{asset.file.url}").read
    end
  end
end

def posts_export
  Post.all.each do |post|
    # ignore contacts
    next if post.post_type == 'contact'

    slug = post.slug.chomp('/')
    post_type = "#{post.post_type}#{ post.post_type == 'blog' ? '' : 's'}"
    # parse html content back to markdown
    markdown = Kramdown::Document.new(post.content, input: 'html').to_kramdown rescue ''
    # remove attributes that the janky wysiwyg editor added
    markdown = markdown.gsub('{: target="_blank"}', '')
    markdown = markdown.gsub('{: .ql-syntax spellcheck="false"}', '')
    # construct a string in the correct markdown + frontmatter format
    string = "---
title: '#{post.title}'
abstract: \"#{post.abstract}\"
status: #{post.status}
published: #{post.publish_date}
tags: #{post.tags.all.map{|t| t.title}.join(',')}
image: #{post.main_asset.file.url.split('/').last rescue ''}
position: #{post.position}
---

#{markdown}
"
    # write to a new markdown file with the correct path
    File.open("posts/#{post_type}/#{slug}.md", 'wb') do |fo|
      fo.write string
    end
  end
end

Running bundle exec padrino rake export gives us a couple of folders of Markdown files, and a folder of all the images that we ever uploaded in the Padrino Admin CMS, ready to dump into the fancy new site's git repo.

3: Rewrite Flickr hotlinks to local files

For historical reasons (because my original shared hosting didn't have much space) most of the images that were in the site content were hotlinked from my Flickr account (remember them?). I was keen to remove that external dependency. Fortunately Flickr allows you to download your archive as a zip file, so we just need to reverse-engineer a way of mapping the CDN link to the original file.

The Flickr URLs looked like this:

https://farm6.staticflickr.com/5537/14063836058_5a91d05c09_b.jpg

The matching image in the zip file would be something like

/path/to/archive/original_filename_14063845389_o.jpg

Unfortunately that still leaves us some work to do to map the CDN link back to the original image.

// import.js
import { readdirSync, readFileSync, copyFileSync, writeFileSync } from 'fs'

function main() {
  // identify Flickr CDN links. The first 10+ digit section appears to be the original image ID
  // e.g. https://farm6.staticflickr.com/5537/14063836058_5a91d05c09_b.jpg
  const regex =
    /https:\/\/farm\d.static.?flickr.com\/\d{4}\/\d{10,}_.{10,}.jpg/g
  // get a list of all the images in the flickr archive
  const images = readdirSync('./flickr_archive')
    // filter out WSL guff - MacOS may need to remove .DS_Store etc
    .filter((file) => !file.includes('Zone.Identifier'))

  function doFolder(type) {
    // in this case we have two folders, /blog and /projects.
    // get a list of all posts from that folder
    const posts = readdirSync(`./posts/${type}`)

    posts.forEach((postName) => {
      const postPath = `./posts/${type}/${postName}`
      // read the post content as a string
      let post = readFileSync(postPath).toString()
      // find any flickr cdn URLs
      const flickrLinks = post.matchAll(regex)
      // for each flickr
      for (const match of flickrLinks) {
        // extract the ID
        const url = match[0]
        const parts = url.replace('https://', '').split('/')
        const id = parts[parts.length - 1].split('_')[0]
        // find the image with a matching ID
        const img = images.find((i) => i.match(id))
        if (!img) {
          console.log(`couldn't find an image matching ${id} in ${postPath}`)
        }
        // copy file to new location
        const from = `./flickr_archive/${img}`
        const to = `./public/images/${img}`
        console.log(`copying ${from} to ${to}`)
        copyFileSync(from, to)

        // update URL in file
        const newUrl = `/images/${img}`
        console.log(`replacing ${url} with ${newUrl}`)
        post = post.replace(url, newUrl)
      }
      // write the post back to disk
      writeFileSync(postPath, post)
    })
  }

  doFolder('blog')
  doFolder('projects')
}

main()

Running node import.js gives us a folder of images that are linked from the content, and updates the links to point to the new paths.

Install glslang using Homebrew

Thu, 15 Dec 2022 00:00:00 GMT

I've been going through Bruno Simon's great Three.js Journey course lately. There's a section on shaders, where Bruno recommends the GLSL Linter extension for VS Code, which requires the OpenGL Reference Compiler to be installed and the path to be set in the extension settings. The video he references uses a convoluted manual process, but if you're able to use Homebrew it becomes much simpler. I've tested this on Ubuntu running in WSL, but it should be the same for Mac OS and other Linux flavours.

Install using Homebrew:

$ brew install glslang

Check it's installed and in your PATH - note the capital V in the command name

$ glslangValidator -v

Get the path to the installed binary

$ which glslangValidator

Drop that path into the Glsl-linter: Validator Path setting in VS Code and the extension should pick it up correctly — you may need to reload VS Code.

If you're following Bruno's naming scheme (e.g. fragment.glsl and vertex.glsl), you may run into this issue where it can't work out which shader type type to use to validate. You can either rename the shader files (e.g. fragment.frag and vertex.vert should work out the box) or add fragment.glsl and vertex.glsl to the Glsl-linter: File Extensions setting.

"glsl-linter.fileExtensions": {
  "fragment.glsl": "frag",
  "vertex.glsl": "vert"
}

Guillermo del Toro and the essence of heavy metal

Wed, 02 Dec 2020 00:00:00 GMT

Pacific Rim is a ridiculous movie about giant mind-controlled mechs rocket-punching enormous kaiju from another world. It was directed by Guillermo del Toro, and in the director's commentary he had this to say:

Ishirō Honda, when he started Gojira, he said to his crew, he gathered everybody in the kitchen of his home and he said "listen guys, if anyone here doesn't believe we're doing a great movie, that we're making a great movie with a giant monster, please leave", and I demand of myself and my crew and my cast the same thing. We have to be unironic. We have to never be postmodern about these things we do. What you see is an exercise in faith and an exercise in law, and I deliver myself completely to every movie I do without a single shred of irony and I wanted my cast to feel that way because you can feel when people don't believe what they're doing. You can sense when people don't believe in that.

I was reminded of this by this tweet from noted metal blog Invisible Oranges, referring to the extraordinary Slugdge:

gastropod-themed progressive death metal band slugdge prove not only that the genre can be ridiculous and savage at the same time, but also that this synchrony makes the music even better

This, to me, is the essence of heavy metal. If you stop to think too hard about e.g. men yelling about vikings over 200bpm blastbeats, the whole thing falls apart, but bypass niceties like "good taste" and you can feel the energy. (This is hardly unique to metal, as anyone who's been reluctantly dragged to a club, only to find themselves lost to the dancefloor might attest.) Cast aside your cynicism; be unironic and un-postmodern, deliver yourself completely to the music, and you'll be rewarded.

While I'm hesitant to make any grand statements about how films like Pacific Rim and the best heavy metal qualify as art, they are absolutely full of craft, and that craft shines through and connects in the most visceral way.

Build a server-rendered Preact app from scratch

Thu, 02 Apr 2020 00:00:00 GMT

In this tutorial, we're going to build a simple server-rendered Preact app, with client-side hydration. We'll start by serving plain HTML with Express, then build up some components using Preact and HTM. After that, we'll add some interactivity by hydrating our components, matching up the server rendered HTML with what the client-side framework thinks it should be. Along the way we'll have some fun (?) with build tools like Nodemon, Rollup, and NPM scripts.

None of the individual parts of this are (relatively speaking) that complex, but there are a lot of moving parts here and it can be hard to find an example that puts them all together. This is an intermediate level tutorial though, so I won't be explaining much of the syntax or JavaScript language features. You'll need to be reasonably comfortable with the terminal, NPM and JavaScript basics.

Why server rendered? Why Preact?

Basically, it's fast. Sending as much as you can via HTML means less work to do on the client to make it work; using a lightweight framework like Preact means you're sending less JavaScript on the wire, which translates to a faster user experience, especially on slower devices (i.e. most Android phones - start with this article by Alex Russell if you want to delve into the numbers). For content-heavy sites, it's almost invariably better to get as much of your content onscreen in HTML as quickly as possible. A lot of JavaScript framework SSR approaches involve building a single-page app then hacking in server rendering, but we're going to start with plain HTML and try to progressively enhance it.

It's also a good exercise in seeing what's going on under the hood -- in my day job I work on a moderately complex React/Next.js application, and there's a lot of magic that goes on under the surface. Understanding what your tools are doing for you is always useful.

An express app that serves HTML
Setting up Nodemon and NPM scripts
Setting up Rollup
Using Preact to render to HTML
Components and Composition
Keeping hydrated
Making it interactive

An Express app that serves HTML

Let's start with a minimum viable Express server. Create a new folder and switch into it, then initialise NPM - the default options are fine.

$ mkdir preact-ssr
$ cd preact-ssr
$ npm init

If you're using git for version control, you probably want to add the node_modules folder to your .gitignore file at this point.

Now we'll install some dependencies we need to get started.

$ npm install express compression

Strictly speaking we don't need compression, but one of reason we've picked Preact and server-rendering is to squeeze out as much performance as possible, so let's roll with the best practice here and save a few bytes at basically no cost to us.

Create a src folder, with a server.js file inside it.

$ mkdir src
$ touch src/server.js

Let's set up our basic app in src/server.js to serve some static HTML.

const express = require('express')
const compression = require('compression')

const app = express() // create the express app
app.use(compression()) // use gzip for all requests

// some basic html to show
const layout = `
  <!DOCTYPE html>
  <html>
    <body>
      <h1>Hello</h1>
    </body>
  </html>
`

app.get('/', (request, response) => {
  // listen for requests to the root path
  response.send(layout) // send the HTML string
})

app.listen(3000) // listen for requests on port 3000

Run the app:

$ node src/server.js

Then visit http://localhost:3000. All being well, you should see the HTML rendered out. Check out the demo repo at this point if you need to check what it should look like.

There is a slight issue here. Try changing the text in the <h1> tag and refresh your page. You should see that it doesn't update until you kill your server (using ctrl + c in your terminal) and restart it. That will get annoying fast, so let's take the time now to fix our workflow.

Setting up Nodemon and NPM scripts

Nodemon is a tool that listens for changes to specified files and restarts the Node process when it sees one. Let's add it as a development dependency.

$ npm install nodemon --save-dev

We'll add an NPM script so we don't have to remember the right incantation every time. You can do this all via the CLI if you like, but let's automate as much as we can. In the "scripts" block of your package.json, let's add a new entry.

"scripts": {
  "nodemon": "nodemon --watch src/server.js src/server.js"
}

This translates roughly to "run node src/server.js, but also watch src/server.js and rerun the command whenever you see that file has changed".

You should now be able to start the app using npm run nodemon, change some text in the <h1> and refresh to see it in the browser immediately. All being well, your app should look something like the demo at this commit.

Setting up Rollup

So far, we've been using Node's require syntax to load dependencies, but we're going to want to add packages that are designed to use ES6's import. Let's use a bundler to paper over the differences between them so we can avoid gazing too deeply into that particular abyss. Webpack is a popular bundler, but configuring it is baffling at the best of times, so we'll go with Rollup. We don't need much, just the ability to use both require and import as needed, on both the server and the client. How hard can it be, right?

First up, we'll install rollup, as well as the node-resolve plugin.

npm install --save-dev rollup @rollup/plugin-node-resolve

Next, let's tell Rollup to take our src/server.js file and compile it into build/server.js. Create a rollup.config.js file at the root of your project. Note that we're exporting an array of config objects, as we'll be adding a client bundle soon. I'd also recommend adding the build folder to your .gitignore file, although it's not mandatory - some people prefer to check in the built files.

// rollup.config.js

import resolve from '@rollup/plugin-node-resolve'

export default [
  {
    input: 'src/server.js', // take our source file
    output: {
      file: 'build/server.js', // compile it into this file
      format: 'cjs', // use the CommonJS format, which works with Node
    },
    plugins: [resolve()], // use the node-resolve plugin so dependencies get imported properly
  },
]

Add another script to the package.json for Rollup. This tells Rollup to use the config file we just created, watch for changes to imported files and recompile when it sees a change.

"rollup": "rollup --config --watch",

Update the Nodemon script to use the built file, so it will restart the server every time Rollup finishes compiling.

"nodemon": "nodemon --watch build/server.js build/server.js"

Now we have two NPM scripts, rollup and nodemon, that we want to run in the background. At first glance, you might want to try $ npm run rollup && npm run nodemon; that won't work though, as the rollup watcher never exits, so the nodemon script never starts. We need a way to run them in parallel. You could do this by hand, but fortunately, there's a package for doing it in a single script. Let's install it.

$ npm install --save-dev npm-run-all

Then we'll add a start script that uses it to call both of the scripts we prepared earlier, in parallel.

"start": "npm-run-all --parallel nodemon rollup"

Now you can start both processes with a single command.

$ npm run start

There were quite a lot of moving parts in there, so compare against the demo repo at this point if something's not working.

If you have a look at your build/server.js file, it should look pretty much like src/server.js, but now we're all set up for importing ES6 modules, so let's do that.

Using Preact to render to HTML

Time to install some more dependencies. We'll install Preact itself, the Preact server-side renderer, and htm, which lets us use JavaScript's tagged template strings to build up our components (you can also use JSX, but that requires an additional compile step in your build process). If you've used JSX with React or another framework before, you'll probably have to be careful your muscle memory doesn't take over here; the syntax isn't that complicated, but it feels more like EJS (or ERB, for the Rubyists) than JSX sometimes.

$ npm install preact preact-render-to-string htm

In your src/server.js file, import what we need at the top of the file.

import render from 'preact-render-to-string'
import { html } from 'htm/preact' // use the provided preact binding

Generate some markup and render it to a string using the functions we imported from preact-render-to-string and htm.

const body = render(html`<h1>Hello from Preact</h1>`)

Then interpolate that variable into our final HTML document.

const layout = `
  <!DOCTYPE html>
  <html>
    <body>
      ${body}
    </body>
  </html>
`

Refresh your browser and you should see the text from your body variable, except now we've rendered it using Preact before turning it into HTML. Have a look at the repo at this point if not.

Components and Composition

One of the reasons Preact, React and other JavaScript frameworks are popular is because they make it easy to create small, separate components, then compose those components together to create larger applications. Let's refactor our single server.js file into components. We'll then render the whole component tree on the server, before adding the client-side JavaScript to make it interactive.

Create a src/components folder, and add List.js and PreactApp.js files to it.

$ mkdir src/components
$ touch src/components/List.js
$ touch src/components/PreactApp.js

90% of web development is looping over lists of things, so in src/components/List.js, we'll create a component that takes an array of data and renders it in a <ul>.

import { html } from 'htm/preact'

const List = ({ data }) => {
  // takes a data prop
  return html`
    <ul>
      <!-- loop over data array -->
      ${data.map(
        (i) => html`
          <li>
            <!-- render out each item -->
            ${i}
          </li>
        `
      )}
    </ul>
  `
}

export default List

In src/components/PreactApp.js, we'll import List.js and pass it some data.

import { html } from 'htm/preact'
import List from './List'

const dataArray = ['Item one', 'Item two', 'Item three']

const PreactApp = () => {
  return html` <${List} data=${dataArray} /> `
}

export default PreactApp

Then in src/server.js, we'll import src/components/PreactApp.js and render that to a string, instead of just writing it inline.

import PreactApp from './components/PreactApp'
...
const body = render(html`
  <h1>Hello from Preact</h1>
  <${PreactApp} />
`)

When you refresh, you should see the <h1>, followed by the list of data from our PreactApp and List components. Compare against the repo at this point if required.

Next up, let's add some interactivity to our List component.

Keeping hydrated

Hydration is the process of reconciling the server-rendered DOM structure with what our client-side app thinks should be happening. When a framework like Preact sets up an app, it works out what it thinks the DOM should look like and works out the most efficient set of changes to get it into that state, then adds things like event listeners to the right elements. When we use hydration, we're saying to Preact "Don't worry about working out what changes are required, the DOM's already in the right state. Just add your event listeners and carry on."

Before we can do that, we need to load some client-side JavaScript, as currently we're just sending HTML with no <script> tags in sight. We have to do a few things:

Create a source file for our client bundle
Configure Rollup to compile it into the right format for the browser
Configure Express to serve it as a static file
Add a <script> tag into the HTML Express is serving
Add some interactive elements into our app!

Start by creating the source file.

$ touch src/client.js

In src/client.js, import the PreactApp component, and tell Preact where in the DOM we want to consider as our app.

import { hydrate } from 'preact'
import PreactApp from './components/PreactApp'

hydrate(PreactApp(), document.getElementById('root'))

In src/server.js, add a <div> with a matching ID around the PreactApp component.

<div id="root">
  <${PreactApp} />
</div>

In rollup.config.js, add a second config object into the array. This has different settings from the server.js file, as the browser doesn't understand Node's CommonJS syntax natively.

import resolve from '@rollup/plugin-node-resolve'

export default [
  {
    input: 'src/server.js',
    output: {
      file: 'build/server.js',
      format: 'cjs', // CommonJS format for Node
    },
    plugins: [resolve()],
  },
  {
    input: 'src/client.js',
    output: {
      file: 'build/client.js',
      format: 'es', // ES Module format for modern browsers
      name: 'client',
    },
    plugins: [resolve()],
  },
]

Now when you run npm run start, you should see a client.js popping into the build folder alongside server.js.

Next, let's tell Express how to serve our client bundle as a static file.

app.get('/client.js', (request, response) => {
  response.sendFile('client.js', {
    root: __dirname, // this will be the build folder
  })
})

Now you can visit http://localhost:3000/client.js and see some compiled JavaScript. Let's pop that in a script tag in src/server.js.

const layout = `
  <!DOCTYPE html>
  <html>
    <body>
      ${body}
      <script type="module" src="client.js" async></script>
    </body>
  </html>
`

Now when you refresh http://localhost:3000, you have a working, hydrated Preact application that does... precisely nothing (if you don't, have a look at the example repo at this stage). Let's fix that.

Making it interactive

All we need to do now is add some functionality into our src/components/List.js component. On the server it gets compiled to HTML, while our client bundle will look out for the matching DOM elements and set up the required listeners when it hydrates. We'll add a button to each list item, another item to show how many times they were clicked, and we'll use Preact's useState hook to track that data.

import { html } from 'htm/preact'
import { useState } from 'preact/hooks'

const List = ({ data }) => {
  // takes a data prop
  // how many clicks have we counted? Default to 0
  const [count, setCount] = useState(0)

  // shared event handler
  const handleClick = () => {
    setCount(count + 1)
  }

  return html`
    <ul>
      ${data &&
      data.map(
        (i) => html`
          <li>
            <!-- listen for button clicks -->
            ${i}: <button onClick=${handleClick}>Click me</button>
          </li>
        `
      )}
      <li>
        <!-- list how many clicks we've seen, with the right plural -->
        ${count} ${count === 1 ? 'click' : 'clicks'} counted
      </li>
    </ul>
  `
}

export default List

You should now have a functional Preact application that still does as much of its work up-front on the server as possible. Again, if you're having issues, try comparing against the example repo. Hopefully that is a useful starting point - there's still a lot of low-hanging fruit that we could build into this (Things like compiling CSS, minifying our scripts, transpiling JavaScript using Babel, and rendering more complex state), but I'll save those for a future tutorial.

If you have feedback or suggestions, then hit me up on Twitter or open an issue on the repo.

Testing server-side Javascript with Jest

Wed, 30 Oct 2019 00:00:00 GMT

If you're using something like Next.js to render JavaScript on the server you might have written something along these lines, to get the user-agent string from the request object if run on the server, or from the global window object if we're in the browser. This will blow up on the server if you receive a request without a user-agent header. (I have no idea what is sending requests with no user-agent header but something was, and it was generating a lot of 500 errors in production.)

const userAgent = req.headers['user-agent'] || window.navigator.userAgent

That's a pretty naive implementation, so let's refactor it into a function. We can use some null propagation operators to catch undefined properties, and wrap it in a try/catch for good measure:

function getUserAgent(req) {
  try {
    return req?.headers?.['user-agent'] || window?.navigator?.userAgent
  } catch (err) {
    return ''
  }
}

Hardly the most elegant solution, but it does the job. To check it does in fact do the job, let's write some tests, using Jest:

it('returns the user agent header if one is supplied', () => {
  const result = getUserAgent({
    headers: {
      'user-agent': 'foo',
    },
  })
  expect(result).toBe('foo')
})

it("falls back to window.navigator if the header isn't present", () => {
  const result = getUserAgent({
    headers: {},
  })
  expect(result).toBe(window.navigator.userAgent)
})

OK, so far so good, but how can we test the case where there's no user-agent header and no window.navigator?

it('returns an empty string', () => {
  const result = getUserAgent({})
  expect(result).toBe('')
})

This will fail, since Jest runs with JSDOM by default. window is part of the global namespace, and just like in a real browser window.navigator is read-only, so you can't set it to undefined. I couldn't get Jest to mock it properly either.

The easiest solution seems to be to extract the server-side test cases to a separate file, where you can tell Jest it's running in a Node environment. Add the Jest environment pragma to the top of your server test files:

/**
 * @jest-environment node
 */

That way, JSDOM isn't even set up for that test suite and you can test weird isomorphic edge cases to your heart's content. The only other thing to watch out for is if you refer to JSDOM in your test setup files, you might need to catch that too:

if (typeof jsdom !== 'undefined') {
  jsdom.reconfigure({
    url: 'https://www.foo.com/bar',
  })
}

Actually, the best solution is not to read the user-agent string at all, but sometimes it has to be done :(.

Simple deploys with NPM and rsync

Tue, 20 Nov 2018 00:00:00 GMT

If you're building a JavaScript heavy static site (perhaps you're using something like vue-cli or create-react-app, or perhaps you're rolling your own Webpack config like I'm doing in the examples), using something like Capistrano for deployment is a lot of extra work, while messing about with FTP is annoying and error-prone. Let's split the difference, and use some simple tools to make life easier.

Requirements

A machine with rsync (any Mac, most (all?) flavours of Linux, Windows 10 Linux subsystem)
Node JS with npm (or yarn)
SSH access to your host

Initial implementation

Throughout the examples, I'm assuming that npm run build will compile your project into a project-name folder with an index.html, which you can then serve from the path of your choice on the server. To start with, we can run the build step, then manually run the command to copy the files to the remote host:

// package.json
"scripts": {
  "build": "webpack --config webpack.production.js"
}

// terminal:
$ rsync -avz --delete project-name deployuser@server.domain:/path/to/static/files/"

This works, but means you have to keep track of that deploy command and all those rsync flags are just begging for a typo. Let's automate it a little.

Make it a script

You can run shell commands from NPM scripts, so we'll clean that up into a single, self-documenting, easily typed command:

// package.json
"scripts": {
  "build": "webpack --config webpack.production.js",
  "deploy": "webpack --config webpack.production.js && rsync -avz --delete project-name deployuser@server.domain:/path/to/static/files/"
}

// terminal
$ npm run deploy

Scripts can call other scripts

NPM scripts can not only call shell commands, they can call other NPM scripts. We'll use a dedicated transfer script, and call that along with build when we deploy:

// package.json
"scripts": {
  "build": "webpack --config webpack.production.js",
  "transfer": "rsync -avz --delete project-name deployuser@server.domain:/path/to/static/files/",
  "deploy": "npm run build && npm run transfer"
}

// terminal
$ npm run deploy

Refactor paths into NPM variables

As a final step, if you're sharing your code publicly you might want to avoid hard-coding the deploy user and path into the repo. NPM lets you add additional global config variables via the .npmrc file, which defaults to ~/.npmrc. We can make our deploy path part of that config and refer to it in package.json with the $npm_config_ prefix:

// .npmrc
deploy_path = deployuser@server.domain:/path/to/static/files/

// package.json
"scripts": {
  "build": "webpack --config webpack.production.js",
  "transfer": "rsync -avz --delete project-name $npm_config_deploy_path",
  "deploy": "npm run build && npm run transfer"
}

// terminal
$ npm run deploy

You can also have an .npmrc file in your project directory, if you don't mind either checking it in to version control or adding it to your project's .gitignore file.

Bonus tip for Ruby users

It's worth noting you can do much the same thing using Ruby and rake instead of Node and npm.

# Rakefile

namespace :deploy do
  desc "Build the website from source"
  task :build do
    status = system("npm run build")
    puts status ? "OK" : "FAILED"
  end

  desc "Deploy website via rsync"
  task :push do
    status = system("rsync -avz --delete project-name deployuser@server.domain:/path/to/static/files/")
    puts status ? "OK" : "FAILED"
  end
end

desc "Build and deploy website"
  task :deploy => ["deploy:build", "deploy:push"] do
end

# terminal
$ rake deploy

If you're using NPM already I don't see much reason to use rake instead but it might be useful if you're using other Ruby tools.

Web Archaeology

Wed, 15 Jan 2014 00:00:00 GMT

There's something wonderful and magical about old web design books. Old is a relative term, of course: in this case, old is a mere thirteen years. But these are records of a bygone age, and most of the subjects have disappeared. Some survive, their old bones reshaped and dressed with new flesh to suit the modern age. Some remain untouched, like flies preserved in amber. But mostly, they are gone, and we can only infer their former existence with the help of secondary sources.

I'm drawn to these tomes. Despite their age, they have something of the New Aesthetic about them. Their subject matter is fundamentally ill-suited to being depicted on the printed page. Shorn of their interactivity, their links to each other, websites lose all the things which set them apart from print. Like trying to understand a ballet's choreography through half a dozen photos, you get fascinating flashes of an idea. The essence, though, is gone from them, as surely as the essence is gone from the stuffed animal relics in the museum.

The books they are contained in have become artefacts in their own right. Freed from the screen, they form a fascinating collage, contents apparently selected for aesthetic value rather than how well they convey any sense of information. Haphazardly scattered across the page, they transcend the limitations of their source material. A still from a grainy pre-Youtube video goes from a tiny window on a bulky CRT screen to a full page spread. Dithered and pixelated almost beyond recognition, they burst from the screen on to the page. They have a curious vitality to them — a reminder, perhaps, of the potential for expression this medium holds. These early pioneers, mapping that landscape, getting lost in the in the digital desert and dying of underexposure, so that others might find what's left behind and take up the quest anew, in a land safer for the sacrifices made.

At the collision of old and new media (and was there ever a term less appropriate? Perhaps it should be old media and less old media, in this case) we can dig through these documents, and marvel at the ingenuity of the ancients.

What can games learn from UX?

Sun, 08 Dec 2013 00:00:00 GMT

It started, as it often does, with a tweet. "What can UX learn from gaming?" asked @TheDrum, followed by a link to this piece. Frankly, anyone who believes games offer a good model for UX to imitate probably hasn't actually tried playing that many games, especially on the One True Format, although a cursory Twitter search for "PS3 Update" should demonstrate that consoles are far from immune from the scourge of thoughtless design. The question needs to be reversed — "What can gaming learn from UX?"

The answer, I think, is quite a lot.

On the web, we live in an age where people designing and building things want to make them work as well as possible, regardless of what device you're on. Even old browsers should still get most of what's on offer. Recently, the idea of Offline First design has taken root— that is, web apps which will still work and behave reasonably even on a flaky or non-existent connection. Let me repeat that: web apps that work with the connection the user has, not the connection you want them to have.

Meanwhile, in the land of AAA, multi-million dollar game releases:

Both of these games can be played single player, but require you to be constantly online regardless. Neither of these games have any particular need to be online; before they were released questions about the reasoning behind making them always online were raised, and went without substantive answers. Both had catastrophic clusterfucks for launches, with thousands of users unable to play the game they paid significant sums of money for until days and sometimes weeks later.

Where are the people in EA and Blizzard looking out for the users? They either don't exist, or they're being ignored by the people making the architectural decisions. Neither leaves a good taste.

EA is, of course, a serial and a serious offender. One of the things that prompted me to write this screed was John Walker's account on Rock Paper Shotgun of his attempt to play Battlefield 4's single player campaign. Even if we leave aside any server or connection issues (which we shouldn't, but bear with me) there are so many things wrong with this process it's difficult to know where to start. Requiring both your own digital distribution service and a fucking browser plugin just to launch the single player campaign is just such a bizarre idea to anyone outside of EA's senior management. There are so many hoops to jump through which add nothing to the experience of most players, and so many possible points of failure just waiting for a little server glitch to throw everything out of sync. Anyone familiar with design will probably have heard that "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away". This is pretty clearly the opposite of that.

It's worth thinking about how Origin fits in this context. Conceived as a competitor for Steam, it is fundamentally the result of EA's compulsion to squeeze every dollar possible from their customers. EA look at the revenue Steam generates for Valve, and think "I want more of that". Fair enough — they, unlike Valve, are a public company who have a responsibility to maximise shareholder profit. In this day and age, however, you cannot simply compete on features — you have to compete on experience. Steam, for all its many and well documented flaws, offers both superior features and a better experience.

Speaking of things Steam is better than, perhaps the crowning glory of inexplicably shit, totally broken game-related UX is the hateful bundle of inscrutable idiocy that comprises Games For Windows Live. While Microsoft are experts in the death by a thousand cuts approach to UX, GFWL was more like a savage beating of ineptitude. While it does now appear to have been taken out and put out of its misery, anyone who's had the misfortune of using the "service" which, like Origin, aimed to replicate most of the features of Steam, will probably have tales to tell of bizarre login loops, game-blocking client updates, server problems, saves going missing, and more. The sting in the tail here is, naturally, that it provides absolutely no added value to players.

Even absent massive cock-ups like GFWL, a litany of more minor complaints can be found in games of all shapes and sizes. Minor UI elements and affordances we take for granted on the web are absent or incomplete. Required password or content codes use fields that can't be pasted into. I'm not even getting into actual gameplay — I could probably write a book on crappy checkpointing alone, but fortunately for all concerned I'd probably have an aneurysm first.

There are glimmers of hope. Steam at least attempts to add value for the user — notably, by competing on price in their legendary sales. Elsewhere, the Humble Bundle retail experience is one of the best online shopping experiences I've had, and their Android app is a good alternative to manually installing a bunch of APKs. The actual gameplay of most games is strong enough to create compelling experiences which can't be replicated in any other medium.

Too often, though, layers of thoughtless decisions transform what should be an enjoyable experience into one that must be endured. If games really want to be seen as leaders in user experience, they need to look at the work being done in other areas of interactive design. More importantly, players need to have someone looking out for their interests during development. Otherwise we're doomed to repeat the mistakes of the past, stuck in GFWL login loop purgatory for ever.