The aim

There are many ways to provision your cloud architecture.

• Manually in Digital Ocean, AWS, Azure
• Docker, Kubernetes, or similar
• Puppet, Chef, Terraform
• Cloudformation or equivalent
• And so on …

Sometimes however you just want an easy, repeatable way to recreate a machine on demand with some stuff already installed and configured. You could be wanting to:

• Create a database server with your database roles and schemas already configured and that you can deploy at a moment’s notice if, for example, you need a fresh white-labelled server with default contents
• Create a simple application server running a self-contained and pre-configured Nginx, firewalls, Redis cache, Consul etc, so you can spin up a new instance ready to use with virtually zero notice
• Define a server with a selection of pre-configured dev apps/sites to support cloud working

Whatever it is you’re looking to accomplish, having a pre-defined pre-configured server with known versions, options, users, and roles ready to go can give you a cheap and simple point-and-click provisioning option without needing to do notable dev ops work.

This is similar to (if not exactly the same as) the way cloud providers and their third party partners create pre-built images for popular server stacks (eg AMIs in the Amazon Marketplace). In fact many of them use Packer to do it.

Pretty much anything you can do with a server is possible, but to keep the example simple we’ll just do Nginx running on Ubuntu 20 and serving a default site.

The tools

This example will be using:

• Packer - a great tool from Hashicorp, makers of Terraform, Vault, and Consul
• Digital Ocean - a cloud provider that offers good servers with very clear, fixed pricing

Packer will use the Digital Ocean API to:

• Spin up a machine
• Deploy and configure it
• Take a snapshot of the running result
• Shut it down again

The end result is a snapshot ready to serve as a template for quickly creating future Droplets (servers).

Getting a Digital Ocean API key

You first need to go to (or sign up free for) your Digital Ocean account and in the menu choose ACCOUNT, API. In there, under Applications & API, Tokens/Keys, Personal access tokens, click to Generate New Token.

Name your token (eg packer), select Write, and click Generate Token. Your new token then appears on the screen (it looks like a long random hexadecimal string).

Click the Copy link next to it and then store it safely (eg Hashicorp Vault or even LastPass or similar). This key lets anyone deploy and run anything on your account (at your expense), so never commit it to any form of source control or insecure filesystem/cloud store.

Start a command prompt or terminal window and set your token as an environment variable. For Windows, replace export with set below.

export DIGITALOCEAN_API_TOKEN=d824...etc...3398

Creating a Packer server definition

You now need a JSON file to define the server. My file is named nginx.packer.json. Use the same name or change the commands below.

{
  "builders": [
    {
      "type": "digitalocean",
      "ssh_username": "root",
      "api_token": "{{ user `DIGITALOCEAN_API_TOKEN` }}",
      "image": "ubuntu-20-04-x64",
      "region": "lon1",
      "size": "s-1vcpu-1gb",
      "droplet_name": "packer-nginx-ubuntu20",
      "private_networking": true,
      "snapshot_name": "nginx-ubuntu20-{{ timestamp }}",
      "user_data": "",
      "tags": [
        "web",
        "ubuntu",
        "nginx"
      ]
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "sites/",
      "destination": "/tmp"
    },
    {
      "type": "shell",
      "scripts": [
        "config.sh"
      ]
    }
  ]
}

There a few things to note in the above.

• It is standard JSON with a few templating replacements, documented on their site if you’re curious
• The block named builders refers to the targets where you can create stuff
  • The api_token user parameter needs to match your environment variable you set earlier
  • The image, region, and size are cloud-specific and documented in Packer
• The block named provisioners contains stuff you want to happen after the server is created
  • The file one deploys the contents of the local sites folder into /tmp on the server
  • The shell one then runs the contents of the local config.sh file on the server

This means we need a few more files in order to accomplish the provisioning:

• config.sh - the shell file to configure the server
• sites/site1.conf - the file to be served by the sample site (note that it is in a subfolder)
• sites/index.html - a dummy web page to serve up (also in the same subfolder)

config.sh

#!/bin/bash

# Update system.
apt-get update -y

# Set the timezone.
ln -fs /usr/share/zoneinfo/Europe/London /etc/localtime
dpkg-reconfigure -f noninteractive tzdata

# Install Nginx.
apt-get install -y nginx
systemctl enable nginx
systemctl start nginx

# Create a new content folder.
mkdir -p /var/www/site1
cp /tmp/index.html /var/www/site1/index.html

# Replace the default site with the new one.
unlink /etc/nginx/sites-enabled/default
rm /etc/nginx/sites-available/default
cp /tmp/site1.conf /etc/nginx/sites-available/site1.conf
ln -s /etc/nginx/sites-available/site1.conf /etc/nginx/sites-enabled/site1.conf

# Restart Nginx.
systemctl restart nginx

sites/site1.conf

server {
 listen 80;
 listen [::]:80;

 # Reinstate the below to restrict by hostname/IP.
 # server_name example.com;

 location / {
  root /var/www/site1;
  index index.html index.htm;
 }
}

sites/index.html

<html>
 <body>
  <h1>Hello, Cruel World.</h1>
 </body>
</html>

Creating your image/snapshot in Digital Ocean

Download and install Packer from here.

I say install, in fact Packer is a single binary in a zip file without an installer. This means you can actually just unzip it into your current folder and use it from there. That said, I’d recommend you place it somewhere else that is in your current path to avoid multiple versions.

Test it by running the below (replace ./packer with packer on Windows) from your command line.

./packer validate nginx.packer.json

This will test Packer by using it to validate your Packer server definition file.

When you’re ready to run it for real, repeat the above command but replacing validate with build. This will use your API key (from the environment variable you created) to:

• Access Digital Ocean
• Provision a server
• Deploy/configure it
• Take a snapshot
• Shut it down

Once you see the messages start (eg ==> digitalocean: Creating droplet...) you can go to your Digital Ocean droplet area and you’ll see it being created and then eventually disappear as it gets provisioned and snapshotted. The snapshot will then appear in the images area.

Meanwhile on the command line as it finishes you’ll see something like:

==> Builds finished. The artifacts of successful builds are:
--> digitalocean: A snapshot was created: 'nginx-ubuntu20-1601411911' (ID: 70917022) in regions 'lon1'

Whilst this is all pretty fast, and there is no server left running at the end, there may still be a (very small) cost implication.

As time passes the image specified in the packer file under builders (Ubuntu 20) will cease to be current. You may therefore get an error saying 422 You specified an invalid image for Droplet creation. If that is the case, the following command will get you a list of available ones in unformatted JSON.

curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $DIGITALOCEAN_API_TOKEN" "https://api.digitalocean.com/v2/images" >images.json

That places the list in images.json, which you can reformat in your editor or online at the jsonformatter site.

If you are running under Windows that command may not be available. I’d suggest you run it in the WSL, Cygwin, GitBash, or similar. As a last resort just compose it in Postman. It won’t work in the browser as it sets an authentication header with your API token.

Using your image/snapshot to provision a server

You can now use that image to create a new server identical to the one Packer spun up and shut down when it created it’s template. The convenience of a system like Docker for defining servers, but for provisioning cloud servers manually on demand, not containers.

• Go to the Digital Ocean Images, Snapshots area
• Select More to the right of your Packer-created snapshot, and choose Create Droplet
• Choose the size/performance of your new droplet server ($5/month is fine for testing)
• Click Create Droplet, wait for it to complete, and you’ll have a running server

In this case it’s serving the contents of sites/site1.conf if you browse to the droplet’s IP address as shown against it’s name in your Digital Ocean control panel.