K Cartlidge

DotNet/Core, Golang, Node, Elixir/Phoenix, Ruby/Rails, Python/Flask, PHP.

Creating Digital Ocean snapshots to use as Droplet templates

The aim

There are many ways to provision your cloud architecture.

Sometimes however you just want an easy, repeatable way to recreate a machine on demand with some stuff already installed and configured. You could be wanting to:

Whatever it is you're looking to accomplish, having a pre-defined pre-configured server with known versions, options, users, and roles ready to go can give you a cheap and simple point-and-click provisioning option without needing to do notable dev ops work.

This is similar to (if not exactly the same as) the way cloud providers and their third party partners create pre-built images for popular server stacks (eg AMIs in the Amazon Marketplace). In fact many of them use Packer to do it.

Pretty much anything you can do with a server is possible, but to keep the example simple we'll just do Nginx running on Ubuntu 20 and serving a default site.

The tools

This example will be using:

Packer will use the Digital Ocean API to:

The end result is a snapshot ready to serve as a template for quickly creating future Droplets (servers).

Getting a Digital Ocean API key

You first need to go to (or sign up free for) your Digital Ocean account and in the menu choose ACCOUNT, API. In there, under Applications & API, Tokens/Keys, Personal access tokens, click to Generate New Token.

Name your token (eg packer), select Write, and click Generate Token. Your new token then appears on the screen (it looks like a long random hexadecimal string).

Click the Copy link next to it and then store it safely (eg Hashicorp Vault or even LastPass or similar). This key lets anyone deploy and run anything on your account (at your expense), so never commit it to any form of source control or insecure filesystem/cloud store.

Start a command prompt or terminal window and set your token as an environment variable. For Windows, replace export with set below.

export DIGITALOCEAN_API_TOKEN=d824...etc...3398

Creating a Packer server definition

You now need a JSON file to define the server. My file is named nginx.packer.json. Use the same name or change the commands below.

{
	"builders": [
		{
			"type": "digitalocean",
			"ssh_username": "root",
			"api_token": "{{ user `DIGITALOCEAN_API_TOKEN` }}",
			"image": "ubuntu-20-04-x64",
			"region": "lon1",
			"size": "s-1vcpu-1gb",
			"droplet_name": "packer-nginx-ubuntu20",
			"private_networking": true,
			"snapshot_name": "nginx-ubuntu20-{{ timestamp }}",
			"user_data": "",
			"tags": [
				"web",
				"ubuntu",
				"nginx"
			]
		}
	],
	"provisioners": [
		{
			"type": "file",
			"source": "sites/",
			"destination": "/tmp"
		},
		{
			"type": "shell",
			"scripts": [
				"config.sh"
			]
		}
	]
}

There a few things to note in the above.

This means we need a few more files in order to accomplish the provisioning:

config.sh

#!/bin/bash

# Update system.
apt-get update -y

# Set the timezone.
ln -fs /usr/share/zoneinfo/Europe/London /etc/localtime
dpkg-reconfigure -f noninteractive tzdata

# Install Nginx.
apt-get install -y nginx
systemctl enable nginx
systemctl start nginx

# Create a new content folder.
mkdir -p /var/www/site1
cp /tmp/index.html /var/www/site1/index.html

# Replace the default site with the new one.
unlink /etc/nginx/sites-enabled/default
rm /etc/nginx/sites-available/default
cp /tmp/site1.conf /etc/nginx/sites-available/site1.conf
ln -s /etc/nginx/sites-available/site1.conf /etc/nginx/sites-enabled/site1.conf

# Restart Nginx.
systemctl restart nginx

sites/site1.conf

server {
	listen 80;
	listen [::]:80;

	# Reinstate the below to restrict by hostname/IP.
	# server_name example.com;

	location / {
		root /var/www/site1;
		index index.html index.htm;
	}
}

sites/index.html

<html>
	<body>
		<h1>Hello, Cruel World.</h1>
	</body>
</html>

Creating your image/snapshot in Digital Ocean

Download and install Packer from here.

I say install, in fact Packer is a single binary in a zip file without an installer. This means you can actually just unzip it into your current folder and use it from there. That said, I'd recommend you place it somewhere else that is in your current path to avoid multiple versions.

Test it by running the below (replace ./packer with packer on Windows) from your command line.

./packer validate nginx.packer.json

This will test Packer by using it to validate your Packer server definition file.

When you're ready to run it for real, repeat the above command but replacing validate with build. This will use your API key (from the environment variable you created) to:

Once you see the messages start (eg ==> digitalocean: Creating droplet...) you can go to your Digital Ocean droplet area and you'll see it being created and then eventually disappear as it gets provisioned and snapshotted. The snapshot will then appear in the images area.

Meanwhile on the command line as it finishes you'll see something like:

==> Builds finished. The artifacts of successful builds are:
--> digitalocean: A snapshot was created: 'nginx-ubuntu20-1601411911' (ID: 70917022) in regions 'lon1'

Whilst this is all pretty fast, and there is no server left running at the end, there may still be a (very small) cost implication.

As time passes the image specified in the packer file under builders (Ubuntu 20) will cease to be current. You may therefore get an error saying 422 You specified an invalid image for Droplet creation. If that is the case, the following command will get you a list of available ones in unformatted JSON.

curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $DIGITALOCEAN_API_TOKEN" "https://api.digitalocean.com/v2/images" >images.json

That places the list in images.json, which you can reformat in your editor or online at the jsonformatter site.

If you are running under Windows that command may not be available. I'd suggest you run it in the WSL, Cygwin, GitBash, or similar. As a last resort just compose it in Postman. It won't work in the browser as it sets an authentication header with your API token.

Using your image/snapshot to provision a server

You can now use that image to create a new server identical to the one Packer spun up and shut down when it created it's template. The convenience of a system like Docker for defining servers, but for provisioning cloud servers manually on demand, not containers.

In this case it's serving the contents of sites/site1.conf if you browse to the droplet's IP address as shown against it's name in your Digital Ocean control panel.