Creating Digital Ocean snapshots to use as Droplet templates
The aim
There are many ways to provision your cloud architecture.
- Manually in Digital Ocean, AWS, Azure
- Docker, Kubernetes, or similar
- Puppet, Chef, Terraform
- Cloudformation or equivalent
- And so on ...
Sometimes however you just want an easy, repeatable way to recreate a machine on demand with some stuff already installed and configured. You could be wanting to:
- Create a database server with your database roles and schemas already configured and that you can deploy at a moment's notice if, for example, you need a fresh white-labelled server with default contents
- Create a simple application server running a self-contained and pre-configured Nginx, firewalls, Redis cache, Consul etc, so you can spin up a new instance ready to use with virtually zero notice
- Define a server with a selection of pre-configured dev apps/sites to support cloud working
Whatever it is you're looking to accomplish, having a pre-defined pre-configured server with known versions, options, users, and roles ready to go can give you a cheap and simple point-and-click provisioning option without needing to do notable dev ops work.
This is similar to (if not exactly the same as) the way cloud providers and their third party partners create pre-built images for popular server stacks (eg AMIs in the Amazon Marketplace). In fact many of them use Packer to do it.
Pretty much anything you can do with a server is possible, but to keep the example simple we'll just do Nginx running on Ubuntu 20 and serving a default site.
The tools
This example will be using:
- Packer - a great tool from Hashicorp, makers of Terraform, Vault, and Consul
- Digital Ocean - a cloud provider that offers good servers with very clear, fixed pricing
Packer will use the Digital Ocean API to:
- Spin up a machine
- Deploy and configure it
- Take a snapshot of the running result
- Shut it down again
The end result is a snapshot ready to serve as a template for quickly creating future Droplets (servers).
Getting a Digital Ocean API key
You first need to go to (or sign up free for) your Digital Ocean account and in the menu choose ACCOUNT, API. In there, under Applications & API, Tokens/Keys, Personal access tokens, click to Generate New Token.
Name your token (eg packer
), select Write, and click Generate Token.
Your new token then appears on the screen (it looks like a long random hexadecimal string).
Click the Copy link next to it and then store it safely (eg Hashicorp Vault or even LastPass or similar). This key lets anyone deploy and run anything on your account (at your expense), so never commit it to any form of source control or insecure filesystem/cloud store.
Start a command prompt or terminal window and set your token as an environment variable.
For Windows, replace export
with set
below.
export DIGITALOCEAN_API_TOKEN=d824...etc...3398
Creating a Packer server definition
You now need a JSON file to define the server.
My file is named nginx.packer.json
.
Use the same name or change the commands below.
{
"builders": [
{
"type": "digitalocean",
"ssh_username": "root",
"api_token": "{{ user `DIGITALOCEAN_API_TOKEN` }}",
"image": "ubuntu-20-04-x64",
"region": "lon1",
"size": "s-1vcpu-1gb",
"droplet_name": "packer-nginx-ubuntu20",
"private_networking": true,
"snapshot_name": "nginx-ubuntu20-{{ timestamp }}",
"user_data": "",
"tags": [
"web",
"ubuntu",
"nginx"
]
}
],
"provisioners": [
{
"type": "file",
"source": "sites/",
"destination": "/tmp"
},
{
"type": "shell",
"scripts": [
"config.sh"
]
}
]
}
There a few things to note in the above.
- It is standard JSON with a few templating replacements, documented on their site if you're curious
- The block named
builders
refers to the targets where you can create stuff- The
api_token
user parameter needs to match your environment variable you set earlier - The
image
,region
, andsize
are cloud-specific and documented in Packer
- The
- The block named
provisioners
contains stuff you want to happen after the server is created- The
file
one deploys the contents of the localsites
folder into/tmp
on the server - The
shell
one then runs the contents of the localconfig.sh
file on the server
- The
This means we need a few more files in order to accomplish the provisioning:
config.sh
- the shell file to configure the serversites/site1.conf
- the file to be served by the sample site (note that it is in a subfolder)sites/index.html
- a dummy web page to serve up (also in the same subfolder)
config.sh
#!/bin/bash
# Update system.
apt-get update -y
# Set the timezone.
ln -fs /usr/share/zoneinfo/Europe/London /etc/localtime
dpkg-reconfigure -f noninteractive tzdata
# Install Nginx.
apt-get install -y nginx
systemctl enable nginx
systemctl start nginx
# Create a new content folder.
mkdir -p /var/www/site1
cp /tmp/index.html /var/www/site1/index.html
# Replace the default site with the new one.
unlink /etc/nginx/sites-enabled/default
rm /etc/nginx/sites-available/default
cp /tmp/site1.conf /etc/nginx/sites-available/site1.conf
ln -s /etc/nginx/sites-available/site1.conf /etc/nginx/sites-enabled/site1.conf
# Restart Nginx.
systemctl restart nginx
sites/site1.conf
server {
listen 80;
listen [::]:80;
# Reinstate the below to restrict by hostname/IP.
# server_name example.com;
location / {
root /var/www/site1;
index index.html index.htm;
}
}
sites/index.html
<html>
<body>
<h1>Hello, Cruel World.</h1>
</body>
</html>
Creating your image/snapshot in Digital Ocean
Download and install Packer from here.
I say install, in fact Packer is a single binary in a zip file without an installer. This means you can actually just unzip it into your current folder and use it from there. That said, I'd recommend you place it somewhere else that is in your current path to avoid multiple versions.
Test it by running the below (replace ./packer
with packer
on Windows) from your command line.
./packer validate nginx.packer.json
This will test Packer by using it to validate your Packer server definition file.
When you're ready to run it for real, repeat the above command but replacing validate
with build
.
This will use your API key (from the environment variable you created) to:
- Access Digital Ocean
- Provision a server
- Deploy/configure it
- Take a snapshot
- Shut it down
Once you see the messages start (eg ==> digitalocean: Creating droplet...
) you can go to your Digital Ocean droplet area and you'll see it being created and then eventually disappear as it gets provisioned and snapshotted. The snapshot will then appear in the images area.
Meanwhile on the command line as it finishes you'll see something like:
==> Builds finished. The artifacts of successful builds are:
--> digitalocean: A snapshot was created: 'nginx-ubuntu20-1601411911' (ID: 70917022) in regions 'lon1'
Whilst this is all pretty fast, and there is no server left running at the end, there may still be a (very small) cost implication.
As time passes the image
specified in the packer file under builders
(Ubuntu 20) will cease to be current.
You may therefore get an error saying 422 You specified an invalid image for Droplet creation.
If that is the case, the following command will get you a list of available ones in unformatted JSON.
curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer $DIGITALOCEAN_API_TOKEN" "https://api.digitalocean.com/v2/images" >images.json
That places the list in images.json
, which you can reformat in your editor or online at the jsonformatter site.
If you are running under Windows that command may not be available. I'd suggest you run it in the WSL, Cygwin, GitBash, or similar. As a last resort just compose it in Postman. It won't work in the browser as it sets an authentication header with your API token.
Using your image/snapshot to provision a server
You can now use that image to create a new server identical to the one Packer spun up and shut down when it created it's template. The convenience of a system like Docker for defining servers, but for provisioning cloud servers manually on demand, not containers.
- Go to the Digital Ocean Images, Snapshots area
- Select More to the right of your Packer-created snapshot, and choose Create Droplet
- Choose the size/performance of your new droplet server ($5/month is fine for testing)
- Click Create Droplet, wait for it to complete, and you'll have a running server
In this case it's serving the contents of sites/site1.conf
if you browse to the droplet's IP address as shown against it's name in your Digital Ocean control panel.