Use Packer like a Pro

I’ve been using Proxmox VE for a while now in my Homelab as an open-source alternative for a virtualization platform like ESXi. One useful feature in Proxmox is the templates that allow us to create LXC or VM templates that can then be cloned as a starting point for new Proxmox resources. Now with these templates, we can have a standard starting point to install our applications on top of, pre-install packages for authentication, security, logging, etc without anyone else needing to think about it as we bake these best practices right into these template resources.

However, creating and managing these templates can become a challenge with how time-consuming and manual it can be. I want to show you how you can make this process more standardized and automated with the use of Packer to modifying your Proxmox templates, orchestrating the building, and packaging of these templates so they are available for use on your Proxmox hosts.

References

Source Code

https://github.com/TechProber/cloud-estate/tree/master/packer-templates

Prerequisite

- A server already has Proxmox VE installed, I am currently running Proxmox VE 7.1.12
- A cup of coffee

Software Requirement

Install jq locally on your local machine

1# archlinux
2$ sudo pacman -S jq
3
4# debian/ubuntu
5sudo apt-get install jq

Install Packer locally on your local machine

 1# archlinux
 2$ sudo pacman -S packer
 3
 4# debian/ubuntu
 5$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
 6$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
 7$ sudo apt-get update && sudo apt-get install packer
 8
 9# homebrew
10$ brew tap hashicorp/tap
11$ brew install hashicorp/tap/packer
12
13# verify installation
14$ packer

For detailed instructions on how to install Packer on other platforms or Linux distributions, please head to this Getting Started guide.

What is Packer

Packer is a utility that allows you to build virtual machine images so that you can define a golden image as code. Packer can be used to create images for almost all of the big cloud providers such as AWS, GCE, Azure, and Digital Ocean, or can be used with locally installed hypervisors such as VMWare, Proxmox, and a few others.

To build an image with packer we need to define our image through a template file. The file uses the JSON format and comprises 3 main sections that are used to define and prepare your image.

Builders

**Builders**: Components of Packer that can create a machine image for a single platform. A builder is invoked as part of a build to create the actual resulting images.

Provisioners

**Provisioners**: Install and configure software within a running machine before that machine is turned into a static image. Example provisioners include shell scripts, Chef, Puppet, etc.

Post Processors

**Provisioners**: Install and configure software within a running machine before that machine is turned into a static image. Example provisioners include shell scripts, Chef, Puppet, etc.

By using packer we can define our golden VM image as code so that we can easily build identically configured images on demand so that all your machines are running the same image and can also be easily updated to a new image when needed.

Create a Proxmox user for Packer

Packer requires a user account to perform actions on the Proxmox API. The following commands will create a new user account packer@pve with restricted permissions.

1pveum useradd packer@pve
2pveum passwd packer@pve
3# Enter new password: ****************
4# Retype new password: ****************
5pveum aclmod / -user packer@pve -role PVEAdmin

Prepare your packer template

To create the template we will use the proxmox builder ](https://packer.io/docs/builders/proxmox.html) which connects through the proxmox web API to provision and configure the VM for us and then turn it into a template. To configure our template we will use a variables file, to import this variables file we will use the -var-file flag to pass in our variables to Packer. These variables will be used in our template file with the following syntax within a string like so passwd/username={{ user 'ssh_username'}}.

The builder block below will outline the basic properties of our desired proxmox template such as its name, the allocated resources, and the devices attached to the VM. To achieve this the boot_command option will be used to boot the OS and tell it to look for the http/user-data file to automate the OS installation process. Packer will start an HTTP server from the content of the http directory (with the http_directory parameter). This will allow Subiquity to fetch the cloud-init files remotely.

Notes: The live installer Subiquity uses more memory than Debian-installer. The default value from Packer (512M) is not enough and will lead to weird kernel panic. Use 1G as a minimum.

The boot_command tells cloud-init to start and uses the nocloud-net data source to be able to load the user-data and meta-data files from a remote HTTP endpoint. The additional autoinstall parameter will force Subiquity to perform destructive actions without asking for confirmation from the user.

Import Notes: Since Ubuntu 21.04, the boot_command has been updated, so please be aware of that.

 1// https://github.com/TechProber/cloud-estate/blob/master/packer-templates/vars/ubuntu-2204.json#L23-L43
 2{
 3  ...
 4  "boot_command": [
 5      "<esc><esc><esc><esc>e<wait>",
 6      "<del><del><del><del><del><del><del><del>",
 7      "<del><del><del><del><del><del><del><del>",
 8      "<del><del><del><del><del><del><del><del>",
 9      "<del><del><del><del><del><del><del><del>",
10      "<del><del><del><del><del><del><del><del>",
11      "<del><del><del><del><del><del><del><del>",
12      "<del><del><del><del><del><del><del><del>",
13      "<del><del><del><del><del><del><del><del>",
14      "<del><del><del><del><del><del><del><del>",
15      "<del><del><del><del><del><del><del><del>",
16      "<del><del><del><del><del><del><del><del>",
17      "<del><del><del><del><del><del><del><del>",
18      "<del><del><del><del><del><del><del><del>",
19      "<del><del><del><del><del><del><del><del>",
20      "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://{{.HTTPIP}}:{{.HTTPPort}}/\"<enter><wait>",
21      "initrd /casper/initrd<enter><wait>",
22      "boot<enter>",
23      "<enter><f10><wait>"
24    ]
25  ...
26}

Finally, we will use the post processors to run some commands locally. This will make an SSH connection to the PVE host and run some commands manually to set up the virtual devices necessary for [ cloud-init. This post-processor is using the shell-local post processor to run the commands on the local machine running packer but you could always move this configuration to something like an ansible playbook to make the configuration more readable and portable.

 1# https://github.com/TechProber/cloud-estate/blob/master/packer-templates/proxmox-packer-template.pkr.hcl#L119-L127
 2post-processor "shell-local" {
 3    inline = [
 4      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --boot c --bootdisk scsi0",
 5      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --ciuser ${var.ssh_username}",
 6      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --cipassword ${var.ssh_password}",
 7      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --serial0 socket --vga serial0",
 8      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --delete ide2"
 9    ]
10  }

You may find the complete packer-var in https://github.com/TechProber/cloud-estate/tree/master/packer-templates/vars

Build your Proxmox template with Packer

Get Started

Source Code - https://github.com/TechProber/cloud-estate/tree/master/packer-templates.

1$ git clone https://github.com/TechProber/cloud-estate
2$ cd packer-templates

File Structure

 1# tree -d -L 3 ./
 2./
 3├── assets
 4├── http
 5├── playbooks
 6│   ├── roles
 7│   │   ├── apt.ops
 8│   │   ├── containerd.ops
 9│   │   ├── docker.ops
10│   │   ├── maintenance.ops
11│   │   ├── minio.ops
12│   │   ├── proxmox.base
13│   │   └── proxmox.bootstrap
14│   └── vars
15└── vars

./vars - where packer-var is defined
./http - where cloud-init configuration is defined
./playbooks - where all automation workloads are defined
./playbooks/roles - specific ansible-roles for different needs
./playbooks/vars - default ansible-playbook variables
./bake - the automation script that handles all the heavy-lifting work

Proxmox API Authentication

After creating a dedicated Packer User followed the guide above, you will also need to export the PM_PASS as an environment variable (the password of the packer user to interact with Proxmox API authentication).

1$ export PM_PASS=<pm_password>

You will also need to manually modify the ./config.yml to connect to your Proxmox Server

1{
2  "proxmox_host": "10.178.0.10",
3  "proxmox_node_name": "pve-01",
4  "proxmox_api_user": "packer@pve",
5  "http_bind_address": "10.178.0.50",
6}

http_bind_address is the IP address of the host machine that has Packer installed.

The ./bake script will take the PM_PASS environment variable and all the attributes defined in the ./config.yml to interact with Packer and Proxmox.

SSH Key Management

For safety concerns, the VM template created by Packer can ONLY be connected via ssh-key-pair. You will need to prepare your ssh-public key before baking the VM. If you do not have an ssh-key-pair yet, you may use the following commands to generate one.

1$ ssh-keygen -m PEM -t rsa -b 4096 -C “user@example.com”

Then, you will need to replace the default ./id_rsa.pub with your newly created public key in the project root directory

1# ./packer-templates
2$ cat ~/.ssh/id_rsa.pub > ./id_rsa.pub

Ansible Local Provisioner

During the VM baking/building period, the entire automation is handled by Ansible. I assume you already have some foundational knowledge about Ansible, if not feel free to check out the Official Sites for more information.

The ansible-local Packer provisioner will execute ansible in Ansible’s local mode on the remote/guest VM using Playbook and Role files that exist on the guest VM. This means Ansible must be installed on the remote/guest VM. Playbooks and Roles can be uploaded from your build machine (the one running Packer) to the VM. Ansible is then run on the guest machine in local mode via the ansible-playbook command. For more information, please check out https://www.packer.io/plugins/provisioners/ansible/ansible-local

To see all the available roles, head over to https://github.com/TechProber/cloud-estate/tree/master/packer-templates/playbooks/roles. You are also welcome to raise PR/issue for feature requests. More roles are coming up soon.

Each VM template relies on a combination of ansible-roles to achieve different features. For instance, there is a specific role for Docker and Docker-Compose. Reference to the sample docker-ubuntu-2204-server VM template

Customization

If you like to create a custom VM template that is not defined in the ./bakery-config.json, you may take the custom VM template as a reference and adjust the roles you would like to be included in the VM template.

Ansible Vault (Optional)

If you plan to use ansible-vault to encrypt ansible variables, you may also need to create the .vault-pass file with a vault password under ./playbooks/.vault-pass (already added to .gitignore). It will be used to encrypt and decrypt sensitive variables.

You will also need to add an extra provisioner in the packer template file as this is for the advanced use case. The .vault-pass file will be passed to /tmp/.vault-pass during the baking period and will be deleted afterward (seen in detailed implementation). Feel free to check out an existing example - https://github.com/TechProber/cloud-estate/blob/master/packer-templates/custom-proxmox-packer-template.pkr.hcl#L108-L112

Import Notes: By default, the custom vm template uses minio which relies on .vault-pass. If you do not wish to install Mini Client, then you will need to manually remove the minio-role in ./playbooks/custom.yml

Bake CLI

The bake CLI is a tool I created for speeding up the process of building multiple VM templates

Check out the help menu with the -h|--help option for more information:

1./bake -h

List VM Templates

List all the available templates with -a|--all option:

1./bake -a

Bake a standard VM (Basic Usage)

1./bake -i [vm-id] ubuntu-2204-server

Bake a custom VM (Advanced Usage)

1./bake -i [vm-id] custom -n [custom-vm-template-name] -b [custom-build]

For the -b|--build option, it ONLY supports minio as custom build for now - custom-proxmox-packer-template.

To extend its use case, feel free to contribute . PRs are always welcome.

Docker Support

To bake/build a VM template that ships with Docker and Docker-Compose , use the following command:

1./bake -i [vm-id] docker-ubuntu-2204-server

Containerd Support

To bake/build a VM template that ships with Containerd and Nerdctl , use the following command:

1./bake -i [vm-id] containerd-ubuntu-2204-server

Changing APT Source

There is a role to change the default APT source - https://github.com/TechProber/cloud-estate/tree/master/packer-templates/playbooks/roles/apt.ops/set-sources.ops, and it is set to the CN(USTC) source by default. You may overwrite the configurations defined in ./playbooks/var/apt.yml

Demo

Create a custom ubuntu-2204-server with Docker and Minio Client installed and configured

1$ ./bake -i 9001 -t custom -n custom-ubuntu-2204-server -c minio

./playbooks/custom.yml

 1---
 2# Bake custom-server
 3
 4- name: "Bake proxmox custom vm template"
 5  hosts: localhost
 6  become: yes
 7
 8  vars_files:
 9    - ./vars/apt.yml
10    - ./vars/maintenance.yml
11
12  vars:
13    - vault_enable: true
14
15  roles:
16    - role: ./roles/apt.ops/set-sources.ops/
17      vars:
18        release: "jammy"
19    - role: ./roles/apt.ops/install-packages.ops/
20      vars:
21        extra_packages:
22          - neofetch
23
24    - role: ./roles/maintenance.ops/key.ops/
25
26    - role: ./roles/docker.ops/
27
28    - role: ./roles/proxmox.bootstrap/

./custom-proxmox-packer-template.pkr.hcl

 1...
 2
 3build {
 4
 5  name = "minio"
 6
 7  sources = ["source.proxmox.bakery-template"]
 8
 9  # Provisioner Configurations
10
11  # SSH public key
12  provisioner "file" {
13    source      = "./id_rsa.pub"
14    destination = "/tmp/id_rsa.pub"
15  }
16
17  # Minio plabyook
18  provisioner "file" {
19    pause_before = "5s"
20    source       = "./playbooks/.vault_pass"
21    destination  = "/tmp/.vault_pass"
22  }
23  provisioner "ansible-local" {
24    playbook_dir            = "./playbooks"
25    playbook_file           = "./playbooks/minio.yml"
26    clean_staging_directory = true
27    extra_arguments = [
28      "--vault-password-file=/tmp/.vault_pass",
29      "--extra-vars \"ansible_user=packer\""
30    ]
31  }
32
33  # Main playbook depends of vm_type
34  provisioner "ansible-local" {
35    pause_before            = "5s"
36    playbook_dir            = "./playbooks"
37    playbook_file           = var.playbook_file
38    clean_staging_directory = true
39    extra_arguments = [
40      "--extra-vars \"ansible_user=packer\""
41    ]
42  }
43
44  # Convert to proxmox vm template
45  post-processor "shell-local" {
46    inline = [
47      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --boot c --bootdisk scsi0",
48      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --ciuser ${var.ssh_username}",
49      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --cipassword ${var.ssh_password}",
50      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --serial0 socket --vga serial0",
51      "ssh root@${var.proxmox_host} qm set ${var.vm_id} --delete ide2"
52    ]
53  }
54}

You should see some output for each of the builders, provisioners, and post-processors.

 1# ./bake -i 9001 -t custom -n docker-ubuntu-2204-server-template -b minio -f ./vars/kevin-ubuntu-2204.json
 2
 3########## Baking docker-ubuntu-2204-server-template template with packer
 4
 5minio.proxmox.bakery-template: output will be in this color.
 6
 7==> minio.proxmox.bakery-template: Creating VM
 8==> minio.proxmox.bakery-template: Starting VM
 9==> minio.proxmox.bakery-template: Starting HTTP server on port 8802
10==> minio.proxmox.bakery-template: Waiting 5s for boot
11==> minio.proxmox.bakery-template: Typing the boot command
12==> minio.proxmox.bakery-template: Waiting for SSH to become available...
13
14...
15
16==> minio.proxmox.bakery-template: Uploading ./id_rsa.pub => /tmp/id_rsa.pub
17    minio.proxmox.bakery-template: id_rsa.pub 743 B / 743 B [=================================================================] 100.00% 0s
18==> minio.proxmox.bakery-template: Pausing 5s before the next provisioner...
19==> minio.proxmox.bakery-template: Uploading ./playbooks/.vault_pass => /tmp/.vault_pass
20    minio.proxmox.bakery-template: .vault_pass 21 B / 21 B [==================================================================] 100.00% 0s
21==> minio.proxmox.bakery-template: Provisioning with Ansible...
22    minio.proxmox.bakery-template: Uploading Playbook directory to Ansible staging directory...
23    minio.proxmox.bakery-template: Creating directory: /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37
24    minio.proxmox.bakery-template: Uploading main Playbook file...
25    minio.proxmox.bakery-template: Uploading inventory file...
26    minio.proxmox.bakery-template: Executing Ansible: cd /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37 && ANSIBLE_FORCE_COLOR=1 PYTHONUNBUFFERED=1 ansible-playbook /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37/minio.yml --extra-vars "packer_build_name=bakery-template packer_builder_type=proxmox packer_http_addr=10.178.0.50:8802 -o IdentitiesOnly=yes" --vault-password-file=/tmp/.vault_pass --extra-vars "ansible_user=packer" -c local -i /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37/packer-provisioner-ansible-local2268832455
27
28...
29
30    minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Install Docker with script] ************************
31    minio.proxmox.bakery-template: changed: [127.0.0.1]
32    minio.proxmox.bakery-template:
33    minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Enable docker to start at boot] ********************
34    minio.proxmox.bakery-template: ok: [127.0.0.1]
35    minio.proxmox.bakery-template:
36    minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Add user to docker group] **************************
37    minio.proxmox.bakery-template: changed: [127.0.0.1]
38    minio.proxmox.bakery-template:
39    minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Post installation message] *************************
40    minio.proxmox.bakery-template: ok: [127.0.0.1] => {
41    minio.proxmox.bakery-template:     "msg": "Use \"newgrp docker\" to use the group immediately"
42    minio.proxmox.bakery-template: }
43    minio.proxmox.bakery-template:
44    minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Check if docker-compose is installed] **************
45    minio.proxmox.bakery-template: ok: [127.0.0.1]
46    minio.proxmox.bakery-template:
47    minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Install docker-compose] ****************************
48    minio.proxmox.bakery-template: changed: [127.0.0.1]
49
50...
51
52    minio.proxmox.bakery-template: PLAY RECAP *********************************************************************
53    minio.proxmox.bakery-template: 127.0.0.1                  : ok=34   changed=21   unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
54    minio.proxmox.bakery-template:
55    minio.proxmox.bakery-template: Removing staging directory...
56    minio.proxmox.bakery-template: Removing directory: /tmp/packer-provisioner-ansible-local/626557c0-c15c-4738-aebb-53538294728e
57==> minio.proxmox.bakery-template: Stopping VM
58==> minio.proxmox.bakery-template: Converting VM to template
59==> minio.proxmox.bakery-template: Adding a cloud-init cdrom in storage pool sata-pool
60==> minio.proxmox.bakery-template: Running post-processor:  (type shell-local)
61==> minio.proxmox.bakery-template (shell-local): Running local shell script: /tmp/packer-shell1584895912
62    minio.proxmox.bakery-template (shell-local): update VM 9001: -boot c -bootdisk scsi0
63    minio.proxmox.bakery-template (shell-local): update VM 9001: -ciuser packer
64    minio.proxmox.bakery-template (shell-local): update VM 9001: -cipassword <hidden>
65    minio.proxmox.bakery-template (shell-local): update VM 9001: -serial0 socket -vga serial0
66    minio.proxmox.bakery-template (shell-local): update VM 9001: -delete ide2
67Build 'minio.proxmox.bakery-template' finished after 5 minutes 58 seconds.
68
69==> Wait completed after 5 minutes 58 seconds
70
71==> Builds finished. The artifacts of successful builds are:
72--> minio.proxmox.bakery-template: A template was created: 9001
73--> minio.proxmox.bakery-template: A template was created: 9001
74The last command took 359.68 seconds.

Up to this point, the custom VM template with Docker, Docker-Compose, and Minio Client pre-installed has been successfully uploaded to the Proxmox server.

Conclusion

To sum up, with such an approach, we can deploy a new VM in minutes and drastically speed up the DevOps process. Ansible is a very powerful tool as it opens up many opportunities to basically automate any shell-based tasks. Next, I plan to write another post to further extend the automation with Terraform and Terragrunt to deploy a VM in Proxmox with the VM template created by Packer.