
Virtualize unRAID Server in VMWare ESXi 7.0
After some intensive research and consideration, I found out that virtualizing unRIAD Server in VMWare ESXi 7.0 seems like a possibility. Due to the …
I’ve been using Proxmox VE for a while now in my Homelab as an open-source alternative for a virtualization platform like ESXi. One useful feature in Proxmox is the templates that allow us to create LXC
or VM
templates that can then be cloned as a starting point for new Proxmox resources. Now with these templates, we can have a standard starting point to install our applications on top of, pre-install packages for authentication, security, logging, etc without anyone else needing to think about it as we bake these best practices right into these template resources.
However, creating and managing these templates can become a challenge with how time-consuming and manual it can be. I want to show you how you can make this process more standardized and automated with the use of Packer to modifying your Proxmox templates, orchestrating the building, and packaging of these templates so they are available for use on your Proxmox hosts.
https://github.com/TechProber/cloud-estate/tree/master/packer-templates
Proxmox VE
installed, I am currently running Proxmox VE 7.1.12
Install jq
locally on your local machine
1# archlinux
2$ sudo pacman -S jq
3
4# debian/ubuntu
5sudo apt-get install jq
Install Packer
locally on your local machine
1# archlinux
2$ sudo pacman -S packer
3
4# debian/ubuntu
5$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
6$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
7$ sudo apt-get update && sudo apt-get install packer
8
9# homebrew
10$ brew tap hashicorp/tap
11$ brew install hashicorp/tap/packer
12
13# verify installation
14$ packer
For detailed instructions on how to install Packer on other platforms or Linux distributions, please head to this Getting Started guide.
Packer is a utility that allows you to build virtual machine images so that you can define a golden image as code. Packer can be used to create images for almost all of the big cloud providers such as AWS, GCE, Azure, and Digital Ocean, or can be used with locally installed hypervisors such as VMWare, Proxmox, and a few others.
To build an image with packer we need to define our image through a template file. The file uses the JSON format and comprises 3 main sections that are used to define and prepare your image.
By using packer we can define our golden VM image as code so that we can easily build identically configured images on demand so that all your machines are running the same image and can also be easily updated to a new image when needed.
Packer requires a user account to perform actions on the Proxmox API. The following commands will create a new user account packer@pve
with restricted permissions.
1pveum useradd packer@pve
2pveum passwd packer@pve
3# Enter new password: ****************
4# Retype new password: ****************
5pveum aclmod / -user packer@pve -role PVEAdmin
To create the template we will use the proxmox builder ](https://packer.io/docs/builders/proxmox.html) which connects through the proxmox web API
to provision and configure the VM for us and then turn it into a template. To configure our template we will use a variables file, to import this variables file we will use the -var-file
flag to pass in our variables to Packer. These variables will be used in our template file with the following syntax within a string like so passwd/username={{ user 'ssh_username'}}
.
The builder block below will outline the basic properties of our desired proxmox template such as its name, the allocated resources, and the devices attached to the VM. To achieve this the boot_command option will be used to boot the OS and tell it to look for the http/user-data
file to automate the OS installation process. Packer will start an HTTP server from the content of the http
directory (with the http_directory
parameter). This will allow Subiquity
to fetch the cloud-init files remotely.
Notes: The live installer Subiquity
uses more memory than Debian-installer. The default value from Packer (512M) is not enough and will lead to weird kernel panic. Use 1G
as a minimum.
The boot_command
tells cloud-init to start and uses the nocloud-net
data source to be able to load the user-data and meta-data
files from a remote HTTP endpoint. The additional autoinstall
parameter will force Subiquity to perform destructive actions without asking for confirmation from the user.
Import Notes: Since Ubuntu 21.04
, the boot_command
has been updated, so please be aware of that.
1// https://github.com/TechProber/cloud-estate/blob/master/packer-templates/vars/ubuntu-2204.json#L23-L43
2{
3 ...
4 "boot_command": [
5 "<esc><esc><esc><esc>e<wait>",
6 "<del><del><del><del><del><del><del><del>",
7 "<del><del><del><del><del><del><del><del>",
8 "<del><del><del><del><del><del><del><del>",
9 "<del><del><del><del><del><del><del><del>",
10 "<del><del><del><del><del><del><del><del>",
11 "<del><del><del><del><del><del><del><del>",
12 "<del><del><del><del><del><del><del><del>",
13 "<del><del><del><del><del><del><del><del>",
14 "<del><del><del><del><del><del><del><del>",
15 "<del><del><del><del><del><del><del><del>",
16 "<del><del><del><del><del><del><del><del>",
17 "<del><del><del><del><del><del><del><del>",
18 "<del><del><del><del><del><del><del><del>",
19 "<del><del><del><del><del><del><del><del>",
20 "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://{{.HTTPIP}}:{{.HTTPPort}}/\"<enter><wait>",
21 "initrd /casper/initrd<enter><wait>",
22 "boot<enter>",
23 "<enter><f10><wait>"
24 ]
25 ...
26}
Finally, we will use the post processors to run some commands locally. This will make an SSH connection to the PVE host and run some commands manually to set up the virtual devices necessary for [ cloud-init. This post-processor is using the shell-local post processor to run the commands on the local machine running packer but you could always move this configuration to something like an ansible playbook to make the configuration more readable and portable.
1# https://github.com/TechProber/cloud-estate/blob/master/packer-templates/proxmox-packer-template.pkr.hcl#L119-L127
2post-processor "shell-local" {
3 inline = [
4 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --boot c --bootdisk scsi0",
5 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --ciuser ${var.ssh_username}",
6 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --cipassword ${var.ssh_password}",
7 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --serial0 socket --vga serial0",
8 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --delete ide2"
9 ]
10 }
You may find the complete packer-var
in https://github.com/TechProber/cloud-estate/tree/master/packer-templates/vars
Source Code - https://github.com/TechProber/cloud-estate/tree/master/packer-templates.
1$ git clone https://github.com/TechProber/cloud-estate
2$ cd packer-templates
1# tree -d -L 3 ./
2./
3├── assets
4├── http
5├── playbooks
6│ ├── roles
7│ │ ├── apt.ops
8│ │ ├── containerd.ops
9│ │ ├── docker.ops
10│ │ ├── maintenance.ops
11│ │ ├── minio.ops
12│ │ ├── proxmox.base
13│ │ └── proxmox.bootstrap
14│ └── vars
15└── vars
After creating a dedicated Packer User followed the guide above, you will also need to export the PM_PASS
as an environment variable (the password of the packer user to interact with Proxmox API authentication).
1$ export PM_PASS=<pm_password>
You will also need to manually modify the ./config.yml
to connect to your Proxmox Server
1{
2 "proxmox_host": "10.178.0.10",
3 "proxmox_node_name": "pve-01",
4 "proxmox_api_user": "packer@pve",
5 "http_bind_address": "10.178.0.50",
6}
http_bind_address
is the IP address of the host machine that has Packer installed.
The ./bake
script will take the PM_PASS
environment variable and all the attributes defined in the ./config.yml
to interact with Packer and Proxmox.
For safety concerns, the VM template created by Packer can ONLY be connected via ssh-key-pair
. You will need to prepare your ssh-public key before baking the VM. If you do not have an ssh-key-pair
yet, you may use the following commands to generate one.
1$ ssh-keygen -m PEM -t rsa -b 4096 -C “user@example.com”
Then, you will need to replace the default ./id_rsa.pub
with your newly created public key in the project root directory
1# ./packer-templates
2$ cat ~/.ssh/id_rsa.pub > ./id_rsa.pub
During the VM baking/building period, the entire automation is handled by Ansible
. I assume you already have some foundational knowledge about Ansible, if not feel free to check out the Official Sites for more information.
The ansible-local
Packer provisioner will execute ansible in Ansible’s local
mode on the remote/guest VM using Playbook and Role files that exist on the guest VM. This means Ansible must be installed on the remote/guest VM. Playbooks and Roles can be uploaded from your build machine (the one running Packer) to the VM. Ansible is then run on the guest machine in local mode via the ansible-playbook
command. For more information, please check out https://www.packer.io/plugins/provisioners/ansible/ansible-local
To see all the available roles, head over to https://github.com/TechProber/cloud-estate/tree/master/packer-templates/playbooks/roles. You are also welcome to raise PR/issue for feature requests. More roles are coming up soon.
Each VM template relies on a combination of ansible-roles
to achieve different features. For instance, there is a specific role for Docker
and Docker-Compose
. Reference to the sample docker-ubuntu-2204-server VM template
If you like to create a custom VM template that is not defined in the ./bakery-config.json
, you may take the custom VM template as a reference and adjust the roles you would like to be included in the VM template.
If you plan to use ansible-vault to encrypt ansible variables, you may also need to create the .vault-pass
file with a vault password under ./playbooks/.vault-pass
(already added to .gitignore
). It will be used to encrypt and decrypt sensitive variables.
You will also need to add an extra provisioner
in the packer template file as this is for the advanced use case. The .vault-pass
file will be passed to /tmp/.vault-pass
during the baking period and will be deleted afterward (seen in detailed implementation). Feel free to check out an existing example - https://github.com/TechProber/cloud-estate/blob/master/packer-templates/custom-proxmox-packer-template.pkr.hcl#L108-L112
Import Notes: By default, the custom vm template uses minio
which relies on .vault-pass
. If you do not wish to install Mini Client, then you will need to manually remove the minio-role
in ./playbooks/custom.yml
The bake
CLI is a tool I created for speeding up the process of building multiple VM templates
Check out the help menu with the -h|--help
option for more information:
1./bake -h
List all the available templates with -a|--all
option:
1./bake -a
1./bake -i [vm-id] ubuntu-2204-server
1./bake -i [vm-id] custom -n [custom-vm-template-name] -b [custom-build]
For the -b|--build
option, it ONLY supports minio
as custom build for now - custom-proxmox-packer-template.
To extend its use case, feel free to contribute . PRs
are always welcome.
To bake/build a VM template that ships with Docker and Docker-Compose , use the following command:
1./bake -i [vm-id] docker-ubuntu-2204-server
To bake/build a VM template that ships with Containerd and Nerdctl , use the following command:
1./bake -i [vm-id] containerd-ubuntu-2204-server
There is a role to change the default APT
source - https://github.com/TechProber/cloud-estate/tree/master/packer-templates/playbooks/roles/apt.ops/set-sources.ops, and it is set to the CN(USTC)
source by default. You may overwrite the configurations defined in ./playbooks/var/apt.yml
Create a custom ubuntu-2204-server with Docker
and Minio Client
installed and configured
1$ ./bake -i 9001 -t custom -n custom-ubuntu-2204-server -c minio
./playbooks/custom.yml
1---
2# Bake custom-server
3
4- name: "Bake proxmox custom vm template"
5 hosts: localhost
6 become: yes
7
8 vars_files:
9 - ./vars/apt.yml
10 - ./vars/maintenance.yml
11
12 vars:
13 - vault_enable: true
14
15 roles:
16 - role: ./roles/apt.ops/set-sources.ops/
17 vars:
18 release: "jammy"
19 - role: ./roles/apt.ops/install-packages.ops/
20 vars:
21 extra_packages:
22 - neofetch
23
24 - role: ./roles/maintenance.ops/key.ops/
25
26 - role: ./roles/docker.ops/
27
28 - role: ./roles/proxmox.bootstrap/
./custom-proxmox-packer-template.pkr.hcl
1...
2
3build {
4
5 name = "minio"
6
7 sources = ["source.proxmox.bakery-template"]
8
9 # Provisioner Configurations
10
11 # SSH public key
12 provisioner "file" {
13 source = "./id_rsa.pub"
14 destination = "/tmp/id_rsa.pub"
15 }
16
17 # Minio plabyook
18 provisioner "file" {
19 pause_before = "5s"
20 source = "./playbooks/.vault_pass"
21 destination = "/tmp/.vault_pass"
22 }
23 provisioner "ansible-local" {
24 playbook_dir = "./playbooks"
25 playbook_file = "./playbooks/minio.yml"
26 clean_staging_directory = true
27 extra_arguments = [
28 "--vault-password-file=/tmp/.vault_pass",
29 "--extra-vars \"ansible_user=packer\""
30 ]
31 }
32
33 # Main playbook depends of vm_type
34 provisioner "ansible-local" {
35 pause_before = "5s"
36 playbook_dir = "./playbooks"
37 playbook_file = var.playbook_file
38 clean_staging_directory = true
39 extra_arguments = [
40 "--extra-vars \"ansible_user=packer\""
41 ]
42 }
43
44 # Convert to proxmox vm template
45 post-processor "shell-local" {
46 inline = [
47 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --boot c --bootdisk scsi0",
48 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --ciuser ${var.ssh_username}",
49 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --cipassword ${var.ssh_password}",
50 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --serial0 socket --vga serial0",
51 "ssh root@${var.proxmox_host} qm set ${var.vm_id} --delete ide2"
52 ]
53 }
54}
You should see some output for each of the builders
, provisioners,
and post-processors
.
1# ./bake -i 9001 -t custom -n docker-ubuntu-2204-server-template -b minio -f ./vars/kevin-ubuntu-2204.json
2
3########## Baking docker-ubuntu-2204-server-template template with packer
4
5minio.proxmox.bakery-template: output will be in this color.
6
7==> minio.proxmox.bakery-template: Creating VM
8==> minio.proxmox.bakery-template: Starting VM
9==> minio.proxmox.bakery-template: Starting HTTP server on port 8802
10==> minio.proxmox.bakery-template: Waiting 5s for boot
11==> minio.proxmox.bakery-template: Typing the boot command
12==> minio.proxmox.bakery-template: Waiting for SSH to become available...
13
14...
15
16==> minio.proxmox.bakery-template: Uploading ./id_rsa.pub => /tmp/id_rsa.pub
17 minio.proxmox.bakery-template: id_rsa.pub 743 B / 743 B [=================================================================] 100.00% 0s
18==> minio.proxmox.bakery-template: Pausing 5s before the next provisioner...
19==> minio.proxmox.bakery-template: Uploading ./playbooks/.vault_pass => /tmp/.vault_pass
20 minio.proxmox.bakery-template: .vault_pass 21 B / 21 B [==================================================================] 100.00% 0s
21==> minio.proxmox.bakery-template: Provisioning with Ansible...
22 minio.proxmox.bakery-template: Uploading Playbook directory to Ansible staging directory...
23 minio.proxmox.bakery-template: Creating directory: /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37
24 minio.proxmox.bakery-template: Uploading main Playbook file...
25 minio.proxmox.bakery-template: Uploading inventory file...
26 minio.proxmox.bakery-template: Executing Ansible: cd /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37 && ANSIBLE_FORCE_COLOR=1 PYTHONUNBUFFERED=1 ansible-playbook /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37/minio.yml --extra-vars "packer_build_name=bakery-template packer_builder_type=proxmox packer_http_addr=10.178.0.50:8802 -o IdentitiesOnly=yes" --vault-password-file=/tmp/.vault_pass --extra-vars "ansible_user=packer" -c local -i /tmp/packer-provisioner-ansible-local/626557b5-6bf5-0aba-7ab2-50b0916afe37/packer-provisioner-ansible-local2268832455
27
28...
29
30 minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Install Docker with script] ************************
31 minio.proxmox.bakery-template: changed: [127.0.0.1]
32 minio.proxmox.bakery-template:
33 minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Enable docker to start at boot] ********************
34 minio.proxmox.bakery-template: ok: [127.0.0.1]
35 minio.proxmox.bakery-template:
36 minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Add user to docker group] **************************
37 minio.proxmox.bakery-template: changed: [127.0.0.1]
38 minio.proxmox.bakery-template:
39 minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Post installation message] *************************
40 minio.proxmox.bakery-template: ok: [127.0.0.1] => {
41 minio.proxmox.bakery-template: "msg": "Use \"newgrp docker\" to use the group immediately"
42 minio.proxmox.bakery-template: }
43 minio.proxmox.bakery-template:
44 minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Check if docker-compose is installed] **************
45 minio.proxmox.bakery-template: ok: [127.0.0.1]
46 minio.proxmox.bakery-template:
47 minio.proxmox.bakery-template: TASK [./roles/docker.ops/ : Install docker-compose] ****************************
48 minio.proxmox.bakery-template: changed: [127.0.0.1]
49
50...
51
52 minio.proxmox.bakery-template: PLAY RECAP *********************************************************************
53 minio.proxmox.bakery-template: 127.0.0.1 : ok=34 changed=21 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
54 minio.proxmox.bakery-template:
55 minio.proxmox.bakery-template: Removing staging directory...
56 minio.proxmox.bakery-template: Removing directory: /tmp/packer-provisioner-ansible-local/626557c0-c15c-4738-aebb-53538294728e
57==> minio.proxmox.bakery-template: Stopping VM
58==> minio.proxmox.bakery-template: Converting VM to template
59==> minio.proxmox.bakery-template: Adding a cloud-init cdrom in storage pool sata-pool
60==> minio.proxmox.bakery-template: Running post-processor: (type shell-local)
61==> minio.proxmox.bakery-template (shell-local): Running local shell script: /tmp/packer-shell1584895912
62 minio.proxmox.bakery-template (shell-local): update VM 9001: -boot c -bootdisk scsi0
63 minio.proxmox.bakery-template (shell-local): update VM 9001: -ciuser packer
64 minio.proxmox.bakery-template (shell-local): update VM 9001: -cipassword <hidden>
65 minio.proxmox.bakery-template (shell-local): update VM 9001: -serial0 socket -vga serial0
66 minio.proxmox.bakery-template (shell-local): update VM 9001: -delete ide2
67Build 'minio.proxmox.bakery-template' finished after 5 minutes 58 seconds.
68
69==> Wait completed after 5 minutes 58 seconds
70
71==> Builds finished. The artifacts of successful builds are:
72--> minio.proxmox.bakery-template: A template was created: 9001
73--> minio.proxmox.bakery-template: A template was created: 9001
74The last command took 359.68 seconds.
Up to this point, the custom VM template with Docker
, Docker-Compose
, and Minio Client
pre-installed has been successfully uploaded to the Proxmox server.
To sum up, with such an approach, we can deploy a new VM in minutes and drastically speed up the DevOps process. Ansible is a very powerful tool as it opens up many opportunities to basically automate any shell-based tasks. Next, I plan to write another post to further extend the automation with Terraform and Terragrunt to deploy a VM in Proxmox with the VM template created by Packer.
You should now have a good starting point for building Proxmox VM templates with Packer. If your looking to extend its usefulness a little further check out these useful articles.
After some intensive research and consideration, I found out that virtualizing unRIAD Server in VMWare ESXi 7.0 seems like a possibility. Due to the …
Building images from a standard Dockerfile typically relies upon interactive access to a Docker daemon, which requires root access on your machine to …