As I mentioned in the previous post, I recently started a new open source VMware Users Group (VMUG) project with some new friends with a goal of automatically provisioning the necessary virtual infrastructure for VMware-oriented hackathons on the VMware Cloud on AWS (VMC) platform using HashiCorp Terraform and Gruntwork Terragrunt, in a simple and cost effective manner. The project is called VMUG Labs and the source code can be found in the GitHub repository.

As indicated by the title of this post, you would only need to make a few tweaks to make this work in your home lab, work lab, or any vSphere environment where you want to deploy nested ESXi hosts, because VMC can use the standard Terraform VMware vSphere provider for provisioning VMs in your software-defined data center (SDDC). I’ll cover how to use Terraform to deploy nested ESXi hosts in your lab in more detail in the next post.

Prerequisites

Once I had provisioned a few firewall rules and a logical network in my SDDC, the first VM that I deployed in VMC (manually) was William Lam’s vSphere ESXi 6.5 U1 virtual appliance to use as my template VM for Terraform.

For those that are unfamiliar, his virtual appliance image facilitates provisioning automation by allowing the user to supply operating system configuration values such as hostname, IP address, et cetera, via vApp properties at provision time for achieving guest customization of the ESXi host VM (aka: nested hypervisor).

Next, you must set the debug vApp property in your VM template to be user configurable because the Terraform vSphere provider v1.3.3 and earlier does not support this. Attempts to deploy VMs from a VM or VM template with any vApp property that is not configurable will fail as documented in issue #394. I made a few minor adjustments to a copy of David Hekimian’s PowerCLI script for doing so and the source code can be found here. Run the following PowerShell script against your nested ESXi VM that you plan to use as a template:

./Enable-VmVappProperties.ps1 -Name 'Nested_ESXi6.5u1_Appliance_Template_v1.0'

Constraints

It is important to note that you are restricted from enabling the promiscuous mode and the forged transmits distributed port group security settings in VMC, so VMs running on nested hypervisors will be isolated (no network connectivity beyond the hypervisor). This constraint was not unexpected, but does limit the possibilities of what we can build somewhat. After much deliberation, our team came to the conclusion that most attendees of VMware-oriented hackathons would primarily want to interact with the virtual infrastructure and we accepted this limitation for the project. This constraint should not apply to your lab environments though.

Next, all VMs must be deployed under specific virtual infrastructure objects in your VMC SDDC- specifically, the Compute-ResourcePool resource pool, the Workloads VM folder, and the WorkloadDatastore datastore that are all found under the SDDC-Datacenter datacenter object.

Configure

To configure Terraform to provision your first nested ESXi host, you need to:

1. Create a Terraform configuration file, such as main.tf.

2. Configure the provider for connecting to vCenter:

   provider "vsphere" {
     version="~> 1.3"
     vsphere_server="vcenter.sddc-34-218-61-195.vmc.vmware.com" # Set this to your VMC SDDC FQDN
     allow_unverified_ssl=false
     user="cloudadmin@vmc.local"
     password="VMware1!" # Set this to your VMC SDDC password
   }
   

3. Configure data sources to retrieve information about the virtual infrastructure objects:

   data "vsphere_datacenter" "dc" {
     name="SDDC-Datacenter"
   }
   
   data "vsphere_resource_pool" "pool" {
     name="Compute-ResourcePool"
     datacenter_id="${data.vsphere_datacenter.dc.id}"
   }
   
   data "vsphere_datastore" "datastore" {
     name="WorkloadDatastore"
     datacenter_id="${data.vsphere_datacenter.dc.id}"
   }
   
   data "vsphere_distributed_virtual_switch" "dvs" {
     name="vmc-dvs"
     datacenter_id="${data.vsphere_datacenter.dc.id}"
   }
   
   data "vsphere_network" "network" {
     name="logical_network1"
     datacenter_id="${data.vsphere_datacenter.dc.id}"
   }
   

4. Configure the data source to retrieve information about the nested ESXi host VM template:

   data "vsphere_virtual_machine" "template" {
     name="Nested_ESXi6.5u1_Appliance_Template_v1.0"
     datacenter_id="${data.vsphere_datacenter.dc.id}"
   }
   

5. And now configure your first nested ESXi VM resource:

   resource "vsphere_virtual_machine" "vm" {
     name="ESXi1"
     guest_id="vmkernel65Guest"
     resource_pool_id="${data.vsphere_resource_pool.pool.id}"
     datastore_id="${data.vsphere_datastore.datastore.id}"
     folder="Workloads"
     num_cpus=2
     memory=6144
     wait_for_guest_net_timeout=0
   
     network_interface {
       network_id="${data.vsphere_network.network.id}"
     }
   
     disk {
       label="sda"
       unit_number=0
       size="${data.vsphere_virtual_machine.template.disks.0.size}"
       eagerly_scrub="${data.vsphere_virtual_machine.template.disks.0.eagerly_scrub}"
       thin_provisioned="${data.vsphere_virtual_machine.template.disks.0.thin_provisioned}"
     }
   
     disk {
       label="sdb"
       unit_number=1
       size="${data.vsphere_virtual_machine.template.disks.1.size}"
       eagerly_scrub="${data.vsphere_virtual_machine.template.disks.1.eagerly_scrub}"
       thin_provisioned="${data.vsphere_virtual_machine.template.disks.1.thin_provisioned}"
     }
   
     disk {
       label="sdc"
       unit_number=2
       size="${data.vsphere_virtual_machine.template.disks.2.size}"
       eagerly_scrub="${data.vsphere_virtual_machine.template.disks.2.eagerly_scrub}"
       thin_provisioned="${data.vsphere_virtual_machine.template.disks.2.thin_provisioned}"
     }
   
     clone {
       template_uuid="${data.vsphere_virtual_machine.template.id}"
     }
   
     vapp {
       properties {
         "guestinfo.hostname" = "esxi1"
         "guestinfo.ipaddress" = "" # Default = DHCP
         "guestinfo.netmask" = ""
         "guestinfo.gateway" = ""
         "guestinfo.vlan" = ""
         "guestinfo.dns" = "8.8.8.8"
         "guestinfo.domain" = ""
         "guestinfo.ntp" = "pool.ntp.org"
         "guestinfo.syslog" = ""
         "guestinfo.password" = "" # Default = VMware1!
         "guestinfo.ssh" = "True" # Case-sensitive string
         "guestinfo.createvmfs" = "False" # Case-sensitive string
         "guestinfo.debug" = "False" # Case-sensitive string
       }
     }
   
     lifecycle {
       ignore_changes= [
         "annotation",
         "vapp.0.properties",
       ]
     }
   }
   

The lifecycle section of the VM resource instructs Terraform to ignore changes to properties matching annotations and vapp.0.properties because the nested ESXi virtual appliance guest customization process removes most of the vApp properties at the end of the guest customization process so that sensitive data is not displayed on the deployed VM and also sets the annotation field. Since Terraform is unaware of the post-provision guest customization process, subsequent executions of terraform plan and/or terraform apply will flag the VM resource as being in a bad state and will recommend that the nested ESXi VM be destroyed and recreated if this is not set.

Automated provisioning

Once your configuration files are set, you’re ready to start the automated provisioning process by running the following commands:

• terraform init: Downloads the vSphere provider and prepares for interacting with the environment
• terraform plan: Pre-flight check
• terraform apply: Deploy your nested ESXi host!
• terraform destroy: Eradicate your nested ESXi VM when you’re done so that you can deploy it again

Reference

You can find lots more information about getting started with Terraform in the official Getting Started Guide.

Enjoy!

• terraform
• vmware
• vmware-cloud-on-aws