Tuesday, 6 August 2013

ESXi Home Lab with Intel Haswell CPU

I am currently running home lab for learning virtualisation and new technologies. This system was integrated with GA-B75M-D3H motherboard, i7 -3770 @ 3.4 GHz by me.
I was craving to add an additional system in my lab with an Intel Haswell CPU from the day this Processors were available to public. However I wanted to confirm about any known issues before burning a hole in my wallet.
These new Haswell motherboards are either installed with Realtek or Intel based LAN controllers and new SATA chipsets are provided by Intel B85/H87 or Z87 controllers. These on-board controllers may impact the ESXi installation or give serious PSOD errors or booting problems which could put the entire build in vain.
Famous bloggers vladan and Ivo Beerens built their system with Asrock Z87 and GA-Z87-D3HP motherboards respectively. They have mentioned about the on-board Gigabit NIC I217V is not recognised by ESXi and had to go with add-on NIC.
I decided to try with Realtek chipset based motherboard which may be supported by ESXi native drivers as it worked without any issue in my previous B75 based IvyBridge system.
I bought a GA-Z87M-D3H Gigabyte motherboard and Intel i7 4770 CPU. This motherboard supports 4 x 8 GB DDR3, 6 x SATA3 6 Gb/s ports, 2 x PCI Slots, 1 x PCIEX16, 1 x PCIEX4 in Micro ATX real-estate.
The reason I went ahead with this motherboard is based on below assessments:
  1. Micro ATX form Factor - will keep the system footprint smaller
  2. Entry level Z87 Chipset with reasonable price - This system is not for gaming purpose
  3. Supports up to 32 GB DDR3 memory - can run more VMs
  4. Supports VT-d - To test device passthrough
  5. PCI 2.0 - In case on-board NIC backfire
  6. 6 x SATA3 6 Gb/s ports (Can be converted as NAS system with i3 or Pentium processors)
My Haswell based test box consists of following parts:
Installation and Testing:
Installed ESXi 5.1 trial version in the USB Drive was smooth. Unfortunately Realtek On-board NIC is not supported by ESXi. So I had to use the PCI Intel NICs that I bought from e-bay. These add-on cards are detected by ESXi without any issue. I have configured and tested these cards with 8192 Jumbo frames as well. I am really happy that I bought couple of these cards in spare which came as handy during this build and the price of the network cards are very much cheaper. On-board SATA controller was detected by ESXi. I tested this by connecting a SATA drive which I pulled out from another desktop.
Before buying this motherboard I read the manual and confirmed myself that VT-d is supported. So my next "test item" was to confirm about VT-d support which will allow us to passthrough the devices to test VMs. I logged in through vSphere Client and selected configuration -> Advanced settings -> Configure Passthrough option.
Only below given controllers are visible in Z87M-D3H motherboard.
  • Intel Haswell integrated Graphics controller
  • Intel HD Audio Controller
  • Intel Lynx USB Host Controller
  • Intel Lynx Point Audio Controller
  • Intel Lynx PCI Express Root Port
  • Realtek 8168 Gigabit Ethernet (Highlighted in the image, more about that is given below)
  • Intel PCI Bridge
I couldnot find AHCI controller visible in Devices Passthrough list. I was expecting AHCI will be available since VT-d was mentioned in motherboard manual. To clarify myself, I created a support case with Gigabyte and put my concern. I received a response from the support team. They were suggesting me to check with 3rd party vendors for AHCI support which they are not having visibility to. I raised query about the VT-d support which they have mentioned in their manual.
After a few days I received a reply from the support engineer with a screenshot. They tested the on-board PCI slot with an add-on NIC card and confirmed that VT-d is working fine. We can appreciate the effort that they took to test with an add-on card. However AHCI visibility in ESXi is not confirmed by them. This may be visible in future versions.
I double checked the VT-d support related image provided by vladan and found Asrock is also not showing up the AHCI controller.

IMHO, VT-d should be 100% useful when the AHCI is also supported. I confirmed this from previous B75M-D3H based testing which I have posted here for reference.
Even though the VT-d passthrough is not working I am not having much impact due to that. I have another system running with FreeNAS which provides NFS and iSCSI services to this ESXi test box. I am not having any issue for running my Linux test VMs from this NAS box. (I will update about this NAS build in my next blog).
On Board LAN controller - Additional Testing:
Since VT-d is allowing me to passthrough the on-board Realtek 8168 chipset, I tested it by, passing it to a test Windows machine running inside a VM. For all my surprise, I can make this network card working by installing the driver in the VM. I used the windows driver available in CD which came with this motherboard.
Moral of the build:
If you are looking for a Haswell based Home lab system build and you will keep your VMs in separate NAS box, Please by a B85 Chipset based motherboard and a Haswell processor i5 or i7 according to your budget. The B85 chipset motherboards are capable of supporting 32 GB of memory and the PCI ports found on these boards can be configured with Intel based add-on network interface cards for supproting the hypervisors. You can buy an additional memory module or add-on NIC from the money that you saved yourself by keeping away from Z87 chipset. IMHO Z87 will be a over kill when AHCI cannot be passedthrough to a VM which may be made as NAS box for nested ESXi host's datastore purpose.