Skip to content

Use with Nested ESXi and vCenter Server

Jump to bottom
kurokobo edited this page Feb 2, 2022 · 7 revisions

Please note this page doesn't describe basic operations like installation, starting the daemon, registration, and so on. Please check README.md first.

In the vCenter Server, by using VirtualBMC for vSphere (0.0.3 or later), you can enable the vSphere DPM: Distributed Power Management feature for Nested-ESXi host that is running in your VMware vSphere environment. The VirtualBMC can act as a BMC of Nested-ESXi same as the physical BMC of physical ESXi.

So you can achieve:

  • Power-On the virtual ESXi in the same way as for physical ESXi. Power-On

  • By enabling vSphere DPM for the cluster with virtual ESXis, the automated power control of ESXi hosts based on the load of the host cluster will partially work in your nested environment. It means the virtual ESXi will be entered standby mode or powered on by vCenter Server fully automatically. DRS

Notice

Please be careful with the recommendations presented in the vSphere DPM in nested environments may not be accurate or meet expectations because nested environment cannot fully mimic the physical environment.

  • The vSphere DPM uses several metrics to determine whether the host should be turned on and off.
  • It relies more strongly on information obtained directly from ESXi's own hardware than just information obtained by IPMI over LAN that can be emulated by VirtualBMC.
  • However, the information available from virtual hardware in a nested environment is very limited.
    • One of the typical information is power consumption. But the power consumption of the nested ESXi itself is always ZERO.
    • This can be checked in the ESXi host's performance chart.
  • Also, the ACPI P-State and C-State are not available in the nested environment, therefore the CPU power management cannot be used.
  • As a result, the cost calculations may not be accurate and the vSphere DPM recommendations may not be as expected.

VirtualBMC for vSphere configuration

This is an example configuration diagram.

VirtualBMC for vSphere with Nested-ESXi and vCenter Server

To act as the BMC of the ESXi, the BMC has to listen on the port 623. The vCenter Server always communicates with the BMC on the port 623. It is designed behavior and seems can't be changed.

So you have to consider:

  • Prepare one dedicated IP address for each ESXi.
  • Bind the fixed IP address and port to your virtual BMC for each ESXi.
  • Run your VirtualBMC daemon by the privileged user because the port below 1025 is restricted and only users with privilege will be able to bind.

In addition, the vCenter Server has an extra sanity check to avoid talking to the wrong ESXi's BMC because of an error in entering the IP address. It means the MAC address you entered in vCenter Server has to be matched with the MAC address included the response from the BMC. The VirtualBMC helps you to pass this check.

Preparing multiple IP addresses

On the VirtualBMC host, the easiest way to get multiple IP addresses is IP aliases.

  • On Ubuntu
    • Add multiple IP addresses to your /etc/netplan/99_config.yaml and invoke sudo netplan apply. See the Static IP Address Assignment in this document.
    • This is an example of 99_config.yaml. 
      network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: false
      dhcp6: false
      addresses:
      - 192.168.0.241/24
      - 192.168.0.242/24
      - 192.168.0.243/24
      - 192.168.0.244/24
      gateway4: 192.168.0.1
      nameservers:
        addresses:
        - 192.168.0.1
  • On CentOS
    • Add multiple IP addresses by nmcli command.
    • This is an example of commands. 
      $ nmcli con show
NAME    UUID                                  TYPE      DEVICE
ens160  1218fb58-9bf9-4a9b-bfc7-c8c12a064293  ethernet  ens160

$ sudo nmcli con mod ens160 +ipv4.addresses "192.168.0.241/24"
$ sudo nmcli con mod ens160 +ipv4.addresses "192.168.0.242/24"
$ sudo nmcli con mod ens160 +ipv4.addresses "192.168.0.243/24"
$ sudo nmcli con mod ens160 +ipv4.addresses "192.168.0.244/24"

Then check the configuration:

$ ip a
...
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:50:56:86:d0:dd brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.241/24 brd 192.168.0.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet 192.168.0.242/24 brd 192.168.0.255 scope global secondary ens160
       valid_lft forever preferred_lft forever
    inet 192.168.0.243/24 brd 192.168.0.255 scope global secondary ens160
       valid_lft forever preferred_lft forever
    inet 192.168.0.244/24 brd 192.168.0.255 scope global secondary ens160
       valid_lft forever preferred_lft forever
...

Start the VirtualBMC daemon

As described before, the daemon has to run by privileged users to use port 623. So you have to run the daemon with sudo or simply run it by root user.

sudo vsbmcd --foreground

Note that --foreground option is useful for troubleshooting.

Create new BMCs

Add your virtual BMC for your Nested-ESXi VMs by vsbmc command. Again, this is an example diagram.

VirtualBMC for vSphere with Nested-ESXi and vCenter Server

In this example, here I have 3 VMs as Nested-ESXis named lab-vesxi01, 02, and 03 on the ESXi in my VMware vSphere environment. And those Nested-ESXis registered as actual ESXis named vESXi01, 02, and 03 in the host cluster in the vCenter Server.

Due to the behavior of vCenter Server, the BMC has to use port 623. To avoid port confliction, you have to specify one IP address for each virtual BMC by --address option.

This is my example:

$ vsbmc add lab-vesxi91 --address 192.168.0.242 --port 623 --viserver 192.168.0.201 --viserver-username [email protected] --viserver-password Vmware123!
$ vsbmc add lab-vesxi92 --address 192.168.0.243 --port 623 --viserver 192.168.0.201 --viserver-username [email protected] --viserver-password Vmware123!
$ vsbmc add lab-vesxi93 --address 192.168.0.244 --port 623 --viserver 192.168.0.201 --viserver-username [email protected] --viserver-password Vmware123!

And then start each virtual BMC with vsbmc start command.

Get the MAC addresses per virtual BMC

Once the new virtual BMC has been created, internally, the fake MAC address has been generated. You can get this by vsbmc list --fakemac command.

These MAC addresses will be used when you register the BMC in vCenter Server in a later step.

$ vsbmc list --fakemac
+-------------+---------+---------------+------+-------------------+
| VM name     | Status  | Address       | Port | Fake MAC          |
+-------------+---------+---------------+------+-------------------+
| lab-vesxi91 | running | 192.168.0.242 |  623 | 02:00:00:56:b4:bc |
| lab-vesxi92 | running | 192.168.0.243 |  623 | 02:00:00:40:54:d2 |
| lab-vesxi93 | running | 192.168.0.244 |  623 | 02:00:00:7e:e5:4c |
+-------------+---------+---------------+------+-------------------+

Alternatively, you can use ipmitool to get the fake MAC address by invoking print lan 1 command to the port 623 of your IP addresses that you specified. The values should match the values you checked in the above procedure.

$ ipmitool -I lanplus -H 192.168.0.242 -p 623 -U admin -P password lan print 1
MAC Address             : 02:00:00:56:b4:bc
Cipher Suite Priv Max   : Not Available
Bad Password Threshold  : Not Available

$ ipmitool -I lanplus -H 192.168.0.243 -p 623 -U admin -P password lan print 1
MAC Address             : 02:00:00:40:54:d2
Cipher Suite Priv Max   : Not Available
Bad Password Threshold  : Not Available

$ ipmitool -I lanplus -H 192.168.0.244 -p 623 -U admin -P password lan print 1
MAC Address             : 02:00:00:7e:e5:4c
Cipher Suite Priv Max   : Not Available
Bad Password Threshold  : Not Available

Note that these fake MAC addresses are truly a fake. You can't communicate to these MAC addresses. It was generated only to pass the sanity check by the vCenter Server.

vCenter Server configuration

Follow Configure IPMI or iLO Settings for vSphere DPM to register the BMC per your ESXi.

The required parameters are:

  • Enter the User name and Password as configured in VirtualBMC.
    • By default, the username and password are configured as admin and password.
  • Enter the dedicated IP address for your virtual ESXi host in the BMC IP address field.
    • In this example, for the ESXi host nested in VM lab-vesxi01, it should be configured as 192.168.0.242.
  • Enter the fake MAC address for your virtual BMC in the BMC MAC address field.
    • In this example, for the ESXi host nested in VM lab-vesxi01, it should be configured as 02:00:00:56:b4:bc.
    • You can check your fake MAC address by vsbmc list --fakemac command.

Once the OK pressed, the sanity check has been invoked. Check the task will be completed.

Test the Power-On by vCenter Server

Once the BMC has been registered, now you can enable vSphere DPM for the cluster. But before this, you should ensure that vCenter Server can power-on your Nested-ESXi correctly.

  • Enter Standby Mode.
    • Select [Power] > [Enter Standby Mode] from the host right-click menu.
    • Wait until the host entered standby mode.
    • Once the host entered standby mode, ensure that the backend VM of the ESXi is stopped.
  • Power-On the ESXi.
    • Select [Power] > [Power On] from the host right-click menu.
    • Ensure that the backend VM of the ESXi is powered on.
    • Wait until the host exited standby mode.
  • Check the rest of your Nested-ESXis in the same way.

Enable vSphere DPM

Finally, you can enable vSphere DPM. Once you select Automatic as the Automation Level, you can observe the automated power control of ESXi hosts based on the load of the host cluster will partially work in your nested environment.

It means the virtual ESXi will be entered standby mode or powered on fully automatically by vCenter Server.

Please be careful with the recommendations presented in the vSphere DPM in nested environments may not be accurate or meet expectations. See the Notice section above.

Clone this wiki locally