Upgrade vCenter with an ISO – Upgrade vCenter HA

Getting vCenter upgraded is fairly straight forward. However, if you have a vCenter HA configuration, it becomes a little more complex. Not difficult but there are some things you have to find before you can upgrade. I love VMware, don’t get me wrong. But, the documentation for upgrading vCenter HA is pretty…. vague and disjointed. So I wanted to go into detail to get through so that you know what to expect when upgrading.

This upgrade process is going to go over how to do this in a vCenter HA configuration. However, you can follow the same steps to do a non-vCenter HA deployment. You just won’t do any of the HA specific steps (skipping the upgrade on the passive and witness nodes). You’ll only update your primary vCenter instance and not the passive or witness nodes.

I’ve spent quite a bit of time with vCenter HA and have run into issues. I’ll mention those related to updating in this post. I do feel vCenter HA isn’t fully baked. It’s extremely quirky. 

vCenter HA Issues

  • You cannot upgrade a vCenter HA cluster unless it’s in maintenance mode. Even if you disable the vCenter HA cluster, it still won’t let you upgrade. Here is the message when I tried to updated the witness node when I’ve disabled vCenter HA:

vCenter HA installing patch is allowed only on passive or witness node

  • So, we put the cluster into maintenance mode before we upgrade. No worries right? Wrong. While you can update the witness and passive nodes, you cannot update the active node (it will throw a similar error as the one above). Well, why don’t I just failover the cluster? So if the passive and witness nodes are in this weird state (see pic below), you cannot failover. It’s like this chicken and egg situation:

vCenter HA status

  • You must acquire the vCenter patch ISO. You have to drill down into the docs to find this out… and frankly, it would be super confusing for someone that is just starting out. 
  • Once you get the cluster corrected (from the above screenshot), vCenter HA may or may not fail over so you can update the active node. It’s like it loses it’s brains as soon as you have two vCenters with differing versions even though the documentation does not mention any of these issues. It can just sit at:

vCenter HA Failover

vCenter HA Upgrade

We will tackle all of this during this article.

  1. The first thing we need to do is grab the vCenter patch ISO. You get that here: https://my.vmware.com/group/vmware/patch
  2. You’ll use the drop downs to select “VC” and then the version you’ll be upgrading to. Such as the following:

vCenter Patch

  1. Click SEARCH and you’ll get:

vCenter Patch ISO

  1. Go ahead and click DOWNLOAD and save it somewhere that you’ll remember
  2. Open consoles and mount the ISO to all of the vCenter VMs. You can also upload the ISO to a datastore and mount it that way. Your call.
  3. Log into the appliances with root and drop yourself to a shell prompt. Do not type shell to bypass this. We are going to work within the Command> prompt:

vCenter shell prompt

  1. If you do not see this prompt when you log in, please reference the Appendix at the end of this article
  2. We are going to type: software-packages install –iso –acceptEulas (that’s two dashes)

vCenter software-packages iso update

  1. As it progresses, it will complete and then reboot.
  2. Once the witness node is upgraded, you will see the following in your HA status:

  1. Go ahead and do the same things for the passive node.
  2. We are going to see the following when we upgrade the passive node:

  1. Now this could be a waiting game. The passive and witness nodes should come back and show UP. This will allow you to initiate a failover so that you can update the active node.
  2. Ideally vCenter HA will sync up properly. But if you are waiting 30+ minutes, it might be time to reboot the active node. I’ve run into this and rebooting the active node seems to fix it.

vCenter HA status

  1. If vCenter is still active on the node you need to upgrade, you can now click on INITIATE FAILOVER. You will need to wait until vCenter is back up to continue. So validate that you can log into vCenter prior to moving forward.
  2. Once vCenter is back online, go ahead and mount the patch ISO to the new passive node.
  3. We are going to type: software-packages install –iso –acceptEulas (that’s two dashes)
  4. That node will reboot and should come back into the cluster
  5. At this point, you should be good to do another INITIATE FAILOVER just to test things out
  6. Go ahead and edit the vCenter HA cluster and go back to Enable vCenter HA
  7. At this point I’ve gotten a couple of errors but the most predominate one is:

vCenter a general system error occurred nodes are not running at the same version

  1. I’ve gone on vCenter and issued a vpxd -v and all of the nodes come back with the same version and build numbers. 
  2. The fix (skip this if your upgrade was successful):
    1. We have to remove vCenter HA and readd it. So let’s get started
    2. Power off the passive and witness nodes and delete them
    3. Let vCenter HA go into error state for the active and passive nodes (where it shows Down for both).
    4. At this point, I’ve seen vCenter HA go into some weird state where it tosses an error about connection state and all the nodes go into Unavailable state. Usually this clears and then shows that vCenter HA as isolated.
    5. SSH into vCenter and get to a bash prompt. Type vcha-destroy
    6. Now, if you get an error you’ll now need to force the destroy. So type vcha-destory -f
    7. This process will take a few minutes

Destroy vCenter HA

    1. I noticed that we have some errors. So, that was worrisome. But, I’m chalking it up to the quirkiness of vCenter HA because when I go look at vCenter HA within vCenter, it’s gone. vCenter HA is ready to be reconfigured
    2. Click on SET UP VCENTER HA
    3. Go through the deployment. You’ve already gone through this since you had vCenter HA up prior. So I won’t go through the steps here (especially since I don’t know which type of deploy you did). However, you do not have to add a second NIC to vCenter and IP it. That is already there. It was never removed during the destroy operation

You should now have a fully functional vCenter HA cluster that’s been updated (since the primary node was updated previously).

Leave a Reply

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.