In this guide, I’ll walk you through the complete upgrade process from VCF 9.0 to VCF 9.1, covering every major component involved in the lifecycle.
Along the way I’ll share what went wrong, what I did to fix it and lessons learned.
Summary
Since this guide is quite extensive, the summary below will help you find the section you need.
Upgrade VCF Operations
Before we start with the upgrade, it’s a good idea to run the pre-check in VCF Operation UI.
To do this, navigate to Fleet Management > Lifecycle > VCF Management.
Under VCF Operations, click Upgrade.
Once the pre-check completes successfully, verify that no issues have been reported. If everything looks good, you’re ready to proceed with the upgrade.
The actual upgrade is performed from a different interface in VCF Operations. You can access it using the this url: https://<your_vcf_operations_fqdn>/admin.
Then navigate to Software Update.
To perform the upgrade, you’ll need the .pak file, which can be downloaded from the Broadcom Support Portal.
After uploading the .pak file, proceed with the installation. The remaining steps are straightforward and can be completed by following the on-screen instructions.
Important: After we upgrade VCF Operations to version 9.1, your VCF Fleet Management VM was powered off. This is expected, as the Fleet Management component still exists in the VCF 9.1 architecture but no longer runs on a dedicated VM.
Upgrade SDDC Manager
Before upgrading SDDC Manager, we need to download all the binaries. I won’t go into that process here, so I’m assuming you have already downloaded all the required binaries into SDDC Manager.
Also, before proceeding with the upgrade, we can run the pre-check to verify whether there are any errors.
Once we have confirmed it’s all good, we can proceed with the upgrade.
Install VCF Management Services
After SDDC Manager has been successfully upgraded, we need to deploy VCF Management Services.
Under Build > Lifecycle, select your VCF Instance and click in Install.
Enter your VCF Operations FQDN.
This is the step where many people get confused.
To perform a clean installation of the VCF Services Runtime, there are a few important prerequisites to keep in mind. You can find the complete list in the official documentation, but the main requirements are:
Now it’s the time where many people get confused.
To perform a clean installation of the VCF services runtime, there are a few important prerequisites to keep in mind. You can find them here, but the main ones are:
A minimum of 12 IP addresses (/28 block) are required.
Up to 30 IP addresses (/27 block) in total can be added and are reserved for new VCF management services components or scaling out existing components.
All required FQDNs must resolve to unique IP addresses that are outside the VCF Services Runtime IP range.
The VCF Services Runtime FQDN is used as a prefix for the virtual machine names.
You can’t use capital letters in VCF services runtime FQDN.
After you deploy VCF Management Services, you’ll see 4 vms + the license server.
Upgrade VCF Automation
Now, we have to return to the VCF Operations UI and navigate to Build > Lifecycle.
Once the required binaries have been downloaded, click Upgrade.
Before proceeding, it’s important to pay attention to the target version we are upgrading to, as the networking requirements differ depending on the release.
9.1.0.400 and later
5 unique IP addresses that reside on the same network where your current Automation nodes are deployed.
The IP addresses can be contiguous or non-contiguous.
1 New FQDN with a unique corresponding IP for the VCF services runtime instance. The corresponding IP must be outside of the CIDR that you already dedicated.
9.1.0.0 through 9.1.0.300
A required block of 8 sequential IP addresses, which mean a /29 subnet mask.
1 New FQDN with a unique corresponding IP for the VCF services runtime instance. The corresponding IP must be outside of the CIDR that you already dedicated.
For more detailed information, consult the Broadcom documentation.
Once everything is prepared, we can run the pre-checks and start the upgrade.
And see it failed!
Even though I thought I had done everything correctly, the deployment failed. Unfortunately, it failed in a way that prevented me from simply retrying the upgrade.
I spent a lot of time investigating different log files, pods, and testing everything I can, until I finally figure out the problem.
Wrong folder! 🤡
At some point, I had moved my VCF Automation virtual machines into a different vCenter folder, and that caused the upgrade to fail.
Why?
During the upgrade, VCF Operations deploys an entirely new VCF Automation cluster and migrates all the existing configuration to it. As one of the final steps, the upgrade process locates the original VCF Automation virtual machines and powers them off before completing the migration.
You can verify which folder the installer expects by connecting to your VCF Automation appliance and running the following command:
k get pd vmsp-platform -n vmsp-platform -o jsonpath='{.spec.values.provider.vsphere}' | jq | grep folder
The installer always expects the virtual machines to be located in the output folder of the command above. If they are not there, the validation fails, the new cluster is never successfully deployed, and you end up with two useless VCF Automation clusters.
You can find a detailed explanation on KB441260.
What now?
The KB recommends moving the virtual machines back to the expected folder and retrying the upgrade. That sounded simple enough… except there was one problem: there was no Retry button.
I tried starting a new upgrade task instead, but it failed with different errors. At that point, the only recommendation left was to open a support case with Broadcom.
Since I’m using my lab, I just decided to move forward, cleanup my broken VCF Automation deployment and install a brand new one.
But how to remove it?
Fortunately, I found another interesting KB that pointed me in the right direction. Since I had nothing to lose, I decided to give it a try.
I’ll write a dedicated article explaining what I did, since it turned out to be kind of fun.
After removing the previous deployment, I followed the same steps I described earlier, but this time I performed a fresh VCF Automation deployment instead of an upgrade. The overall process is very similar to the upgrade workflow, but much cleaner since it starts from a blank environment.
Although I wasn’t able to complete the upgrade path successfully, I still wanted to share my experience because it may save someone else a significant amount of troubleshooting time.
My biggest lesson was:
Do not move the VCF Automation VMs to another folder.
You can see in the images below that I manage to have my VCF Automation on version 9.1 after all.
Upgrade NSX Manager
Continuing the saga—now it’s time to upgrade NSX Manager.
As always, we return to the VCF Operations UI and navigate to Build > Lifecycle.
Select the Management Domain and open the Upgrade page. The first step is to create an upgrade plan.
Start by selecting the target version.
In my case, I’ll upgrade all the components to VCF 9.1.0.0.
Next, choose the upgrade strategy. There are two available options:
- Optimized Maintenance Window Upgrade
- Sequential Upgrade
While the first one is focused on minimize downtime by completing the upgrade within a single maintenance window, the second one upgrades the components in a more gradual, step-by-step fashion.
For production environments, use the Optimized maintenance window upgrade option.
The table below summarizes the main differences between the two approaches:
I chose the second option so I could better understand each stage of the process.
Once the upgrade plan has been created, run the pre-checks before proceeding.
It’s also important review the official Broadcom documentation for the NSX upgrade.
My pre-checks reported two errors. Fortunately, both were related to backups.
Don’t ignore the warnings either.
In my case, there was three, which can be addressed using the following KBs:
- “vCenter backup age – Resolved after configuring a valid vCenter backup.
- “vCenter cloud system health – See KB336395
- Check for data inconsistencies in DB – See KB440901
After configuring valid backups for both vCenter and NSX Manager, I ran the pre-checks again. This time, everything passed successfully, allowing me to proceed with the upgrade.
And this time, it all went well.
NSX upgraded!
Upgrade vCenter Server
In order to upgrade the vCenter Server, we go back to the Lifecycle page and under the Upgrade Sequence, click Configure.
Here we can define the upgrade mechanism for the vCenter upgrade.
Note: This step applies only to the vCenter upgrade and is different from the combined vCenter + NSX upgrade covered in the previous section.
To reduce the maintenance window, I selected the vCenter Reduced Downtime Upgrade option.
Next, confirm that you have a valid backup of the vCenter Server.
You must also provide a temporary IP address that will be used during the upgrade process.
Then, select the deployment size that best matches your environment.
Once the upgrade plan has been configured, run the vCenter pre-check.
Note: Always run the pre-checks!
In my case, I received the warning shown below. It can be safely ignored, but if you’re curious about it, see the KB380723 for more details.
After the pre-check, schedule the upgrade and proceed with the installation.
Once the process is finished, you should see that the vCenter Server has been upgraded successfully.
Upgrade ESXi
We are almost there. It’s time to upgrade the ESXi hosts.
Before starting the upgrade, we need to create a desired image in the vCenter Server.
To do this, open the vCenter Server UI, navigate to Lifecycle Manager, and click Import Image.
You can download the VMware-ESXi-9.1.0.0.25370933-depot.zip file from the Broadcom Support Portal.
Once the file has been downloaded, import it into Lifecycle Manager and create a new image using the ESXi version you just imported.
After the image is ready, go back to the VCF Operations UI and navigate to Build > Lifecycle > VCF Instance > Image Management.
Click Import Image.
Select the image you created in the previous step.
Create a new upgrade plan.
Select the cluster you want to upgrade.
Assign the image you just imported to the cluster.
Configure the upgrade options.
Select the standalone host, if any.
And Finish.
Before proceeding with the upgrade, run the pre-checks to verify that there are no issues. If all checks pass, you can start the upgrade.
As you can see, the upgrade completed successfully without any problems.
Upgrade NSX Edges Nodes
Finally, it’s time to upgrade the NSX Edge nodes.
Let’s configure how the upgrade will be performed.
First, select the target Edge Cluster.
In my case, I selected the checkbox below to prevent the Edge nodes from being upgraded in parallel.
Review the configuration and click Finish.
Once the pre-checks have completed successfully, you can start the upgrade.
And… done!
Conclusion
As you can see, I have successfully upgraded all of my VCF components to version 9.1.0.
