PowerCLI esxcli – A general system error occurred: Too many outstanding operations

I recently noticed that some of our scheduled automation’s failing with an error message – A general system error occurred: Too many outstanding operations.

The failure happens on some hosts when we use esxcli through PowerShell. It only affects ESXi 8 hosts – both 8.0 and 8U1 hosts. The issue is with an ESXi host because restarting the host resolves the problem for a while.

In vCenter, there is also a log entries in vpxd.log

2023-05-16T07:33:09.491Z error vpxd[04546] [Originator@6876 sub=Default opID=25c7611e] [VpxLRO] -- ERROR lro-33762714 -- 52c4a79f-6dd5-ab87-6ce1-09a069c89e77(52ae3e10-ab1d-348b-e010-c839e5709e02) -- ManagedMethodExecuter-56 -- vmodl.reflect.ManagedMethodExecuter.executeSoap: :vmodl.fault.SystemError
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Too many outstanding operations"
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f3764b07400, h:140, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-56/vpxa>]: executeSoap
--> Received SOAP response fault from [<<io_obj p:0x000000e087d6bae8, h:26, <TCP '127.0.0.1 : 21548'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: executeSoap
--> A general system error occurred: Too many outstanding operations"
--> }
--> Args:
-->
--> Arg moid:
--> "ha-cli-handler-network-nic"
--> Arg version:
--> "vim25/5.0"
--> Arg method:
--> "vim.EsxCLI.network.nic.get"
--> Arg argument:
--> (vmodl.reflect.ManagedMethodExecuter.SoapArgument) [
-->    (vmodl.reflect.ManagedMethodExecuter.SoapArgument) {
-->       name = "nicname",
-->       val = "<nicname xmlns="urn:vim25">vusb0</nicname>"
-->    }
--> ]

This error appears in the vpxd.log file several times with different “Arg method” values. From what I can see, all those failing methods are using vim.EsxCLI.

We will continue looking into this issue and open a case with VMware.

PowerCLI: Find FibreChannel HBA adapters that are not online

Simple PowerCLI to look for FC HBA adapters that are offline.

Write-Host "Getting all ESXi hosts"
$esxi_hosts = Get-Cluster | Get-VMHost -State Connected,Maintenance 

foreach ($esxi_host in $esxi_hosts) {
    $host_hbas = $esxi_host | Get-VMHostHba -Type FibreChannel | Sort-Object
    foreach ($host_hba in $host_hbas) {
        $host_hba_name = $host_hba.Name
        $host_hba_status = $host_hba.Status
        Write-Host $esxi_host - $host_hba_name - $host_hba_status
        if ($host_hba_status -inotmatch "online") {
            Write-Host "$esxi_host - $host_hba_name - $host_hba_status" -ForegroundColor Red
        }
    }
}   

PowerCLI: Find detached LUNs on ESXi hosts.

Script to find detached LUNs.

$clusters = Get-Cluster | Sort-Object -Property Name

foreach ($cluster in $clusters) {
  Write-Host $cluster
  $esxi_hosts = $cluster | get-Vmhost -State Connected,Maintenance | Sort-Object -Property Name

  foreach ($esxi_host in $esxi_hosts) {
    Write-Host $esxi_host
    ($esxi_host | Get-ScsiLun | where {$_.ExtensionData.OperationalState -eq "off"}).CanonicalName
  }
}

Here (link) is the PowerCLI code by LucD to attach or detach disks.

PowerCLI: Get ESXi host standard and distributed switch uplink statuses

I recently had trouble with ESXi network uplinks, and I combined a script to get all network uplink statues from Standard and Distributed switches connected to each ESXi host. This script will loop through all hosts and display the status of all virtual switch uplinks.

$esxi_hosts = Get-VMHost -State Connected,Maintenance | Sort-Object -Property Name

foreach ($esxi_host in $esxi_hosts) {
    Write-Host "Checking host - $esxi_host" -ForegroundColor Green
    try {
        $esxcli = $esxi_host | Get-EsxCli -V2

        # Get all standard switches
        Write-Host "Getting standard switches"
        $stantard_switches = $esxi_host | Get-VirtualSwitch -Standard

        Write-Host "Checking standard switches"
        foreach ($stantard_switch in $stantard_switches) {
            $nics = ($stantard_switch).Nic | Sort-Object
            foreach ($nic in $nics) {
                $nic_status = $null
                $nic_status = ($esxcli.network.nic.get.Invoke(@{nicname="$nic"})).LinkStatus
                if ($nic_status -ne "Up") {
                    Write-Host "$esxi_host - $switch_name - $nic - $nic_status" -ForegroundColor Red
                }
                else {
                    Write-Host "$esxi_host - $stantard_switch - $nic - $nic_status"
                }
            }
        }

        ### Get all distributed switches
        Write-Host "Checking distributed switches"
        $dist_switches = $esxcli.network.vswitch.dvs.vmware.list.Invoke()
        foreach ($dist_switch in $dist_switches) {
            $switch_name = $dist_switch.Name
            $nics = ($dist_switch).Uplinks | Sort-Object
            foreach ($nic in $nics) {
                $nic_status = $null
                $nic_status = ($esxcli.network.nic.get.Invoke(@{nicname="$nic"})).LinkStatus
                if ($nic_status -ne "Up") {
                    Write-Host "$esxi_host - $switch_name - $nic - $nic_status" -ForegroundColor Red
                }
                else {
                    Write-Host "$esxi_host - $switch_name - $nic - $nic_status"
                }
            }
        }
    }
    catch {
        Write-Host "ERROR - failed to get NIC status info"
    }
}

PowerCLI: Configure default VM compatibility

I decided to raise the default VM compatibility to version 19 on all clusters. Instead, on a cluster level, I decided to configure the compatibility level on the virtual data center level.

# Set data center VM compatibility level
$datacenter = Get-Datacenter
$spec = New-Object Vmware.vim.DatacenterConfigSpec
$spec.DefaultHardwareVersionKey = "vmx-19"
$datacenter.ExtensionData.ReconfigureDatacenter_Task($spec,$true)

For the data center-level config to take effect, I needed to clear the cluster-level settings.

# Clear the cluster level settings
$cluster = Get-Cluster
$spec_cluster = New-Object VMware.Vim.ComputeResourceConfigSpec
$spec_cluster.DefaultHardwareVersionKey = ""
$spec_cluster.MaximumHardwareVersionKey = ""
$cluster.ExtensionData.ReconfigureComputeResource_Task($spec_cluster,$true)

PowerCLI: Find VMs with several tags

I have recently started using more tags, and I often need to find VMs with multiple tags.
I’ve been using this method, which has worked so far.

$tag1 = Get-Tag -Category "Cat1" -Name "Tagname1"
$tag2 = Get-Tag -Category "Cat2" -Name "Tagname2"
$tag3 = Get-Tag -Category "Cat3" -Name "Tagname3"

$vm1 = Get-VM -Tag $tag1
$vm2 = Get-VM -Tag $tag2
$vm3 = Get-VM -Tag $tag3

$vm_result = $vm1 | Where-Object {$vm2 -Contains $_} | Where-Object {$vm3 -Contains $_}

$vm_result

Also check this for another option – https://communities.vmware.com/t5/VMware-PowerCLI-Discussions/Get-VM-List-based-on-Multiple-Tags/td-p/2893873