vQuicky – For the impatient like me

> Too many disk operations on a datastore will lead to storage vmotion time out to occur

> The time out is a consequence of overhead per disk during storage vmotion. Over head involves opening, closing and processing disks

> Default time out is 100 seconds but can be increased by adding a parameter fsr.maxSwitchoverSeconds

> To add the row – power off the virtual machine –>in Options tab –>Advanced general –> Configuration parameters —–>Add row

> If manually editing the .vmx file, remember to remove the virtual machine from inventory first, modify .vmx and then re-add it back


In my lab I was moving many virtual machines around and got this error.

“The migration has exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source.  To avoid this failure, either increase the maximum allowable switchover time or wait until the VM is performing a less intensive workload.”

This happens when a Virtual Machine with many disks is unable to complete Storage vmotion. In my case, I had a high latency FREENAS box hosting iscsi luns and in an attempt to rebuild it, I was moving many machines to local hypervisor disks. Remember that storage vmotion requires time to open, close and process disks in its final copy phase. So storage vmotion working on many disks will timeout because of this over head. So basically, large number of operations occurring on the same datastore will cause it to timeout.

To fix this we increase the fsr.maxSwitchoverSeconds to more than 100 seconds which is the default. To do this you will have to power down the machine –> Edit settings –> Options Tab –> Advanced general –> Configuration parameters –> Add row and add the “fsr.maxSwitchoverSeconds = 200”

Remember if you are manually editing the .vmx file, power off the virtual machine, remove it from the inventory – modify the .vmx file and then re-add it to the inventory.

Please do comment or correct me 🙂

Upcoming Blog List

I have been procrastinating but the following are my upcoming posts.

1. SSL Configuration on host – How to configure custom certs on host.

2. Talk about what NSPAN, ERSPAN are – which are supported in ESXi 5.1 Networking

3. Talk about LACP which is supported in new ESXi 5.1 Networking – Link Aggregation Control Protocol to bind links to push traffic

Atleast  these three before this week – I promise!


The other day my vSphere client started showing logs (in the events tab) as XXX.Snapshot… This was actually on another pc and I thought the hypervisor went FooBar. I was operating a vSphere 4.1 box and using a vSphere client 4.1 obviously.

Turns out it was localization issue. Localization issue means vSphere client’s default language is different than what the vCenter is installed as. That did not still make sense to me that why would that matter in the logs – everything else was in English.

Regardless, the fix was to set -locale en_US to the vSphere client. You do that as follows,
1. Click start–>run
2. Type vSphere client executable path
“c:/……./vpxclient.exe” -locale en_US
3. Click ok

That should do it. Make sure you have the language pack installed on your OS incase you were using something else as a base language.

Let me know if you want to add to this or correct me 🙂

vSphere Web Client Architecture

If you ever want to know how the vsphere webclient works but without doing a developer level deep dive, remember the following picture.

The Inventory service runs as a separate service in vcenter 5 and now in vsphere 5.1 vcenter server, the Inventory service comes as a separate module and can be installed on a completely separate server. Now the Inventory service, as mentioned above, obtains optimized data from the vcenter. The data is optimized as in what ever the user is seeing in the web browser, only that data is requested.

This is fed to the application server that runs java/virgo/spring. The flex based vSphere web client gets its info from the Application server.

Hope this was a little helpful. Feel free to discuss or correct me 🙂

ESXi 5.1 Supports SR-IOV. But What is SR-IOV?

I was going through all the exciting things that ESXi 5.1 supports. But one thing caught my eye and it was something I had never heard of.

vQuicky – For the Impatient Like me 🙂

> SR-IOV stands for Single-root I/O Vritualization and is now supported in ESXi 5.1

> SR-IOV is different from VMdirect-path in that it allows multiple vms to share the same nic and have higher throughput as they bypass the hypervisor layer

> SR-IOV supported PCIe card allows one PCIe card to be presented to many virtual machines directly without going through the hypervisor layer

> SR-IOV runs on the concept of Physical functions and virtual functions. PFs have configuration capability to the PCIe device while VFs do not. VFs are attached to the virtual machines and are of the same device type as the PF/PCIe device.

> PCI SIG  maintain the SR-IOV specification and say that one SR-IOV supported card can have as many as 256 VFs in theory.

> SR-IOV needs to be supported in BIOS, OS and Hypervisor for it to work correctly.

> SR-IOV can cause issues with physical switches as physical switches do not know about the VFs. Inters-witching is another functionality that I believe is available that allows direct traffic flow between VFs.

> You cannot have a different type of VF from that of a PF or the PCIe device type.

> If you have a SR-IOV support HBA card, then you will need NPIV, N_PORT_ids to manage WWNs on the virtual HBAs.

> MR-IOV is multiple systems using VFs and sharing one PCIe device.


So what is single-root i/o virtualization or sr-iov in short. Here goes.

Single-root I/O virtualization is a technology or infact a specification that allows you to present a single physical PCIe device to appear as multiple PCIe devices. For instance if you had a single network PCIe card, with SR-IOV you can now present this PCIe card to the virtual machines and these virtual machines will look at it as if its their own dedicated PCIe physical card. You will now have the ability to present one PCIe card to multiple virtual machine instances. This is different than VMdirectpath. SR-IOV allows hypervisor bypass by allowing a virtual machine to attach a virtual function to itself and share the same network card. More about virtual functions below.

SR-IOV works by introducing the concept of Physical Functions and Virtual Functions, PF and VF in short respectively. Physical functions are full blown PCIe functions while virtual functions are light weight and are associated with a virtual machine. A virtual function is assigned to a virtual machine, the traffic flows directly to it bypassing the virtual layer allowing higher throughput. Basically, PFs have full configuration capability to the PCIe card, aka, they are full featured PCIe functions and can do configuration changes to the card. VFs lack the configuration capability and can only move data in and out.

SR-IOV requires support in BIOS and also in the operating system and/or the hypervisor as well. Since VFs can’t be treated as full PCIe devices because of their lack in their configuration capability, the OS need extra drivers to work with them.

The PCI SIG SR-IOV (these guys maintain this specification) says that one instance can have as many as 256 VFs. So a quad port SR-IOV PCIe card will have 1024 VFs in theory. Remember that VF replies on the physcial hardware of the card itself so practically the number of VFs may be lower. For SR-IOV fibre channel HBAs, you will have logical HBAs sharing a physical HBA and you will need NPIV enabled so you can manage the WWNs and N_Port_IDS on a single HBA.

Some things you have to be careful about is that switches will not be aware of the VF ports and can cause some issues. I read that the SR-IOV allows some functionality where VF Switching is possible, that means you can send traffic between VFs and not have to use a physical switch. You also cannot have a VF of one type other than that of the PF.

I picked up some information for comments from Scott Lowe’s blog. The servers below support SR-IOV. Apparently there is also MR-IOV (Multi Root – i/o virtualization) where multiple systems will share PCIe VFs. I heard it was terribly complicated but Googling did not result in too much info.

Dell: PowerEdge 11G Gen II Servers: T410, R410, R510, R610, T610, R710, T710 and R910.
Further information is available at

Fujitsu: PRIMERGY RX900 S2 using BIOS version 01.05.

HP: Proliant DL580 G7 and Proliant DL585 G7. Further information is available at

Comment if you liked it or if I said something wrong 🙂

Unable to Connect to the vCenter Inventory Service

My vCenter Web client popped up an error that said “Unable to connect to the vcenter inventory service : https://ip:10443


Remember, in vCenter 5.0, the inventory service is running as a separate service and not as part of the vcenter service. The inventory service works for the web client. It manages the web client inventory objects and property queries that the web client requests when a user is browsing it. The web client runs efficiently by only requesting queries that the user is seeing on their screen – this enhances user experience and navigation.

In vCenter 5.1 – the inventory service is a completely separate component than the vcenter and can be installed on a separate box itself.

In my case, as expected, the service was off and I started the service on the vcenter server. The issue was resolved.

VM Replication in ESXi 5.1

rjapproves QuickY (For the impatient like me!)

  • VMware introduced virtual machine replication in today’s ESXi 5.1 announcement.
  • Virtual machine replication replicates your primary virtual machine to another datacenter so you can quickly bring up your virtual machines in a disaster event at the primary site.
  • A virtual machine is copied to the other site, a full copy followed by changed block copies are done.
  • Intelligent copy where application and/or database consistency is maintained using Microsoft Volume shadow copy service(VSS)
  • Replication is free product and is added as a feature
  • Replication runs off a agent that comes with each hypervisor install and a vSphere Replication appliance that runs on each vcenter at per vcenter basis. As of now, its maximum of 10 appliances per vcenter supporting 500 virtual machine replications per site.
  • Replication can be done on a per virtual machine and per hard disk basis.
  • The target virtual machine will not power on if the primary (source) virtual machine is pingable by vcenter server and/or is still powered on.
  • Recovery vm once powered on will have to be manually connected to the network via the vcenter web console (one click action).


This is exciting, VMware finally takes up Microsoft’s Hyper-V challenge and offers replication as a free product add on for all of us to enjoy.

So what is vmware replication really all about? Think of vmware replication as basically having a clone of your virtual machine on another hypervisor which may or may not be on the same site. Ideally, the replication will occur from one physical site to another. The catch is that this clone is constantly being updated – aka – any changes in the source site are transferred to the target site depending on the interval for replication you set.

The hypervisor inherently has a agent and there is a vmware replication appliance per vcenter whose job is to receive replication data and also keep track of the statuses. A vCenter can have upto 10 maximum vmware replication appliances. I am not clear on the maximum virtual machines that can be replicated but it seems like there is a cap on 500 virtual machines per vCenter.

The way it works, in the web based vcenter portal, you simply enable replication by a click and choose a virtual machine to replicate to another datacenter hypervisor. You get to choose how often the replication is to run, basically a RPO for your virtual machine. VMware allows a low RPO of 15 mins to a maximum of 24 hours. You can also set the destination virtual machine to have its disks as thin as opposed to the thick disks at the primary site. This allows you to save storage space on the target site.

Once that is set, you are all done, the initial replication will be a complete copy of your virtual machine over the wire. Now if you have a huge virtual machine, you can choose to pre-seed it to cut down on replication time. You can either copy it/clone it to a usb storage or FTP and then pre-seed that at the destination. Once Pre-seed is done, only changed blocks are copied. Change block tracking, i believe, is automatically enabled and only blocks that are changed are copied over. Remember the VA Agent is responsible to send over changed blocks and these are received by the VMware replication appliance on the other end.


Some cool things you can do is specifically pick which disks in a virtual machine you want to replicate and which you choose not to. For instance you can have a designated disk as swap disk for a virtual machine and you may choose not to replicate that disk.

To failover the virtual machine you simply have to initiate it from the vcenter web client, however remember, failover will not work or will not be allowed if your primary vm is still powered on and/or vcenter can ping it. When the target virtual machine comes up, it will disconnected from networks. The idea is for an administrator to look at the target and then to connect it to the network and not to have them blindly power on. This is to prevent accidental power ons that may result in network conflicts if any could happen.

I am quite sure that VMware is using the Virtual Disk Development Kit(VDDK) to get this all to work. The host agents that come loaded with the ESXi 5.1 install on the box use the VDDK to attach/detach disks and also ship data over wire. There are no security issues as of today, and replication is pretty secure.

I don’t have specific details yet but I will add more as we play along.

Do comment if you have any more information or need to correct/add info.

VMware Announces no more vRAM Entitlements!

Lets not even debate the whole structure about the vRAM pricing but VMware at VMWorld just announced (a minute back) that there are no more vRAM license entitlements. No vRAM, no per vm licensing model. Only per socket model which makes a lot of things easier. This was received with a lot of applause!

What you need to know about Jumbo Frames

rjapproves QuickY (For the impatient like me!)

> Jumbo frames pack more data in each network frame, so a frame with more payload, less cpu overhead, more data throughput
> Anything more than 1500 MTU is a jumbo frame and the maximum supported is 9000MTU.
> Can use for inter-vm traffic, vkernel traffic(iscsi), vmotion traffic
> Need to be enabled for the entire stack end-to-end to reap benefits. L2 switches not configured will drop frames, L3 will fragment them if not configured
> Enable Jumbo frames  – esxi 4 – esxcfg-vswitch -m 9000 vswitch0 and for vkernel – esxcfg-vmknic -m 9000 vmkernelname
> Enable Jumbo frames – ESXi 5 – esxcli network vswitch standard set -m 9000 -v vswitch0 and for vkernel – esxcli network ip interface set -m 9000 -i vmk0
> For DVS, use the client and edit settings–Advanced
> List all network info – esxi 4 – esxcfg-vswitch -l or esxcfg-vmknic -l
> List all network info – ESXi 5 – esxcli network interface list

In Depth

I have been into vmware for sometime but never really got to know what Jumbo frames were all about. So here goes. Jumbo Frames in simple terms allow you to send more data packed in frames. This is beneficial for higher throughput and also for lower cpu overhead. Remember, the more the frames, the more cpu resources you are burning up to send all the frames. This is basically the same through out the stack and by stack I mean your hypervisor-switch-router-… etc.

So a Jumbo frame is a frame where the frame has a payload of more than 1500 MTU(Maximum Transmission Unit). Now typically Jumbo frames max out at 9000MTU. Remember that this roughly translates to 1500 bytes for standard and 9000 bytes for a jumbo frame. Actually the real size for a 1500MTU is actually 1522 bytes of data where the 22 bytes include source and destination mac addresses(12 bytes total) + optional 802.1q vlan tag (4bytes) + ethertype(2bytes) and CRC32 error correction trailer (4 bytes).  A blog i read about while researching claimed to have seen some cisco switches that could go up to 9216 bytes.

As mentioned above now by enabling a jumbo frame, you can pack as much data in one frame, thus the hypervisor has to send less frames to send the same amount of data, improving speeds and also lower cpu overhead.  You can use jumbo frames for traffic between vms, vmotion or even iscsi which is supported in ESX(i) 4 and ESXi 5.

The only catch with jumbo frames is that it is a end to end setup. That means from your hypervisor to your switch to the other end should support jumbo frames. Some routers, when they get a jumbo frame, fragment it and assemble them on the other end and it only adds overhead which is just bad design. Also if the switches are not configured with Jumbo frames, they will just drop it so you will break connectivity!

How to enable Jumbo Frames?

You have to enable jumbo frames on the vswitch or the dvs that you may be using. You can either use the client to do that or from the service console,
For ESX(i) 4 – esxcfg-vswitch -m 9000 vswitch0    —> Remember this command sets the MTU on the nics.
For ESXi 5 – esxcli network vswitch standard set -m 9000 -v vSwitch0
Remember the above commands will set all nics associated to that vSwitch to MTU 9000.
You can also enable jumbo frames for vKernel interfaces as follows.
For ESX(i) 4 – esxcfg-vmknic -m 9000 portgroup_name  –> I read that you actually need to delete and recreate the vmk in ESX(i) 4 but did not test it.
For ESXi 5 – esxcli network ip interface set -m 9000 -i vmkernel_interface  —> vmkernel_interface as in vmk0 or vmk1

To list all switch info 
For ESX(i) 4 – esxcfg-vswitch -l or esxcfg-vmknic -l
For ESXi 5 – esxcli network interface list

Comment to discuss more or correct me 🙂

Resources – 

Running Usbarbitrator Start Issue

So I am not sure why but my home lab breaks almost all the time.

This time I got this error running ESXi 5,
“Running usbarbitrator start”
My box was stuck for a long time and then booted. My box does run off of a usb stick but that shouldn’t have been an issue as it worked just fine until now.
Googling –
So the USB Arbitrator service allows for usb passthrough. Now if this service is running, it will not allow you to access usb devices in the service console. Now I am not sure why this is causing an issue for my box. Vkernel log shows,
“usb passthrough enabled; all eligible devcices will be unclaimed by kernel drivers…”
Now I logged in and went directly to the console and disabled the usbarbitrator service. I couldn’t login to the client and it kept kicking me away.
“chkconfig usbarbitrator off”
Official vmware doc says you need to reboot your server once you turn the usbarbitrator service off. KB Article here