Category Archives: Virtualization

Everything about Virtualization



> After deploying vcenter appliance, while using the web client you see a message Inventory service failed to connect at port 10433.

> Rebooting the vcenter server wont fix it.

> You will have to re-run the vCenter Setup wizard with defaults.

> Once done, re- add the Active directory, re-add vcenter permissions and you should be good.

> vcenter hosts and other info should remain intact.


I did not really deep dive in this issue and vmware isn’t clear in any of its KB articles either. But when you see the issue mentioned above in the vquicky, simply re-run the vcenter appliance setup wizard with defaults in place. Remember to stop the vcenter and inventory services before you run it. Once done, go ahead and add Active directory credentials and then reboot the vcenter.

Once up, login to the web client using root/vmware – and assign the vcenter permissions to the domain user group in case you have one. (you should its good practice).

Once thats done you should be good to go!

Do comment/post 🙂


vQuicky –

> DRS has a bug that shows multiple vmotions or svmotions for a single vm

> The bug reports operation not allowed in current state.

> The DRS operation, however succeeds.

> The bug is harmless and can be ignored according to vmware.

inDepth  – 

So, in the latest KB articles come up a DRS bug in 5.1 where you will see multiple vmotions or svmotions for a single vm. The alerts end up throwing an error saying – “The operation is not allowed in the current state.”

The kicker is that the vmotion or svmotion that was being performed completes without any issue.

You will also see some logs in the vpxd log file as follows –


[07208 error ‘drmLogger’ opID=task-internal-1-2cbc1369-5e-72-b5] [DrmExecute::ExecuteVMotion] VM (/vpx/vm/#2/) is already in VMotion, so not executing another vmotion
> 2013-01-16T15:55:32.018+01:00 [07208 info ‘Default’ opID=task-internal-1-2cbc1369-5e-72-b5] [VpxLRO] — ERROR task-13275 — vm-2 — Drm.ExecuteVMotionLRO: vim.fault.InvalidState:

> –> Result:
> –> (vim.fault.InvalidState) {
> –> dynamicType = <unset>,
> –> faultCause = (vmodl.MethodFault) null,
> –> msg = “”,
> –> }
> –> Args:
> –>


The vmware kb article says that this is a bug that can be safely ignored. Apparently they are working on a fix but watch out for your monitoring systems ringing the alarm bells 🙂

You can find the kb article here.


vQuicky – For the impatient like me! 

> VMware bundles vSphere with Operations manager – calls it vSphere with Operations Management

> Licensed per cpu and not per vm.

> Bundled operations management is standard – with custom dashboard

> ROI is via denser hypervisors due to the analytics engine in vcops that helps in right sizing vms.

> Need extra license for charge back manager.

> VCOPS for enterprise is licensed per vm as usual.


News is that on Feb 12th, VMware released vSphere with operations management bundled software. This is more targeted towards small business customers and also towards having more faster adoption of operations manager.

Licensing will be based on the number of cores as opposed to per vm. But this is the licensing only when its bought bundled. You will not get vmware charge back and that would be a separate product that you will have to shell out money for. To justify your licensing costs by buying the bundle – vmware says that you can use the operations manager to actually properly size the vms to their work loads and be able to have more denser hypervisors allowing for cap ex savings. This surely is a true statement because vcops (as its known) helps in consolidating a lot of your vms and sizing them properly and reclaiming cpu/disk and memory. Right sizing environments is of ample importance as your vm foot print grows.

The bundle has vcops standard and below is the pricing. It also allows you to have custom dashboards!

VMware Online Store - Product Information - Google Chrome_2013-02-14_23-10-29

It makes sense for vmware to do this. vCloud suite – vcops is designed and built for large environments however this bundle helps small to medium businesses to take advantage of vcops and be able to have more detailed insight on their datacenters.

What do you think about this bundle?



VMware MyLearn Free E-Learning Course

Just saw this and thought I should let you know – VMware is offering free e-learning course via the mylearn. The course is for virtualizing Microsoft SQL server on vSphere 5.

One really cool part of this course is that it discusses disaster recovery scenarios with and without the use of Mirroring.

You can find the training here.



If Everytime I had a nickel for when my home lab broke – I would be filthy rich!

As always my vcenter 5.1 broke. Started throwing unable to connect to vcenter/sdk error. When I logged in as the SSO admin I did not see vcenter registered. For some reason it disappeared.

I am still yet to fix the issue but came across this KB which helps you with re-registering vcenter components to each other. It is by no way intuitive and is all command line.

This will be very handy now that vcenter has all its services separate! You have to make sure all the moving parts, SSO, Inventory Service, web service and vcenter are connected and aware of each other – not in that particular order anyway.

Here is the KB

Update – I only have vague update on my issue with vcenter. I gave up and went ahead to reinstall just the vcenter vm – found out it kept giving me inventory server error. vCenter talks to the inventory server on https://inventory-server-url:10443 however it kept failing.

I re-installed the inventory server and then it worked fine, so the above issue was possibly due to something messy in the inventory server.

More as I know it.


So twitter told me that redhat released its enterprise virtualization 3.1 today that is based on the Kernel-based virtual machine or KVM in short. It has some stunning capabilities and features. From what I read – this is the follow up from the 3.0 version that was out in January 2012.

There are lot of new features added all across the board from supported storage, processors to networking. Some interesting features are creation of snapshots is now supported, port mirroring is now possible, hot plugging and unplugging of disks is supported, direct disk block access is also now possible. Red Hat also took a page from vmware here and is now providing initial support for storage live migration.

For the cpu, Red Hat virtualization 3.1 now supprots Intel Core i3, i5, and i7 CPUs — code named “Sandy Bridge”, and AMD 15h, or Opteron G4, CPUs — code named “Bulldozer”.

Below is a nice concise from site.

  • Support for up to 160 logical CPUs per VM
  • Support for up to 2 TB of memory per VM
  • Support for the latest x86 processors
  • Improved cross-platform web administration portal
  • Updated reporting dashboard
  • New network capabilities, including hot plugging and unplugging of vNICs, bridge-less network support, port mirroring, configurable MTU.
  • Enhanced disk storage
  • Technology preview of Storage Live Migration
  • Enhanced per user portal with resource quota functionality
  • New virtual desktop auto start policy
  • Improved WAN optimizations
  • Improved Virtual Desktop Client
  • Localized Administration portal including English, French, Spanish, Simplified Chinese and Japanese
  • Integration with Red Hat Storage (containing the GlusterFS technology)
  • Linux Command Line Interface using REST API
  • Python SDK using the REST API

You can also download the release here.


Ever wanted to be able to graphically display the esxtop statistics but always had to work with tools like performance monitor that are awesome but still a little intimidating to work with?

I had to capture some analytics from esxtop on how some hypervisors were doing on disk usage, typical commands, reads and writes. Now I did need to present the data and as always yellowbricks was helpful.

Here goes,

Run esxtop and then use the “f” key to filter out the data you may not need. Its really up to you. If you don’t filter, the end file will just be huge and if you have space to spare you should be fine. Once done, you can actually save this to a new or a default config file by the “W” key.

Once saved,

$> esxtop -b -d 2 -n 100 > esxtop.csv

Here above -b says batch mode and -d is the number of seconds to capture data and -n is number of iterations. So here this will run for 200 seconds and dump the data in esxtop.csv in the present working directory.

I also learnt that using the -a switch above will capture all data but be careful about the time of capture as it can really blow up.

Now that we have the esxtop.csv, a great tool to use is the esxplot from VMware Labs.

Simply go here and download the tool – it runs on both windows and linux. Simply run the executable in windows and then go to File –> Import –> Dataset. You can then import the csv and then the hypervisor will show up. Double click the hypervisor and choose what info that needs to be plotted using the shift key for multiple graphs.




vQuicky – For the impatient like me..

>Vmware introduces multi-hypervisor support in vCenter

> This allows one to manage multiple hypervisors using vCenter

> Currently only Hyper-V 2008/2008 R2 is supported with hope of support in the future for KVM, Citrix XEN server and even OpenStack.

> The manager allows you to add, remove connect, disconnect and view host configuration.

> You can even provision new virtual machines on the hosts and edit the vm settings.

> It is fully supported and available for download – link below.

inDepth – 

Vmware on 19th of this month, released the vCenter Multi-hypervisor manager. As you can guess – with this one can use vCenter to manage multiple hypervisors from other vendors such as HyperV, KVM, Citrix XEN server and who knows may be even Openstack.

Currently it only supports HyperV but this is a 1.0 version and is a start in that direction. It only supports Hyper-V severs 2008 and Hyper-V servers 2008 R2. So I haven’t been able to download and try it. But if you want to just click here and scroll all the way down and download the server and the client.

From the release notes I picked up some of the features that it will be capable of.

>Management of non-esxi hosts with the ability to provision vms.

> Edit vm settings and also install guest OS’es on the vms.

> It can also integrate vcenter’s authorization mechanism across ESXi and non-ESXi hosts.

For now this seems pretty basic of what vmware can do and I am sure Openstack folks will not find it cool considering Openstack can manage other hypervisors. However this is a great start for vmware. This will allow any firm to mix multiple hypervisors and be able to offer customers the same level of service while having a single pane of glass management layer.

Also note that this is not available via the web client and is supported online the the vcenter client 5.1

Below is what the release notes says about its features –

  • Third-party host management including add, remove, connect, disconnect and view the host configuration.
  • Ability to provision virtual machines on third-party hosts.
  • Ability to edit virtual machine settings.
  • Integrated vCenter Server authorization mechanism across ESX and third-party hosts inventories for privileges, roles, and users.
  • Automatic discovery of pre-existing third-party virtual machines
  • Ability to perform power operations with hosts and virtual machines.
  • Ability to connect and disconnect DVD, CD-ROM, and floppy drives and images to install operating systems.

Security is taken care of by HTTPS communications.


vQuicky – For the impatient like me

> Double cloud proxy is the fastest reverse proxy to work with

> It is simple to run and does not need to be installed or anything.

> It is simple to implement and capture api data


So we were running into some issues with an application using vcenter via the vcenter APIs. Now we were unsure about the issue and needed a reverse proxy that can spit out all the api calls going back and forth from the application to the vcenter API.

Now before I say more – having vcenter set to trivial logging will give you all the info needed but thats a lot of lines to go through in the logs and the logs can grow beyond a decent size very easily. You can also use onyx, but I had no experience with it.

Doublecloud to the rescue. Doublecloud proxy is a java based application written by Steve Jin and it can’t get any better than this. This not only captures all the api calls in soap but also can spit out java code – how cool is that. It also has python code in the works – basically spitting out python code for a call.

Its super easy to run this. Simple download the java program and make sure you have java installed on your system. Once done just doubleclick the proxy jar file. You will a gui as follows.

Click play, and it shows the following dialog box. Fill in the dns name or ip for the vcenter and leave the port mappings as default.

Once you are done, use your vmware client or what ever application that wants to connect to vcenter and use the proxy’s ip and your regular vcenter id and password to connect to it. For instance if doublecloud proxy is run on your computer then the vmware client should connect to localhost on port 1545 with your regular login id and password.

Below is what you will see in the doubleproxy cloud gui while vmware client is connected the vcenter.

Once you have all the info here – you can save it to a file as well.

More info –


vQuicky – For the impatient like me

> No webclient support for plugin installs especially vmware update manager.

> Need to use vsphere client to update your hosts

> Don’t forget to check if your hosts are able to resolve to the vum/vcenter server using the dns or else scanning of hosts will fail


Over the weekend I spent time rebuilding my lab. My lab has servers running on usb sticks @ ESXi 5.0 . I needed them to be on 5.1 but really didn’t want to rekick these boxes.

The only right way to do it was to rebuild my vcenter part of the lab from 5.0 to 5.1. Now the way to do that was to blow away what I have right now and do a proper upgrade because with 5.1 – the Single Sign On has been introduced. Now I can cheat by doing a vcenter simple install – basically an all in one vcenter instance but I wanted to mimic a typical production environment.

So I rolled out 4 vms – a webclient, a SSO, a Inventory service and a vCenter server which also had VUM installed.

The order of installation is SSO first, then comes the inventory service followed by the webclient and then the vcenter server. Once the install is done, you login to the webclient using the [email protected] default id and add the domain in the SSO configuration part. Once done you can login as a domain user and mess around with your vcenter.  Also don’t forget, if you look closely by default, [email protected] has admin access to the vcenter. In a production environment, you may not want this. Again its not a big deal but really, you typically don’t want any stale accounts be granted the admin role when not needed.

I had to login to the vcenter client and get the plugin installed and start to patch my two hosts. Now patch failed at the initial scan. I forgot to update the hosts with the right dns entries so they can lookup the vum server to pull the downloads. After a quick dns update all worked well.

Hopefully I will have a demo going for you in a video.

Feel free to comment or correct me 🙂