Had no client access to the host nor to the vcenter. Needed vcenter vm which was already built added to the host’s inventory to get the show on the road.

SSH into the host

$ vim-cmd solo/registervm /vmfs/volumes/fullvmpath/vmname.vmx

That’s it!


vQuicky – For the impatient like me 🙂

> VMware communications are all encrypted over SSL

> VMware uses self generated ssl certificates to encrypt session information.

> VMware uses standard X.509 Version 3 certificates which conform to Privacy Enhanced Mail and the key used is a RSA that ranges from 412 to 4096 bits with a recommendation of 2048 bits

> Download Openssl-Win32 (link below) and install it on a windows box to create the certificate signing request.

> Preconfigure openssl.cfg (below) and create your rui.csr and rui.key from the config file.

openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

> Either ship the certificate signing request to a third party SSL trust or create a rui.crt certificate using openssl commands

openssl x509 -req -days 365 -in rui.csr -signkey rui.key -out rui.crt

> Once done, copy rui.crt and rui.key to /etc/vmware/ssl

> If https error pops up, then the certificate has a passphrase – quick way to restore host is recreate vmware signed certificates by using the following command to restore vclient connectivity.



I rebuilt my lab, however I am tired of seeing the above. So lets learn how to add SSL certs to our hosts and make them more secure. We all know that vSphere encrypts session information using digital certs. In my case for my lab, default certificates are fine by if you were working for a huge organization then standard SSL certificates may be a requirement. ESXi creates standard certificates by default which are not signed by a CA (certificate authority) and can also be vulnerable to man in the middle attacks.

VMware uses standard X.509 version 3 – also known as X.509v3 certificates to encrypt session information over SSL. This also applies to any communication between vCenter and Esxi host as well. Remember when you want to replace the default certificates, the new ones must conform to Privacy Enhanced Mail or PEM format. Privacy Enhanced Mail stores data in a Base-64 encoded distinguised encoding rules -DER format. As always the key used to sign certificates must be a standard RSA Key with an encryption length that ranges from 412 to 4096 bits. The whitepaper recommends a length of 2048 bits.

How to do it?

Before we dig deeper, remember if an ESXi 5.0 host is part of a HA Cluster, changing the SSL cert will break HA. To avoid this, make sure you are running vCenter 5.0 U1 or later. Now that Update 1 is out, you might as well upgrade to that before doing anything.

You need to download OpenSSL to create a self signed certificate. Now I am using Windows and downloaded the Win32 OpenSSL from slproweb. You can download it here. Once done simply install it.

Now we can either do it manually by answering all prompts and then removing the passphrase encrypt key or by manually editing the openssl.cfg file that you find in the directory location below.

In the file just replace the bold parts. I picked this up from the kb article.

# vSphere OpenSSL example configuration file start.
HOME = .
oid_section = new_oids

[ new_oids ]

[ ca ]
default_ca = CA_default # The default ca section

[ CA_default ]

dir = ./demoCA # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
new_certs_dir = $dir/newcerts # default place for new certs.
certificate = $dir/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/cakey.pem# The private key
RANDFILE = $dir/private/.rand # private random number file
x509_extensions = usr_cert # The extentions to add to the cert
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
default_days = 5475 # how long to certify for
default_crl_days = 30 # how long before next CRL
default_md = sha512 # which md to use.
preserve = no # keep passed DN ordering
policy = policy_match

[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 2048
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extentions to add to the self signed cert
input_password = testpassword
output_password = testpassword
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ] # change these settings for your environment
countryName = US
stateOrProvinceName = New York
localityName = New York
0.organizationName = Customer Name
organizationalUnitName = IT
commonName =
emailAddress = [email protected]

[ req_attributes ]

[ usr_cert ]

basicConstraints =CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid,issuer

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:, DNS:, DNS: vc41 #examples only

[ v3_ca ]
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid:always,issuer:always
basicConstraints = CA:true

[ crl_ext ]
authorityKeyIdentifier =keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints =CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid,issuer:always
proxyCertInfo =critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# vSphere OpenSSL example configuration file end.

Once saved, open up the Openssl directory which is typically located at C:\OpenSSL-Win32\bin

We now want to get the certificate signing request and the key created from the config file above. The command is

>openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

I initially did not pre configure the openssl.cfg. You can do it that way as well but remember to remove the passphrase from the key as you don’t want to enter the passphrase for the http daemon every time.  If you were wanting to use a commercial signed certs then just send them this rui.csr and they will send in the certificate. To create your own self-signed certificate run the following command.

>openssl x509 -req -days 365 -in rui.csr -signkey rui.key -out rui.crt

Once we have the above, you will have three files in the /bin directory which are rui.key, rui.crt and rui.csr. We need the rui.crt and the rui.key files.

Use WINSCP to copy these two files to the host. In the host navigate to /etc/vmware/ssl and copy the files.  Once copied, restart the management agents. Alternatively you can reboot as well. If something screws up no worries as you can restore the vmware certificates by recreating them. You can ssh to the host and recreate the vmware signed certs by the following command.


>/sbin/ restart

If all went well, get to the https://ipaddress and you should see your self signed cert installed.

Now you have your certificate installed!

Please comment or correct me if you find some errors 🙂


vQuicky – For the impatient like me

> Too many disk operations on a datastore will lead to storage vmotion time out to occur

> The time out is a consequence of overhead per disk during storage vmotion. Over head involves opening, closing and processing disks

> Default time out is 100 seconds but can be increased by adding a parameter fsr.maxSwitchoverSeconds

> To add the row – power off the virtual machine –>in Options tab –>Advanced general –> Configuration parameters —–>Add row

> If manually editing the .vmx file, remember to remove the virtual machine from inventory first, modify .vmx and then re-add it back


In my lab I was moving many virtual machines around and got this error.

“The migration has exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source.  To avoid this failure, either increase the maximum allowable switchover time or wait until the VM is performing a less intensive workload.”

This happens when a Virtual Machine with many disks is unable to complete Storage vmotion. In my case, I had a high latency FREENAS box hosting iscsi luns and in an attempt to rebuild it, I was moving many machines to local hypervisor disks. Remember that storage vmotion requires time to open, close and process disks in its final copy phase. So storage vmotion working on many disks will timeout because of this over head. So basically, large number of operations occurring on the same datastore will cause it to timeout.

To fix this we increase the fsr.maxSwitchoverSeconds to more than 100 seconds which is the default. To do this you will have to power down the machine –> Edit settings –> Options Tab –> Advanced general –> Configuration parameters –> Add row and add the “fsr.maxSwitchoverSeconds = 200”

Remember if you are manually editing the .vmx file, power off the virtual machine, remove it from the inventory – modify the .vmx file and then re-add it to the inventory.

Please do comment or correct me 🙂

Upcoming Blog List

I have been procrastinating but the following are my upcoming posts.

1. SSL Configuration on host – How to configure custom certs on host.

2. Talk about what NSPAN, ERSPAN are – which are supported in ESXi 5.1 Networking

3. Talk about LACP which is supported in new ESXi 5.1 Networking – Link Aggregation Control Protocol to bind links to push traffic

Atleast  these three before this week – I promise!


The other day my vSphere client started showing logs (in the events tab) as XXX.Snapshot… This was actually on another pc and I thought the hypervisor went FooBar. I was operating a vSphere 4.1 box and using a vSphere client 4.1 obviously.

Turns out it was localization issue. Localization issue means vSphere client’s default language is different than what the vCenter is installed as. That did not still make sense to me that why would that matter in the logs – everything else was in English.

Regardless, the fix was to set -locale en_US to the vSphere client. You do that as follows,
1. Click start–>run
2. Type vSphere client executable path
“c:/……./vpxclient.exe” -locale en_US
3. Click ok

That should do it. Make sure you have the language pack installed on your OS incase you were using something else as a base language.

Let me know if you want to add to this or correct me 🙂

vSphere Web Client Architecture

If you ever want to know how the vsphere webclient works but without doing a developer level deep dive, remember the following picture.

The Inventory service runs as a separate service in vcenter 5 and now in vsphere 5.1 vcenter server, the Inventory service comes as a separate module and can be installed on a completely separate server. Now the Inventory service, as mentioned above, obtains optimized data from the vcenter. The data is optimized as in what ever the user is seeing in the web browser, only that data is requested.

This is fed to the application server that runs java/virgo/spring. The flex based vSphere web client gets its info from the Application server.

Hope this was a little helpful. Feel free to discuss or correct me 🙂

ESXi 5.1 Supports SR-IOV. But What is SR-IOV?

I was going through all the exciting things that ESXi 5.1 supports. But one thing caught my eye and it was something I had never heard of.

vQuicky – For the Impatient Like me 🙂

> SR-IOV stands for Single-root I/O Vritualization and is now supported in ESXi 5.1

> SR-IOV is different from VMdirect-path in that it allows multiple vms to share the same nic and have higher throughput as they bypass the hypervisor layer

> SR-IOV supported PCIe card allows one PCIe card to be presented to many virtual machines directly without going through the hypervisor layer

> SR-IOV runs on the concept of Physical functions and virtual functions. PFs have configuration capability to the PCIe device while VFs do not. VFs are attached to the virtual machines and are of the same device type as the PF/PCIe device.

> PCI SIG  maintain the SR-IOV specification and say that one SR-IOV supported card can have as many as 256 VFs in theory.

> SR-IOV needs to be supported in BIOS, OS and Hypervisor for it to work correctly.

> SR-IOV can cause issues with physical switches as physical switches do not know about the VFs. Inters-witching is another functionality that I believe is available that allows direct traffic flow between VFs.

> You cannot have a different type of VF from that of a PF or the PCIe device type.

> If you have a SR-IOV support HBA card, then you will need NPIV, N_PORT_ids to manage WWNs on the virtual HBAs.

> MR-IOV is multiple systems using VFs and sharing one PCIe device.


So what is single-root i/o virtualization or sr-iov in short. Here goes.

Single-root I/O virtualization is a technology or infact a specification that allows you to present a single physical PCIe device to appear as multiple PCIe devices. For instance if you had a single network PCIe card, with SR-IOV you can now present this PCIe card to the virtual machines and these virtual machines will look at it as if its their own dedicated PCIe physical card. You will now have the ability to present one PCIe card to multiple virtual machine instances. This is different than VMdirectpath. SR-IOV allows hypervisor bypass by allowing a virtual machine to attach a virtual function to itself and share the same network card. More about virtual functions below.

SR-IOV works by introducing the concept of Physical Functions and Virtual Functions, PF and VF in short respectively. Physical functions are full blown PCIe functions while virtual functions are light weight and are associated with a virtual machine. A virtual function is assigned to a virtual machine, the traffic flows directly to it bypassing the virtual layer allowing higher throughput. Basically, PFs have full configuration capability to the PCIe card, aka, they are full featured PCIe functions and can do configuration changes to the card. VFs lack the configuration capability and can only move data in and out.

SR-IOV requires support in BIOS and also in the operating system and/or the hypervisor as well. Since VFs can’t be treated as full PCIe devices because of their lack in their configuration capability, the OS need extra drivers to work with them.

The PCI SIG SR-IOV (these guys maintain this specification) says that one instance can have as many as 256 VFs. So a quad port SR-IOV PCIe card will have 1024 VFs in theory. Remember that VF replies on the physcial hardware of the card itself so practically the number of VFs may be lower. For SR-IOV fibre channel HBAs, you will have logical HBAs sharing a physical HBA and you will need NPIV enabled so you can manage the WWNs and N_Port_IDS on a single HBA.

Some things you have to be careful about is that switches will not be aware of the VF ports and can cause some issues. I read that the SR-IOV allows some functionality where VF Switching is possible, that means you can send traffic between VFs and not have to use a physical switch. You also cannot have a VF of one type other than that of the PF.

I picked up some information for comments from Scott Lowe’s blog. The servers below support SR-IOV. Apparently there is also MR-IOV (Multi Root – i/o virtualization) where multiple systems will share PCIe VFs. I heard it was terribly complicated but Googling did not result in too much info.

Dell: PowerEdge 11G Gen II Servers: T410, R410, R510, R610, T610, R710, T710 and R910.
Further information is available at

Fujitsu: PRIMERGY RX900 S2 using BIOS version 01.05.

HP: Proliant DL580 G7 and Proliant DL585 G7. Further information is available at

Comment if you liked it or if I said something wrong 🙂

Unable to Connect to the vCenter Inventory Service

My vCenter Web client popped up an error that said “Unable to connect to the vcenter inventory service : https://ip:10443


Remember, in vCenter 5.0, the inventory service is running as a separate service and not as part of the vcenter service. The inventory service works for the web client. It manages the web client inventory objects and property queries that the web client requests when a user is browsing it. The web client runs efficiently by only requesting queries that the user is seeing on their screen – this enhances user experience and navigation.

In vCenter 5.1 – the inventory service is a completely separate component than the vcenter and can be installed on a separate box itself.

In my case, as expected, the service was off and I started the service on the vcenter server. The issue was resolved.

VM Replication in ESXi 5.1

rjapproves QuickY (For the impatient like me!)

  • VMware introduced virtual machine replication in today’s ESXi 5.1 announcement.
  • Virtual machine replication replicates your primary virtual machine to another datacenter so you can quickly bring up your virtual machines in a disaster event at the primary site.
  • A virtual machine is copied to the other site, a full copy followed by changed block copies are done.
  • Intelligent copy where application and/or database consistency is maintained using Microsoft Volume shadow copy service(VSS)
  • Replication is free product and is added as a feature
  • Replication runs off a agent that comes with each hypervisor install and a vSphere Replication appliance that runs on each vcenter at per vcenter basis. As of now, its maximum of 10 appliances per vcenter supporting 500 virtual machine replications per site.
  • Replication can be done on a per virtual machine and per hard disk basis.
  • The target virtual machine will not power on if the primary (source) virtual machine is pingable by vcenter server and/or is still powered on.
  • Recovery vm once powered on will have to be manually connected to the network via the vcenter web console (one click action).


This is exciting, VMware finally takes up Microsoft’s Hyper-V challenge and offers replication as a free product add on for all of us to enjoy.

So what is vmware replication really all about? Think of vmware replication as basically having a clone of your virtual machine on another hypervisor which may or may not be on the same site. Ideally, the replication will occur from one physical site to another. The catch is that this clone is constantly being updated – aka – any changes in the source site are transferred to the target site depending on the interval for replication you set.

The hypervisor inherently has a agent and there is a vmware replication appliance per vcenter whose job is to receive replication data and also keep track of the statuses. A vCenter can have upto 10 maximum vmware replication appliances. I am not clear on the maximum virtual machines that can be replicated but it seems like there is a cap on 500 virtual machines per vCenter.

The way it works, in the web based vcenter portal, you simply enable replication by a click and choose a virtual machine to replicate to another datacenter hypervisor. You get to choose how often the replication is to run, basically a RPO for your virtual machine. VMware allows a low RPO of 15 mins to a maximum of 24 hours. You can also set the destination virtual machine to have its disks as thin as opposed to the thick disks at the primary site. This allows you to save storage space on the target site.

Once that is set, you are all done, the initial replication will be a complete copy of your virtual machine over the wire. Now if you have a huge virtual machine, you can choose to pre-seed it to cut down on replication time. You can either copy it/clone it to a usb storage or FTP and then pre-seed that at the destination. Once Pre-seed is done, only changed blocks are copied. Change block tracking, i believe, is automatically enabled and only blocks that are changed are copied over. Remember the VA Agent is responsible to send over changed blocks and these are received by the VMware replication appliance on the other end.


Some cool things you can do is specifically pick which disks in a virtual machine you want to replicate and which you choose not to. For instance you can have a designated disk as swap disk for a virtual machine and you may choose not to replicate that disk.

To failover the virtual machine you simply have to initiate it from the vcenter web client, however remember, failover will not work or will not be allowed if your primary vm is still powered on and/or vcenter can ping it. When the target virtual machine comes up, it will disconnected from networks. The idea is for an administrator to look at the target and then to connect it to the network and not to have them blindly power on. This is to prevent accidental power ons that may result in network conflicts if any could happen.

I am quite sure that VMware is using the Virtual Disk Development Kit(VDDK) to get this all to work. The host agents that come loaded with the ESXi 5.1 install on the box use the VDDK to attach/detach disks and also ship data over wire. There are no security issues as of today, and replication is pretty secure.

I don’t have specific details yet but I will add more as we play along.

Do comment if you have any more information or need to correct/add info.

VMware Announces no more vRAM Entitlements!

Lets not even debate the whole structure about the vRAM pricing but VMware at VMWorld just announced (a minute back) that there are no more vRAM license entitlements. No vRAM, no per vm licensing model. Only per socket model which makes a lot of things easier. This was received with a lot of applause!