Category Archives: Virtualization

Everything about Virtualization


So I blew it, I tried upgrading vcenter but found it was easier for me to just uninstall the vcenter 5.0 and reinstall it with vcenter 5.1

However I underestimated the processes. I had to install vcenter SSO service first, then vcenter inventory service and finally the vcenter server. I do not recall that there was this dependency with 5.0 but its clear that in 5.1 we need those.
It just surprised me. It just threw an error if it should overwrite the vcenter server thats already registered with the current inventory service from the 5.0 version, I chose yes.
What I would ideally like is independent boxes doing one thing only. As in a separate machine doing db and separate vcenter, sso and inventory service boxes. I run two Dell T110 boxes and one AMD box that I built.
I might have to order more ram to max them to be able to support all my desires.
On a second thought, wanting to run nested vmware over OpenStack. Get a taste of both worlds!
More as I know it. How does your lab look?


Very few I have talked to heard about VXLan so here is a post. I will do my best to cover most of the info.

vQuicky – For the impatient like me

> VXLAN – Virtual eXtensible Local Area Network

> VXLAN is a way to move towards Network Virtualization

> VXLAN is an Mac-over-ip based methodology/protocol

> VXLAN is an attempt to solve the datacenter problems of 4095 vlans or 4095 mac address limits in top of the rack switches (TOR switches)

> VXLAN is a way to float “virtual domains” called VNI (VXLAN Network Identifier) over current hardware and network layers (Layer 2 and layer 3)

> Only VMs with in the same VXLAN segment can talk to each other.

> The VNI is a 24 bit segment id which allows for 16 million VXLAN segment ids in a single administrative domain.

>  The VNI is a outer header address with the inner MAC address of the VM

> VXLAN can also be considered similar to a L2 tunneling over L3 however it is stateless

> VTEP – VXLAN Tunnel End Point is where the VXLAN packet terminates and is within the hypervisor that houses the VM

>  VTEP’s can also be a VXLAN aware physical server or even a physical switch.

> VXLAN’s currently don’t take security into account as of today but its in works to make this a more inherent protocol to provide some sort of security towards a rogue node in L2 layer.

> There are recommendations of coupling VXLANs using IPSEC


Before I even attempt to write about VXLAN – let me snag this nice picture – courtesy of VMware.

Get the idea? Well VXLAN is a way to solve the datacenter issues where you are limited by vlans, switch ports and/or mac address limits. Its all about network virtualization and VMware is to work with Cisco and hopefully other software vendors to make this happen.

Looks like VMware finally realized that even though provisioning a vm can take any where from a few minutes to seconds (if you have SSDs) – getting the network part done can take days and sometimes even weeks. Clearly this is a big deal for Enterprises. There are ways to speed things up however they all still reply on the vlan, mac address and switch port limits.

So how does a typical vm to vm communication take place over VXLAN. Any machines on the same VNI are said to be in the same VXLAN Overlay Network. Now a vm in the VXLAN overlay network may be unaware that it is actually participating in MAC-Over-IP network virtualization. Now if this vm wants to send data to another vm – it does this by sending the packets to that mac address. Now the VXLAN Tunnel End point (VTEP) which lives on the host looks up the VNI to which this vm belongs to. The VTEP also checks if the destination vm belongs to the same VNI as well and if it does then it takes the outer ip address, outer mac and the VXLAN header are inserted to the original mac frame. Now the packet is sent over to the destination which is the remote ip of the VTEP hypervisor that hosts the destination vm.

When the other VTEP receives the packet, it verifies if the VNI is valid and if the vm belongs to it, then stripes off the headers to the mac frame that were added by the source VTEP and then passes the packet to the VM. The destination vm also learns about the inner source mac of the source vm and the outer ip address as well. It records/stores this info so if that vm needs to send a response or has to communicate in the future, it does not have to flood the request.

A brief about broadcast communications – Any broadcast traffic with the VXLAN is done by adding a header that includes VXLAN VNI information along with the IP header and the UDP header. Now this broadcast traffic is sent to the IP Multicast group that has the VXLAN overlay network defined to it. For this to be possible there must be a association of the IP multicast group to a specific VXLAN overlay group and this is done at the management layer and provided to the VTEP through a management channel.

Within VMware’s context – the encapsulation is performed between the virtual NIC of the guest VM and the logical port on the virtual switch. This makes VXLAN transparent to both the guest VMs and the underlying Layer 3 network. Gateway services between VXLAN and non-VXLAN hosts are performed by Security Edge gateway appliance. The Edge gateway translates VXLAN segment IDs to VLAN IDs, so that non-VXLAN hosts can communicate with VXLAN virtual servers.

Feel free to comment 🙂



In a conference here about Openstack HA which is one of the most wanted requirements by enterprise customers. An over view of PaceMaker Linux Hugh availability stack for Openstack.

I am pretty new to this so glad I signed up for this and looking forward to find out more about it.

Will write more as I know it.



I am here sitting down with the VP Of Engineering at Desktone – a managed hosted desktop virtualization firm and RiverBed that does WAN optimization

These two seem pretty cool stuff. While Desktone uses its own proprietary technology to do desktop virtualization, Riverbed uses intelligent packet analysis to only send differences across WAN. It’s smart enough to pick differences between multiple protocols as well.

VMware will soon allow Openstack hosting!


At OpenStack summit in SanDiego and news from the VMware booth is that VMware ESXi will soon support spinning up OpenStack instances on VMware.

So does that mean the compute node will manage ESXi? Or does it mean ESXi will manage the compute nodes?
It surely is exciting. VMware already bought Nycera which is virtualization for networking and PAAS with Cloudfoundry.

I suspect it will be more than just this!

More as we know it 🙂


Wanted to move my domain to something that does not sound too gadgety. Thanks to Jason Castello, my site is now called RJAPPROVES – and every post I do – RJ Approves this message 🙂


vQuicky – For the impatient like me 😉

>vSphere 5.1 enhancements include support SPAN and RSPAN for network monitoring and analysis

> SPAN is a feature that allows you to mirror a target port to analyze traffic. It stands for Switch Port Analyzer

> In SPAN, you have a source port that is mirrored to a destination port. A single source SPAN port can be mirrored to multiple destination ports but it won’t work vice-versa

> In a SPAN session, both the source SPAN port and the destination SPAN port are on the same physical switch.

> For multiple switch analysis, RSPAN is used. It stands for Remote SPAN.

> RSPAN works exactly as SPAN however all the source SPAN port traffic is flooded in a special RSPAN VLAN. A port on this vlan can be used to analyze traffic.

> For WAN traffic analysis, ERSPAN is used which stands for Encapsulated Remote Switch Port Analyzer.

> ERSPAN is for routable traffic which spans over WAN.

> ERSPAN uses a ERSPAN source session, a routable ERSPAN GRE-Encapsulated traffic and a ERSPAN destination session. The source and destination sessions live on different switches across networks.

> Remember to enable promiscuous mode to pick up traffic.


While going through whats new in vSphere 5.1, it is clear that they talk about ESXi 5.1 supporting network monitoring and troubleshooting features – SPAN and RSPAN. For as long as I have been in IT, I did not have a clear understanding of what these are so here goes.

SPAN – SPAN stands for Switch Port Analyzer. Think of this as port mirroring where you have a span port that mirrors all traffic going in and out of the mirrored port. The mirroring span port or destination span port is where you attach your traffic analyzer to check on the traffic that is nothing but a mirror of the source or mirrored span port. Traffic analyzer can be any thing such as wireshark for instance. The source port being monitored can be a switched or a routed port that is subjected to network analysis. You can also monitor bi-directional traffic or just sent or received traffic.

From my reading, a source port can be a anything such as a ether channel, fast ethernet, gigabit ethernet etc. A source port can also be monitored by multiple span sessions. As for the destination port, for a SPAN session, they should reside on the same switch and one destination port can participate in one span session only. So that means, it can only mirror traffic of one source span port. It cannot also self mirror – as in it cannot be the source port and the port cannot be a ether channel group either.

RSPAN – RSPAN stands for Remote SPAN. Now from above, it is easy enough to mirror a port on the same physical switch to sniff traffic but what if traffic is traversing across another switch or over the network? RSPAN allows you to monitor traffic all over your network. It is similar to SPAN in functionality but the only difference is that traffic is that mirrored traffic is flooded in the special RSPAN VLAN. Now you can hook up to any destination port that is part of this RSPAN VLAN and pick up traffic. SPAN and RSPAN work only at Layer 2 or LAN.

ERSPAN – ERSPAN stands for Encapsulated Remote Switch Port Analyzer. To be able to analyze traffic over WAN, use the ERSPAN feature. The way this works is that ERSPAN has a ERSPAN Source session, routable ERSPAN GRE-Encapsulated traffic and a ERSPAN destination session. For this to work you separately configure ERSPAN source and destination sessions on different switches.

Please comment or correct me if needed 🙂

More reading –

ESXi White paper –



Had no client access to the host nor to the vcenter. Needed vcenter vm which was already built added to the host’s inventory to get the show on the road.

SSH into the host

$ vim-cmd solo/registervm /vmfs/volumes/fullvmpath/vmname.vmx

That’s it!


vQuicky – For the impatient like me 🙂

> VMware communications are all encrypted over SSL

> VMware uses self generated ssl certificates to encrypt session information.

> VMware uses standard X.509 Version 3 certificates which conform to Privacy Enhanced Mail and the key used is a RSA that ranges from 412 to 4096 bits with a recommendation of 2048 bits

> Download Openssl-Win32 (link below) and install it on a windows box to create the certificate signing request.

> Preconfigure openssl.cfg (below) and create your rui.csr and rui.key from the config file.

openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

> Either ship the certificate signing request to a third party SSL trust or create a rui.crt certificate using openssl commands

openssl x509 -req -days 365 -in rui.csr -signkey rui.key -out rui.crt

> Once done, copy rui.crt and rui.key to /etc/vmware/ssl

> If https error pops up, then the certificate has a passphrase – quick way to restore host is recreate vmware signed certificates by using the following command to restore vclient connectivity.



I rebuilt my lab, however I am tired of seeing the above. So lets learn how to add SSL certs to our hosts and make them more secure. We all know that vSphere encrypts session information using digital certs. In my case for my lab, default certificates are fine by if you were working for a huge organization then standard SSL certificates may be a requirement. ESXi creates standard certificates by default which are not signed by a CA (certificate authority) and can also be vulnerable to man in the middle attacks.

VMware uses standard X.509 version 3 – also known as X.509v3 certificates to encrypt session information over SSL. This also applies to any communication between vCenter and Esxi host as well. Remember when you want to replace the default certificates, the new ones must conform to Privacy Enhanced Mail or PEM format. Privacy Enhanced Mail stores data in a Base-64 encoded distinguised encoding rules -DER format. As always the key used to sign certificates must be a standard RSA Key with an encryption length that ranges from 412 to 4096 bits. The whitepaper recommends a length of 2048 bits.

How to do it?

Before we dig deeper, remember if an ESXi 5.0 host is part of a HA Cluster, changing the SSL cert will break HA. To avoid this, make sure you are running vCenter 5.0 U1 or later. Now that Update 1 is out, you might as well upgrade to that before doing anything.

You need to download OpenSSL to create a self signed certificate. Now I am using Windows and downloaded the Win32 OpenSSL from slproweb. You can download it here. Once done simply install it.

Now we can either do it manually by answering all prompts and then removing the passphrase encrypt key or by manually editing the openssl.cfg file that you find in the directory location below.

In the file just replace the bold parts. I picked this up from the kb article.

# vSphere OpenSSL example configuration file start.
HOME = .
oid_section = new_oids

[ new_oids ]

[ ca ]
default_ca = CA_default # The default ca section

[ CA_default ]

dir = ./demoCA # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
new_certs_dir = $dir/newcerts # default place for new certs.
certificate = $dir/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/cakey.pem# The private key
RANDFILE = $dir/private/.rand # private random number file
x509_extensions = usr_cert # The extentions to add to the cert
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
default_days = 5475 # how long to certify for
default_crl_days = 30 # how long before next CRL
default_md = sha512 # which md to use.
preserve = no # keep passed DN ordering
policy = policy_match

[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 2048
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extentions to add to the self signed cert
input_password = testpassword
output_password = testpassword
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ] # change these settings for your environment
countryName = US
stateOrProvinceName = New York
localityName = New York
0.organizationName = Customer Name
organizationalUnitName = IT
commonName =
emailAddress = [email protected]

[ req_attributes ]

[ usr_cert ]

basicConstraints =CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid,issuer

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:, DNS:, DNS: vc41 #examples only

[ v3_ca ]
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid:always,issuer:always
basicConstraints = CA:true

[ crl_ext ]
authorityKeyIdentifier =keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints =CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid,issuer:always
proxyCertInfo =critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# vSphere OpenSSL example configuration file end.

Once saved, open up the Openssl directory which is typically located at C:\OpenSSL-Win32\bin

We now want to get the certificate signing request and the key created from the config file above. The command is

>openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

I initially did not pre configure the openssl.cfg. You can do it that way as well but remember to remove the passphrase from the key as you don’t want to enter the passphrase for the http daemon every time.  If you were wanting to use a commercial signed certs then just send them this rui.csr and they will send in the certificate. To create your own self-signed certificate run the following command.

>openssl x509 -req -days 365 -in rui.csr -signkey rui.key -out rui.crt

Once we have the above, you will have three files in the /bin directory which are rui.key, rui.crt and rui.csr. We need the rui.crt and the rui.key files.

Use WINSCP to copy these two files to the host. In the host navigate to /etc/vmware/ssl and copy the files.  Once copied, restart the management agents. Alternatively you can reboot as well. If something screws up no worries as you can restore the vmware certificates by recreating them. You can ssh to the host and recreate the vmware signed certs by the following command.


>/sbin/ restart

If all went well, get to the https://ipaddress and you should see your self signed cert installed.

Now you have your certificate installed!

Please comment or correct me if you find some errors 🙂


vQuicky – For the impatient like me

> Too many disk operations on a datastore will lead to storage vmotion time out to occur

> The time out is a consequence of overhead per disk during storage vmotion. Over head involves opening, closing and processing disks

> Default time out is 100 seconds but can be increased by adding a parameter fsr.maxSwitchoverSeconds

> To add the row – power off the virtual machine –>in Options tab –>Advanced general –> Configuration parameters —–>Add row

> If manually editing the .vmx file, remember to remove the virtual machine from inventory first, modify .vmx and then re-add it back


In my lab I was moving many virtual machines around and got this error.

“The migration has exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source.  To avoid this failure, either increase the maximum allowable switchover time or wait until the VM is performing a less intensive workload.”

This happens when a Virtual Machine with many disks is unable to complete Storage vmotion. In my case, I had a high latency FREENAS box hosting iscsi luns and in an attempt to rebuild it, I was moving many machines to local hypervisor disks. Remember that storage vmotion requires time to open, close and process disks in its final copy phase. So storage vmotion working on many disks will timeout because of this over head. So basically, large number of operations occurring on the same datastore will cause it to timeout.

To fix this we increase the fsr.maxSwitchoverSeconds to more than 100 seconds which is the default. To do this you will have to power down the machine –> Edit settings –> Options Tab –> Advanced general –> Configuration parameters –> Add row and add the “fsr.maxSwitchoverSeconds = 200”

Remember if you are manually editing the .vmx file, power off the virtual machine, remove it from the inventory – modify the .vmx file and then re-add it to the inventory.

Please do comment or correct me 🙂