PACEMAKER LINUX HIGH AVAILABILITY STACK FOR OPENSTACK!

In a conference here about Openstack HA which is one of the most wanted requirements by enterprise customers. An over view of PaceMaker Linux Hugh availability stack for Openstack.

I am pretty new to this so glad I signed up for this and looking forward to find out more about it.

Will write more as I know it.

RIVERBED AND DESKTONE WITH OPENSTACK- Good stuff!

20121015-113957.jpg

I am here sitting down with the VP Of Engineering at Desktone – a managed hosted desktop virtualization firm and RiverBed that does WAN optimization

These two seem pretty cool stuff. While Desktone uses its own proprietary technology to do desktop virtualization, Riverbed uses intelligent packet analysis to only send differences across WAN. It’s smart enough to pick differences between multiple protocols as well.

VMware will soon allow Openstack hosting!

20121015-112104.jpg

At OpenStack summit in SanDiego and news from the VMware booth is that VMware ESXi will soon support spinning up OpenStack instances on VMware.

So does that mean the compute node will manage ESXi? Or does it mean ESXi will manage the compute nodes?
It surely is exciting. VMware already bought Nycera which is virtualization for networking and PAAS with Cloudfoundry.

I suspect it will be more than just this!

More as we know it 🙂

GeekTernet IS NOW RJAPPROVES

Wanted to move my domain to something that does not sound too gadgety. Thanks to Jason Castello, my site is now called RJAPPROVES – and every post I do – RJ Approves this message 🙂

ESXi 5.1 SUPPORTS SPAN, RSPAN AND ERSPAN – BUT WHAT ARE THESE?

vQuicky – For the impatient like me 😉

>vSphere 5.1 enhancements include support SPAN and RSPAN for network monitoring and analysis

> SPAN is a feature that allows you to mirror a target port to analyze traffic. It stands for Switch Port Analyzer

> In SPAN, you have a source port that is mirrored to a destination port. A single source SPAN port can be mirrored to multiple destination ports but it won’t work vice-versa

> In a SPAN session, both the source SPAN port and the destination SPAN port are on the same physical switch.

> For multiple switch analysis, RSPAN is used. It stands for Remote SPAN.

> RSPAN works exactly as SPAN however all the source SPAN port traffic is flooded in a special RSPAN VLAN. A port on this vlan can be used to analyze traffic.

> For WAN traffic analysis, ERSPAN is used which stands for Encapsulated Remote Switch Port Analyzer.

> ERSPAN is for routable traffic which spans over WAN.

> ERSPAN uses a ERSPAN source session, a routable ERSPAN GRE-Encapsulated traffic and a ERSPAN destination session. The source and destination sessions live on different switches across networks.

> Remember to enable promiscuous mode to pick up traffic.

inDepth

While going through whats new in vSphere 5.1, it is clear that they talk about ESXi 5.1 supporting network monitoring and troubleshooting features – SPAN and RSPAN. For as long as I have been in IT, I did not have a clear understanding of what these are so here goes.

SPAN – SPAN stands for Switch Port Analyzer. Think of this as port mirroring where you have a span port that mirrors all traffic going in and out of the mirrored port. The mirroring span port or destination span port is where you attach your traffic analyzer to check on the traffic that is nothing but a mirror of the source or mirrored span port. Traffic analyzer can be any thing such as wireshark for instance. The source port being monitored can be a switched or a routed port that is subjected to network analysis. You can also monitor bi-directional traffic or just sent or received traffic.

From my reading, a source port can be a anything such as a ether channel, fast ethernet, gigabit ethernet etc. A source port can also be monitored by multiple span sessions. As for the destination port, for a SPAN session, they should reside on the same switch and one destination port can participate in one span session only. So that means, it can only mirror traffic of one source span port. It cannot also self mirror – as in it cannot be the source port and the port cannot be a ether channel group either.

RSPAN – RSPAN stands for Remote SPAN. Now from above, it is easy enough to mirror a port on the same physical switch to sniff traffic but what if traffic is traversing across another switch or over the network? RSPAN allows you to monitor traffic all over your network. It is similar to SPAN in functionality but the only difference is that traffic is that mirrored traffic is flooded in the special RSPAN VLAN. Now you can hook up to any destination port that is part of this RSPAN VLAN and pick up traffic. SPAN and RSPAN work only at Layer 2 or LAN.

ERSPAN – ERSPAN stands for Encapsulated Remote Switch Port Analyzer. To be able to analyze traffic over WAN, use the ERSPAN feature. The way this works is that ERSPAN has a ERSPAN Source session, routable ERSPAN GRE-Encapsulated traffic and a ERSPAN destination session. For this to work you separately configure ERSPAN source and destination sessions on different switches.

Please comment or correct me if needed 🙂

More reading – http://www.ciscozine.com/2008/09/29/how-to-analyze-traffic-with-span-feature/

ESXi White paper – http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-51-Network-Technical-Whitepaper.pdf

REGISTERING A VM USING COMMAND LINE

vQuicky

Had no client access to the host nor to the vcenter. Needed vcenter vm which was already built added to the host’s inventory to get the show on the road.

SSH into the host

$ vim-cmd solo/registervm /vmfs/volumes/fullvmpath/vmname.vmx

That’s it!

CONFIGURE SELF SIGNED CERTIFICATES IN ESXi HOST

vQuicky – For the impatient like me 🙂

> VMware communications are all encrypted over SSL

> VMware uses self generated ssl certificates to encrypt session information.

> VMware uses standard X.509 Version 3 certificates which conform to Privacy Enhanced Mail and the key used is a RSA that ranges from 412 to 4096 bits with a recommendation of 2048 bits

> Download Openssl-Win32 (link below) and install it on a windows box to create the certificate signing request.

> Preconfigure openssl.cfg (below) and create your rui.csr and rui.key from the config file.

openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

> Either ship the certificate signing request to a third party SSL trust or create a rui.crt certificate using openssl commands

openssl x509 -req -days 365 -in rui.csr -signkey rui.key -out rui.crt

> Once done, copy rui.crt and rui.key to /etc/vmware/ssl

> If https error pops up, then the certificate has a passphrase – quick way to restore host is recreate vmware signed certificates by using the following command to restore vclient connectivity.

/sbin/generate-certificates

inDepth

I rebuilt my lab, however I am tired of seeing the above. So lets learn how to add SSL certs to our hosts and make them more secure. We all know that vSphere encrypts session information using digital certs. In my case for my lab, default certificates are fine by if you were working for a huge organization then standard SSL certificates may be a requirement. ESXi creates standard certificates by default which are not signed by a CA (certificate authority) and can also be vulnerable to man in the middle attacks.

VMware uses standard X.509 version 3 – also known as X.509v3 certificates to encrypt session information over SSL. This also applies to any communication between vCenter and Esxi host as well. Remember when you want to replace the default certificates, the new ones must conform to Privacy Enhanced Mail or PEM format. Privacy Enhanced Mail stores data in a Base-64 encoded distinguised encoding rules -DER format. As always the key used to sign certificates must be a standard RSA Key with an encryption length that ranges from 412 to 4096 bits. The whitepaper recommends a length of 2048 bits.

How to do it?

Before we dig deeper, remember if an ESXi 5.0 host is part of a HA Cluster, changing the SSL cert will break HA. To avoid this, make sure you are running vCenter 5.0 U1 or later. Now that Update 1 is out, you might as well upgrade to that before doing anything.

You need to download OpenSSL to create a self signed certificate. Now I am using Windows and downloaded the Win32 OpenSSL from slproweb. You can download it here. Once done simply install it.

Now we can either do it manually by answering all prompts and then removing the passphrase encrypt key or by manually editing the openssl.cfg file that you find in the directory location below.

In the file just replace the bold parts. I picked this up from the kb article.

# vSphere OpenSSL example configuration file start.
HOME = .
RANDFILE = $ENV::HOME/.rnd
oid_section = new_oids

[ new_oids ]

[ ca ]
default_ca = CA_default # The default ca section

[ CA_default ]

dir = ./demoCA # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
new_certs_dir = $dir/newcerts # default place for new certs.
certificate = $dir/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/cakey.pem# The private key
RANDFILE = $dir/private/.rand # private random number file
x509_extensions = usr_cert # The extentions to add to the cert
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
default_days = 5475 # how long to certify for
default_crl_days = 30 # how long before next CRL
default_md = sha512 # which md to use.
preserve = no # keep passed DN ordering
policy = policy_match

[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]
default_bits = 2048
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extentions to add to the self signed cert
input_password = testpassword
output_password = testpassword
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ] # change these settings for your environment
countryName = US
stateOrProvinceName = New York
localityName = New York
0.organizationName = Customer Name
organizationalUnitName = IT
commonName = vc.domain.com
emailAddress = [email protected]

[ req_attributes ]

[ usr_cert ]

basicConstraints =CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid,issuer

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: vc.domain.com, DNS: vc41.domain.com, DNS: vc41 #examples only

[ v3_ca ]
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid:always,issuer:always
basicConstraints = CA:true

[ crl_ext ]
authorityKeyIdentifier =keyid:always,issuer:always

[ proxy_cert_ext ]
basicConstraints =CA:FALSE
nsComment = “OpenSSL Generated Certificate”
subjectKeyIdentifier =hash
authorityKeyIdentifier =keyid,issuer:always
proxyCertInfo =critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

# vSphere OpenSSL example configuration file end.

Once saved, open up the Openssl directory which is typically located at C:\OpenSSL-Win32\bin

We now want to get the certificate signing request and the key created from the config file above. The command is

>openssl req -new -nodes -out rui.csr -keyout rui.key -config openssl.cfg

I initially did not pre configure the openssl.cfg. You can do it that way as well but remember to remove the passphrase from the key as you don’t want to enter the passphrase for the http daemon every time.  If you were wanting to use a commercial signed certs then just send them this rui.csr and they will send in the certificate. To create your own self-signed certificate run the following command.

>openssl x509 -req -days 365 -in rui.csr -signkey rui.key -out rui.crt

Once we have the above, you will have three files in the /bin directory which are rui.key, rui.crt and rui.csr. We need the rui.crt and the rui.key files.

Use WINSCP to copy these two files to the host. In the host navigate to /etc/vmware/ssl and copy the files.  Once copied, restart the management agents. Alternatively you can reboot as well. If something screws up no worries as you can restore the vmware certificates by recreating them. You can ssh to the host and recreate the vmware signed certs by the following command.

>/sbin/generate-certificates

>/sbin/services.sh restart

If all went well, get to the https://ipaddress and you should see your self signed cert installed.

Now you have your certificate installed!

Please comment or correct me if you find some errors 🙂

THE 100 SECOND sVMOTION TIMEOUT

vQuicky – For the impatient like me

> Too many disk operations on a datastore will lead to storage vmotion time out to occur

> The time out is a consequence of overhead per disk during storage vmotion. Over head involves opening, closing and processing disks

> Default time out is 100 seconds but can be increased by adding a parameter fsr.maxSwitchoverSeconds

> To add the row – power off the virtual machine –>in Options tab –>Advanced general –> Configuration parameters —–>Add row

> If manually editing the .vmx file, remember to remove the virtual machine from inventory first, modify .vmx and then re-add it back

inDepth

In my lab I was moving many virtual machines around and got this error.

“The migration has exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source.  To avoid this failure, either increase the maximum allowable switchover time or wait until the VM is performing a less intensive workload.”

This happens when a Virtual Machine with many disks is unable to complete Storage vmotion. In my case, I had a high latency FREENAS box hosting iscsi luns and in an attempt to rebuild it, I was moving many machines to local hypervisor disks. Remember that storage vmotion requires time to open, close and process disks in its final copy phase. So storage vmotion working on many disks will timeout because of this over head. So basically, large number of operations occurring on the same datastore will cause it to timeout.

To fix this we increase the fsr.maxSwitchoverSeconds to more than 100 seconds which is the default. To do this you will have to power down the machine –> Edit settings –> Options Tab –> Advanced general –> Configuration parameters –> Add row and add the “fsr.maxSwitchoverSeconds = 200”

Remember if you are manually editing the .vmx file, power off the virtual machine, remove it from the inventory – modify the .vmx file and then re-add it to the inventory.

Please do comment or correct me 🙂

Upcoming Blog List

I have been procrastinating but the following are my upcoming posts.

1. SSL Configuration on host – How to configure custom certs on host.

2. Talk about what NSPAN, ERSPAN are – which are supported in ESXi 5.1 Networking

3. Talk about LACP which is supported in new ESXi 5.1 Networking – Link Aggregation Control Protocol to bind links to push traffic

Atleast  these three before this week – I promise!

VSPHERE CLIENT SHOWS XXX IN EVENT LOG!?

The other day my vSphere client started showing logs (in the events tab) as XXX.Snapshot… This was actually on another pc and I thought the hypervisor went FooBar. I was operating a vSphere 4.1 box and using a vSphere client 4.1 obviously.

Turns out it was localization issue. Localization issue means vSphere client’s default language is different than what the vCenter is installed as. That did not still make sense to me that why would that matter in the logs – everything else was in English.

Regardless, the fix was to set -locale en_US to the vSphere client. You do that as follows,
1. Click start–>run
2. Type vSphere client executable path
“c:/……./vpxclient.exe” -locale en_US
3. Click ok

That should do it. Make sure you have the language pack installed on your OS incase you were using something else as a base language.

Let me know if you want to add to this or correct me 🙂