Patching ESXi 5.1 embedded on internal usb-storage
So I came across a unique issue tonight while doing some host installs.
How do you patch an Embedded ESXi 5.1 host, that only has internal USB Storage for data?
Well an easy way would be to copy the patches to a flash drive, plug the flash drive in and mount the usb-flash drive.
Well only if it was that simple. But it can be!
So doing some quick Googling, I found I was not the only engineer out there who wanted to do something along the same lines.
Here's the process:
Download and copy the esxi-patch.zip file over to your usb-storage device (ensuring that your usb-storage-flash device is formated in FAT) esxi will not read ntfs. Also no need to extract the .zip's the patcher will do that for you.
Start the SSH and ESXi Shell process on your host.
Login via SSH
run '/etc/init.d/usbarbitrator stop'
Plug your usb-thumbdrive in.
run the following commands to make sure that your host see's the usb-storage device:
esxcli storage core device list | grep -i usb
esxcli storage filesystem list
If you see your usb-storage devices listed in the above commands, simply run:
ls -la /vmfs/volumes
And your usb-storage device should be already mounted!
Now you just have to apply the patches.
esxcli software vib update -d "/vmfs/volumes/path-to-usb-storage/esxi-patchname.zip"
Note: It will seem like it's not doing anything after you hit enter, just be patient, it's updating!
Glue, what does it mean to you?
Over the past few months, I have had the chance to have some constructive conversations with some fellow engineers and manager's. One of the repeating topics that kept arising during our conversations was the concept of 'glue'. What is 'glue'? you might ask, to most people glue is the sticky white/clear paste that you put in between two pieces of paper to hold them together. On the other hand if you were on the topic of construction and kindergarten materials, you would be correct in your assumption. What about on the topic of professionalism and internal growth? Some would say that glue, or the accountability and responsibility of tasks and actions for a person can show glue not only in the work place but their personal life. According to Deborah Mackin, The ability to handle the tasks assigned to them, move through them, keeping them balanced and the ability to hold their head up and move forward while doing those tasks can be classified as glue. Professionalism, responsibility and accountability can help to be managed and controlled by the manager or team lead of any group or team of people, but they do not always distinctly lie on the management itself. These concepts are simple and easy to understand. Whether you decide to apply them to your career or personal life, are completely up to you. As for myself, since there are still many things for a experienced employee to work on and improve. Finding what the 'glue' is for me, can help to distinguish me from the ordinary employee, to the stand out employee.
P2V’ing CentOS5.1 with an LVM
So this week I accepted the task of P2V'ing a machine for one of our customers that has long been 'untouched' for years. Doing some initial digging I found that the machine was built using CentOS5.1 or otherwise known as the free RHEL5.1 kernel (2.4.x). Doing some brain storming with a fellow colleague we discussed the various options of getting the machine from a physical to virtual status, with the least amount of possibilities in failing the kernel and having to start over. The following options were considered:
Clonezilla
VMware P2V
Shadowprotect Image Creator
Using a Netbackup Restore
Doing a hot rsync transfer of the kernel.
After some deliberation, we decided that we would try the Clonezilla image. Of course taking this course required that we have a downtime for the machine so that it could be properly cold imaged.
Unfortunately, the clonezilla images did not work, and we ended up going with a P2V. Since, the images did not work, the task of getting the P2V was passed to me, since I have the experience with Linux and VMware that would be required to get this working.
After about 4 hours of waiting for the p2v to finish, I was finally able to try and turn the VM on and see what happened. Not being suprised, first time I powered on the vm I received a kernel panic. So I did a google search to see if the centos5.1 kernel was even still available, and sure enough after about 15 minutes of searching, I finally found the iso's that I was looking for.
Next step, to boot the vm up in safe mode to see what I was dealing with. Good news, safe mode found the kernel, but had to mount it as /mnt/sysimage; unfortunately this means that there is a driver issue and the machine was unable to locate the correct drive on the boot. So I started through the process of rotating through the drivers, first starting with the LSI Logic drivers (mptbase, mptscsih); tried a reboot, no success. Note: I also tried recreating the boot by reruning setup, but this also did not help.
What ended up working was changing the drivers in /etc/modprobe.conf to use the mptbase driver on the main scsiadapter, and all other scsiadapters to use the ata_piix driver, then running kudzu, and then doing a mkinitrd kernel.smb.img kernel.img, reboot, and bingo it booted.
Also to note, I rebooted after changing the /etc/modprobe.conf file original, and was able to do an lvdisplay, pvdisplay, vgdisplay and show all lvms present and accounted for, I think this was the key to stepping forward. Also good to remember that if your boot/os is not on an LVM, booting into safe mode constantly to fix this is not needed, it can all be done while the kernel is running live.
Here's a helpful VMware thread I found
Citrix VIAB and VMware ESX
So there’s a known issue with VDIB where if you go from ‘test-image’ back to ‘edit-image’ the VM can not find the correct VMDK file.
Easy fix.
The error will read can not find “NAMEOFVM-0001.vmdk” this is a snapshot file that Citrix deleted when going back to the edit-image version of the VM.
Download the .vmx file to your desktop and open it with Wordpad (must use wordpad because of formatting).
Look for the following line.
scsi0:0.fileName = "NAMEOFVM.vmdk"
If it says ‘scsi0:0.fileName = "NAMEOFSERVER-001.vmdk"’
Remove the ‘-001’ file from the line so it reads as the above. Save it, and upload the file through VIC back into the folder for the VM.
Now power on the server and it will power up correctly.
IBM x3650 and VMware ESXi.4.1U2 Embedded
Today I had the honor of doing the base installation/configuration of one of our customers new x3650-M4's. A few things that I found out how to do today were the automated firmware upgrade provided by IBM, what can I say, it was extremely easy.
Using IBM's provided tool I downloaded the patches, it automatically creates a bootable iso, that I burned off to a DVD.
Put the DVD in so that the system can boot to it, hit enter a couple times and away it goes. All automated!
Next step was to roll back from a ESXi5.0 embedded install we received to ESXi4.1u2; nothing other than purely for RAM capacity limitations in 5.0. This was a sinch as well, download the IBM provided iso, burn to a DVD, and again insert at boot, install to the provided USB internal stick.
In all I think it took ~ 2 hours to download, burn, and upgrade/install the firmware and OS. Pretty smooth imho.
Not your typical everyday VMhost install, but easy enough!
Patching ESXi5 with PowerCLI and ESXiShell
Using some free time I had at work (after hours of course), I decided to patch our develop environment.
After going to the VMware Update site and downloading all the available patches, I used WinSCP to copy over the patches from my machine to the host.
You will have to start the ESXi Shell and SSH services on your host.
I did the following, since you will have to reboot your host for these patches to take affect.
1. Power off any powered on VMs
2. Place your host in maintenance mode
3. SSH into your host, and navigate to an available datastore /vmfs/volumes/datastore/...
4. for each update, create it's own directory, mv the .zip file into that directory and unzip it. (command to unzip is unzip lol.)
5. after you have unzip'd the patch, open powercli, and do connect-viserver, type in the name of your server, and a login box should pop up.
6. Type 'Patch-VMHost "hostname" -noreboot '
A progress bar should appear at the top of your powercli screen, and if the screen populates with some information, just scroll up and it should say that the host requires a reboot to finish patching.
6a. (if reboot required) go back to your putty session, and simply type 'reboot'
Rinse and repeat per patch.
Certifications
It's been a while since i've actually had time to post, but most recently over the past few months I have been nose deep in my new job.
I have been fortunate to work for a VMware and IBM partner, and have received several certifications and accrediations from the following I would like to happily announce:
VTSP - VMware Technical Sales Professional 5
VSP - VMware Sales Professional 5
IBM Storage and Solutions Specialist - Storwize v7000 Technical Solutions V1
CIPP Partner Technical Pre-Sales Training Plan - vCloud
Infrastructure Virtualization Technical Post-Sales Accreditation: vSphere 5 Upgrade
Infrastructure Virtualization Technical Sales Accreditation
VBCA Technical Sales Accreditation
IaaS Technical Sales Accreditation 1
VMware Competitive Accreditation (vCA) Program (Version 5 Update)
edit: Added a few more today 8/14/2012
Infrastructure Virtualization Technical Post-Sales Accreditation: Assessment Fundamentals 5
Infrastructure Virtualization Sales Accreditation 5
CIPP Partner Technical Post-Sales Training Plan - Your Cloud Whiteboard
Business Continuity Technical Post-Sales Accreditation: Overview and Design 5
Business Continuity Technical Sales Accreditation 5
Backups
It seems like it's been never ending, but for the last few weeks I was put in charge of getting our customer's backup 'situation' squared away.
Just like normal, I don't like to leave things in a state where it needs tending to. I like to finish it out completely, and tweak afterwords if necessary. After examining our disk2disk2tape solution, I noticed that most of our customers are using straight disk2tape, because it runs MUCH faster than d2d2t. At first, I kept asking myself why is this? What could be causing such slow write/read speeds on an FC san? Well the way SBE works, is that it actually does the backup over the network, and is not smart enough to realize that the luns are on the same SAN, and just to make the changes on each lun volume. So since the backup servers are physically connected to the network with 1GB/s ethernet cards, the backups were averaging around 700MB/min write speeds for the D2D and around 1.7GB/min from D2T. After analyzing the server, and realizing that it's only using one of its 6 available NICs, I teamed the 4 port intel pro/1000 nic, to form a 4GB/s 802.3AG interface.
The backups from disk to disk are now running at ~ 3.4GB/min, a dramatic increase from the single interface speeds I was seeing before. Now came the daunting illusion, which is an LTO5 tape drive still only writing at 2GB/min?!? I did a quick look at the properties of the tape drive and noticed that it's writing and reading in 64K blocks, now most people would be like yea, so? Well in order for the tape drive to write the blocks of data from the SAN to the tape drive, it actually has to break the 'segments' up from however big the SAN sends it over (I was thinking around 256 to 512k) to a 64k block. I upped the buffering and block's from 64k to 256k and the block to 1mb, and am now getting around 10-15GB/min from disk to tape!!
All I can say is I went from not knowing a single thing about SBE, to being nose deep in the way it runs, and have not only corrected the backup issues, but made them more efficient as well!
Yay!
Links
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
| « Apr | ||||||
| 1 | 2 | 3 | 4 | |||
| 5 | 6 | 7 | 8 | 9 | 10 | 11 |
| 12 | 13 | 14 | 15 | 16 | 17 | 18 |
| 19 | 20 | 21 | 22 | 23 | 24 | 25 |
| 26 | 27 | 28 | 29 | 30 | 31 | |