Patching ESXi 5.1 embedded on internal usb-storage
So I came across a unique issue tonight while doing some host installs.
How do you patch an Embedded ESXi 5.1 host, that only has internal USB Storage for data?
Well an easy way would be to copy the patches to a flash drive, plug the flash drive in and mount the usb-flash drive.
Well only if it was that simple. But it can be!
So doing some quick Googling, I found I was not the only engineer out there who wanted to do something along the same lines.
Here's the process:
Download and copy the esxi-patch.zip file over to your usb-storage device (ensuring that your usb-storage-flash device is formated in FAT) esxi will not read ntfs. Also no need to extract the .zip's the patcher will do that for you.
Start the SSH and ESXi Shell process on your host.
Login via SSH
run '/etc/init.d/usbarbitrator stop'
Plug your usb-thumbdrive in.
run the following commands to make sure that your host see's the usb-storage device:
esxcli storage core device list | grep -i usb
esxcli storage filesystem list
If you see your usb-storage devices listed in the above commands, simply run:
ls -la /vmfs/volumes
And your usb-storage device should be already mounted!
Now you just have to apply the patches.
esxcli software vib update -d "/vmfs/volumes/path-to-usb-storage/esxi-patchname.zip"
Note: It will seem like it's not doing anything after you hit enter, just be patient, it's updating!
P2V’ing CentOS5.1 with an LVM
So this week I accepted the task of P2V'ing a machine for one of our customers that has long been 'untouched' for years. Doing some initial digging I found that the machine was built using CentOS5.1 or otherwise known as the free RHEL5.1 kernel (2.4.x). Doing some brain storming with a fellow colleague we discussed the various options of getting the machine from a physical to virtual status, with the least amount of possibilities in failing the kernel and having to start over. The following options were considered:
Clonezilla
VMware P2V
Shadowprotect Image Creator
Using a Netbackup Restore
Doing a hot rsync transfer of the kernel.
After some deliberation, we decided that we would try the Clonezilla image. Of course taking this course required that we have a downtime for the machine so that it could be properly cold imaged.
Unfortunately, the clonezilla images did not work, and we ended up going with a P2V. Since, the images did not work, the task of getting the P2V was passed to me, since I have the experience with Linux and VMware that would be required to get this working.
After about 4 hours of waiting for the p2v to finish, I was finally able to try and turn the VM on and see what happened. Not being suprised, first time I powered on the vm I received a kernel panic. So I did a google search to see if the centos5.1 kernel was even still available, and sure enough after about 15 minutes of searching, I finally found the iso's that I was looking for.
Next step, to boot the vm up in safe mode to see what I was dealing with. Good news, safe mode found the kernel, but had to mount it as /mnt/sysimage; unfortunately this means that there is a driver issue and the machine was unable to locate the correct drive on the boot. So I started through the process of rotating through the drivers, first starting with the LSI Logic drivers (mptbase, mptscsih); tried a reboot, no success. Note: I also tried recreating the boot by reruning setup, but this also did not help.
What ended up working was changing the drivers in /etc/modprobe.conf to use the mptbase driver on the main scsiadapter, and all other scsiadapters to use the ata_piix driver, then running kudzu, and then doing a mkinitrd kernel.smb.img kernel.img, reboot, and bingo it booted.
Also to note, I rebooted after changing the /etc/modprobe.conf file original, and was able to do an lvdisplay, pvdisplay, vgdisplay and show all lvms present and accounted for, I think this was the key to stepping forward. Also good to remember that if your boot/os is not on an LVM, booting into safe mode constantly to fix this is not needed, it can all be done while the kernel is running live.
Here's a helpful VMware thread I found
Patching ESXi5 with PowerCLI and ESXiShell
Using some free time I had at work (after hours of course), I decided to patch our develop environment.
After going to the VMware Update site and downloading all the available patches, I used WinSCP to copy over the patches from my machine to the host.
You will have to start the ESXi Shell and SSH services on your host.
I did the following, since you will have to reboot your host for these patches to take affect.
1. Power off any powered on VMs
2. Place your host in maintenance mode
3. SSH into your host, and navigate to an available datastore /vmfs/volumes/datastore/...
4. for each update, create it's own directory, mv the .zip file into that directory and unzip it. (command to unzip is unzip lol.)
5. after you have unzip'd the patch, open powercli, and do connect-viserver, type in the name of your server, and a login box should pop up.
6. Type 'Patch-VMHost "hostname" -noreboot '
A progress bar should appear at the top of your powercli screen, and if the screen populates with some information, just scroll up and it should say that the host requires a reboot to finish patching.
6a. (if reboot required) go back to your putty session, and simply type 'reboot'
Rinse and repeat per patch.
Linux load balancing and failover
Now that I have my server up and running and have fixed the replication between the 2 VMs, it's time to get failover and load balancing configured between the two. I'll be updating this post with more information as I test the environment and figure out what I need to add and configure.
Got cron finally working the way I want to, figured out a few things:
Need to have an empty line after your commands in cron, and for the most part, you have to put the fq name of the service in. ex. /usr/bin/cron or /usr/sbin/zypper, etc.
TBC
MySQL Tweaking
So recently been noticing a bit a only what I can describe as 'lag' when queuing data from the mysql database on my server. Since I've never really played with the performance tweaking aspects of mysql on my server; I thought I should take a look into it.
After some quick googling I found this script that runs itself and gives you a very easy to read, performance tweaking readout of the mysql server settings, and tips on how to improve it. I have to say, right off the bat, it pointed out some key things that I could change to even enhance the security settings that I seem to have overlooked.
I've got to give the writer of the script props, because it works great!
Here's a link to it: http://www.debianadmin.com/check-your-mysql-server-performance-with-mysqltuner.html
Now just need to research into getting some optimize scripts to run weekely on the databases to get rid of my fragmented tables. Back to googling!
VMW e.x.p on Fedora 16
Well, after getting dual boot to work with WIN8CP and Fedora 16, I decided to move to F16 as more of a permanent OS. I've spent the last few nights until about 4 or 5 am working on some of these things. So please excuse my poor thought process, havn't had too much sleep lately.
Reminder this is for VMware Workstation 8 Tech Preview NOT 8.0.2
Step 1:
Get hardware raid permanently mounted in /etc/fstab ... pretty easy, figure out what the name of the raid array is by going into the disk utilities app and add it to /etc/fstab with the normal defaults 0 0 settings.
Step 2:
Download and install nVidia Video drivers:
This took a few more steps. Right away the nvidia drivers setup requires you to kill your x server. Easy step, type 'init 3' and re run the setup. Next thing it asked me to do was disable the Nouveau driver, ran a quick google search on disabling Nouveau drivers in Fedora 16, instructions worked perfectly, rebooted, setup the 2nd monitor. Next step.
Step 3:
Download and install VMware Workstation 8 Tech Preview.
I ran into a few issues right away with the installation, not having the correct packages installed was the biggest issue. I ended up adding the development repo's for the new fedora 3.3.1.5 kernel and installing the necessary packages with it; then run a yum update, and reboot. After the reboot I was still getting errors that seem to progress along with the version of 8.0.2, I ended up having to install 8.0.2 first, run the patch for 8.0.2 thats available in the forums on the vmware site, and then upgrade 8.0.2 to e.x.p since the patch does not work for e.x.p. Note: I had to upgrade to e.x.p because of the hardware version and version of vmware tools that are installed on my vms. They would not boot unless I powered them on in my windows OS, downgraded the hardware level and vmware tools, and then tried to power them on in Fedora. The next issue I came across was, the 'Enter License Key" function in the GUI would not work, just click and nothing. After find this thread (http://communities.vmware.com/message/2029214#2029214) on the community site, a little light bulb went off in my head about the license file. So I tried to edit the license file with the S# of the e.x.p version and restart the vmware service. No luck. After a few hours of doing more google searching and trying different possibilities. In that folder you will find 3 different types of licenses, ws, ws.vl, ws.eval; which I can only assume are the different types of licenses available for purchase. After running a 'cat license-ws-90-e*-201202' I noticed that the license type for each file was different. Since e.x.p is available free to the general public, the Serial Number for the license can be found on the website where you can download VMW-TP, I assumed that it was a full version of the product. There are also 2 different versions of license files in that folder, what looks to be the version for 8.0.2, and the version for e.x.p. I copied the license-ws-90-e1-201202 file to the /etc/vmware/ directory, since that correlates to the type of license I needed. Then after running another cat on the original license file, I merely just edited the new license file with "Serial = 'XXXXX-XXXXX-XXXXX-XXXXX-XXXXX'", saved and restarted the VMware service. Tada!! You should now be able to start the VM, but may have a new issue, an error about the Kernel version being different then what is found. I simply rebooted the machine, and re-ran VMW, and it started to work.
Step 4: Get my sound/keyboard/mouse drivers installed
TBD
Hope this helps!
Linux
I'm really starting to like having a dev environment as well, I'm getting a chance to play with linux automation on a whole new level then I did before. My next idea is to figure out pgsql replication using rsync. Havn't done that one before and I think it'll be something that I will enjoy.
zypper dup = dumb idea
So I decided to go ahead and try the dist upgrade for my vm this weekend.
Like an idiot I forgot to take a snapshot first.
Learned a bunch about how php5 functions though, and how to manually configure it. So that's something on the plus side.
Links
| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
| « Apr | ||||||
| 1 | ||||||
| 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| 9 | 10 | 11 | 12 | 13 | 14 | 15 |
| 16 | 17 | 18 | 19 | 20 | 21 | 22 |
| 23 | 24 | 25 | 26 | 27 | 28 | 29 |
| 30 | ||||||