A quick and dirty munin plugins to count number of VM running on RHEL/CentOS based KVM host.

So recently I was configuring munin to monitor some QEMU/KVM hosts which based on generic RHEL servers (Noted, not RHEV) which run libvirtd and QEMU/KVM.

So here is a plugins that I created, it is quick and dirty but this should work as expected. Just copy and paste the plugins file into /etc/munin/plugins/ directory and make sure it is executable (755, ideally), then you should be good.

So here is the content of the file.

[root@localhost plugins]# cat /etc/munin/plugins/vm_count
#!/bin/sh

case $1 in
   config)
        cat <<'EOM'
graph_title Number of VMs
graph_vlabel VMcount
vmcount.label VMcount
vmcount.graph_category Vserver
EOM
        exit 0;;
esac

i=`ps auxww | grep [/]usr/libexec/qemu-kvm | wc -l`
echo -n "vmcount.value "
echo $i

And it is how it would work.

# You should be able to execute it directly from system shell. In this example I had 17 VMs running on the host.

[root@localhost plugins]# pwd
/etc/munin/plugins
[root@localhost plugins]# ./vm_count
vmcount.value 17

# Alternatively, you can test it with munin-run. This is how the script will look like when it is being loaded

[root@localhost plugins]# munin-run vm_count
vmcount.value 17




# And here is the parameters of this plugins.

[root@localhost plugins]# munin-run vm_count  config
graph_title Number of VMs
graph_vlabel VMcount
vmcount.label VMcount
vmcount.graph_category Vserver

Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N

So,  because of my job duty and I have to deal with Citrix Cloudstack day by day. Recently we are deploying a new advanced zone and for some reason we are seeing errors like this during deploy of our first VM instance.


2013-07-22 22:09:27,625 WARN  [api.commands.DeployVMCmd] (Job-Executor-50:job-534828) Exception:
com.cloud.exception.AgentUnavailableException: Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N
................
Caused by: com.cloud.utils.exception.CloudRuntimeException: Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N
................
2013-07-22 22:09:27,626 WARN  [cloud.api.ApiDispatcher] (Job-Executor-50:job-534828) class com.cloud.api.ServerApiException : Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N

So, basically, what Cloudstack doing is to
1. check if there is any valid systemvm template (in this case systemvm-kvm-3.0.0) deployed to the zone.
2. If things works as it should, you should be able to find the installed/downloaded template from table cloud.vm_template, cloud.template_zone_ref and template_host_ref. Hence, if you scan through the template list from the Web GUI, you should be able to see the template be downloaded.

In my case, the template was not downloaded as it should (or marked as downloaded at DB layer), and if you look at the table cloud.template_host_ref, there is some abnormality here.

mysql> select * from  template_host_ref where id=11111\G
*************************** 1. row ***************************
            id: 11111
       host_id: *masked*
   template_id: *masked*
       created: 2013-07-18 17:50:43
  last_updated: 2013-07-22 20:04:52
        job_id: 75a75e55-5280-4ba5-b823-cadbcbe2cc7a
  download_pct: 0
          size: 0
 physical_size: 0
download_state: DOWNLOAD_ERROR
     error_str: No route to host
    local_path: /mnt/SecStorage/04ab8f0b-c4e0-34a4-80b3-457c433acde3/template/tmpl/2/1686/dnld6951269530983090325tmp_
  install_path: NULL
           url: http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2
     destroyed: 0
       is_copy: 0

So, basically the things are 1) download_pct is 0 (while it should be 100 if download succeed), 2) download_state is DOWNLOAD_ERROR (while it should be DOWNLOADED if download successed and 3) error_str is "No route to host".

In my case, the template installation procedures was not completed (though I have completed the cloud-install-sys-tmplt script per official installation guide), at least at DB layer.

So I double checked the secondary storage to make sure the template file is completely downloaded (IMPORTANT!!!, if the file is not there, go through installation guide and re-run cloud-install-sys-tmplt script) and hacked the DB by updating the cloud.template_host_ref table. (Replace "N" with the correct account id and template id respectively)

mysql> updated template_host_ref set download_pct=100, download_state='DOWNLOADED', error_str=NULL, localpath='template/tmpl/N/N' where id=11111\G
*************************** 1. row ***************************
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Now cloudstack could launch VM as it should.

QEMU/KVM atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)

So I am getting this annoying warning message when I press Enter on my KVM VM.

# 
atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)
atkbd.c Use 'setkeycodes 00 <keycode>' to make it known.

So now whenever I hit enter, the annoying warning will be coming up and this can be easily cleaned up by using the showkey and setkeycodes command.

So from the tty console (it may not work on x-windows), i executed showkey

# showkey
kb mode was UNICODE
[ if you are trying this under X, it might not work
since the X server is also reading /dev/console ]

press any key (program terminates 10s after last keypress)...

Now you can press 'Enter' (or whatever key that could cause the annoying message), it will show the keycode

keycode  28 press

It means that the Enter key is associated to keycode 28. Now you can use setkeycodes to map the Enter key.

# setkeycodes 0x00 28

0x00 here is the scancode, if you are interested on what it means, you could google it and there is some other articles out there explaining this.

Now, right after you typed the command, you should now be safed from this annoying warning message.

Should you want this be sustained across reboot, you may want to add it to /etc/rc.local

# echo 'setykeycodes 0x00 28' >> /etc/rc.local

virtio_nic or e1000 on Linux KVM?

If you are RHEV / QEMU KVM user, you probably launched some virtual machines and yeah it is just as easy as couples of clicks. However, did you ever notice the reason why we have to fill in OS Type and Version ? And, actually VM still can boot up even you didnt fill in the exact OS type and Version for your VM.

The reason is that OS type and Version are used to define whether para-virtualized device will be used for the particular VM. Linux para-virtualized driver starts to run on kernel 2.6.25 or later and if you are running with a VM guest based on older kernel, chances that you will not be able to take advantages from it. For e.g. if you have a VM based on CentOS 4 (running with old 2.6.9 kernel) and you selected OS Type as CentOS 4, the VM will be configured to use simulated block device (/dev/sdX) and Intel e1000 emulated network card and these devices are actually not doing any better to paravirtualized device. However, if you start a VM by defining OS type Ubuntu 10.04 (i.e. kernel version 2.6.32 which does come with para-virtualized driver) and then you really load a Ubuntu 10.04 to it, it would recognize the para-virtualized device and make use of para-virtualized driver. In situation like this, your VM will be presented with para-virtualized block device (/dev/vdX) and virtio network card (virtio_nic) which would take full advantages of para-virtualized kernel.

Migrating AWS CentOS Linux images to KVM environment.

For some reason, I would like to port the AWS EC2 Linux images from AWS environment to inhouse KVM environment. I choose to

So this time I am choosing a EBS-based CentOS based template, centos-x64-6.0-core (ami-03559b6a). I think below mechanism would work the same on instance-store template as well.

I quickly launched an instance from ami-03559b6a and then logging into the instances to have a glance.



[root@ip-10-195-81-211 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvde1            6.0G 1010M  4.7G  18% /
tmpfs                 296M     0  296M   0% /dev/shm

[root@ip-10-195-81-211 ~]# fdisk -l

Disk /dev/xvde: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xaae7682d

    Device Boot      Start         End      Blocks   Id  System
/dev/xvde1   *           1         783     6289416   83  Linux

It is an instance running on xenserver, the root disk is 6G. In order to clone the root disk, I created an EBS volume with size sightly larger than 6G. In this example I use add a 10G EBS volume.



[root@ip-10-195-81-211 ~]# fdisk -l

Disk /dev/xvde: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xaae7682d

    Device Boot      Start         End      Blocks   Id  System
/dev/xvde1   *           1         783     6289416   83  Linux

Disk /dev/xvdj: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/xvdj doesn't contain a valid partition table

So now /dev/xvde is the root disk, /dev/xvde1 is the partition in-use while /dev/xvdj is the disk attached to this instance and there is no partiton being created here.

We will now create a file system on /dev/xvdj and then create a raw disk image on top of that.


[root@ip-10-195-81-211 ~]# mkfs.ext4 /dev/xvdj
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done                           
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

[root@ip-10-195-81-211 ~]# mkdir /target
[root@ip-10-195-81-211 ~]# mount /dev/xvdj /target

It will take some time to run dd though, depends on the disk image size. In this example we are going to create a disk image with 6442MB in size (same as the size of /dev/xvde showed in fdisk).

[root@ip-10-195-81-211 ~]# time dd if=/dev/zero of=/target/diskimage.raw bs=1M count=6442
6442+0 records in
6442+0 records out
6754926592 bytes (6.8 GB) copied, 178.33 s, 37.9 MB/s

real    2m58.363s
user    0m0.010s
sys    0m9.586s
[root@ip-10-195-81-211 ~]# ls -la /target/diskimage.raw
-rw-r--r-- 1 root root 6754926592 Apr 12 02:33 /target/diskimage.raw




And then using losetup to setup the disk image as loopback disk /dev/loop0.

[root@ip-10-195-81-211 ~]# losetup -fv /target/diskimage.raw
Loop device is /dev/loop0

Use diskdump to clone /dev/xvde to /dev/loop0, it will take some time as well.

[root@ip-10-195-81-211 ~]# time dd if=/dev/xvde of=/dev/loop0
12582912+0 records in
12582912+0 records out
6442450944 bytes (6.4 GB) copied, 705.782 s, 9.1 MB/s

real    11m45.844s
user    0m9.865s
sys    1m24.088s

Once the diskdump is done, we mount up the disk image, chroot to it and replace the kernel on it. One trick here is that, we will need to check the partition offset so that we will start from correct disk location. To check the existing offset, we will need to check back the fdisk output of /dev/xvde

[root@ip-10-195-81-211 ~]# fdisk -l /dev/xvde

Disk /dev/xvde: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xaae7682d

    Device Boot      Start         End      Blocks   Id  System
/dev/xvde1   *           1         783     6289416   83  Linux
So from above screenshot, we know that the partition is starting from first track (1) and each track size is 63*512. So our offset here is 1*63*512. We mount the partition to /clone.

[root@ip-10-195-81-211 ~]# mkdir /clone
[root@ip-10-195-81-211 ~]# mount -o loop,offset=$((63*512)) /target/diskimage.raw /clone

Compare / and /clone, they should be the same here. (it will be sightly different actually, as there are some run time file on /)

[root@ip-10-195-81-211 ~]# df -h / /clone
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvde1            6.0G 1010M  4.7G  18% /
/dev/loop1            6.0G 1001M  4.7G  18% /clone

As the original kernel of the VM instance is based on Xen paravirtualized kernel, we will have to install the generic system kernel or otherwise it wont be able to boot it our KVM environment. So, we chroot to the new virtual partition /clone and install the original kernel.

[root@ip-10-195-81-211 /]# yum -y install kernel kernel-devel


Once the kernel is installed. We may need to edit /boot/grub/grub.conf so that it will boot the correct kernel. In this example, I added below red line to define the new kernel (you will have to check the path of the new kernel though.

[root@ip-10-195-81-211 /]# cat /boot/grub/menu.lst
#===========
default=0
timeout=2
title CentOS (6.0 - /boot/vmlinuz-2.6.32-220.7.1.el6.x86_64)
    root (hd0,0)
    kernel /boot/vmlinuz-2.6.32-220.7.1.el6.x86_64 ro root=/dev/vda1
    initrd /boot/initramfs-2.6.32-220.7.1.el6.x86_64.img
title CentOS (6.0 - vmlinuz-2.6.32-131.17.1.el6.x86_64)
    root (hd0,0)
    kernel /boot/vmlinuz-2.6.32-131.17.1.el6.x86_64 ro root=/dev/xvde1
    initrd /boot/initramfs-2.6.32-131.17.1.el6.x86_64.img
#===========

Please be notice that the original drive on above was /dev/xvde1 while the new drive will be /dev/vda1 (if you are going to make use of paravirtualized driver on KVM hypervisor) or /dev/sda1 (if you are going to run it with generic driver).

Also we will need to replace /dev/xvde to /dev/vda on /etc/fstab

[root@ip-10-195-81-211 /]# grep "/dev/xvde" /etc/fstab
/dev/xvde1    /        ext4    defaults    1 1
[root@ip-10-195-81-211 /]# sed -i s%/dev/xvde%/dev/vda% /etc/fstab
[root@ip-10-195-81-211 /]# grep "/dev/vda" /etc/fstab
/dev/vda1    /        ext4    defaults    1 1

Once above preparation are done, we could quit from the chroot environment, unmount the virtual partition and detach the loopback device.

[root@ip-10-195-81-211 ~]# umount /clone
[root@ip-10-195-81-211 ~]# losetup -d /dev/loop0 

To load the diskimage on our standalone KVM server, we will have to scp / sftp the disk image from our EC2 VM to our on-premise server. As the image itself is a bit large, we may want to compress it with gzip or whatever compression tools before sending over.


Due to the fact that the image is a raw image, we will need to start the VM by defining raw disk option. The following screen-shots basically show what we need to do to facilitate this via virt-manager on kvm hypervisor node.

1. Launch a new node with existing disk image

2. Pointing to the path of the raw image. One thing have to be careful here is that, we will have to choose the proper OS Type (Linux) and Version (RHEL/CentOS 6) or otherwise it wont be able to present the device as para-virtualized device.

3. Define the memory and CPU core assigned to this VM, nothing special though.

4.  Proper Architecture have to be defined (x86_64 or i686)

Once the VM configured, we could start the VM however it will basically wont boot due to the GRUB failure. We will need to boot it via rescue CD / CentOS CD and issue below command to mount the volume and install grub.

# mkdir /vm; mount /dev/vda1 /vm
# mount –o bind /dev /vm/dev
# mount –o bind /sys /vm/sys
# chroot /vm

And then run grub-install and associated configuration.

# grub-install /dev/vda
# grub
grub> device (hd0) /dev/vda
device (hd0) /dev/vda
grub> root (hd0,0)
root (hd0,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd0)
 setup (hd0)
  Checking if "/boot/grub/stage1" exists... yes
  Checking if "/boot/grub/stage2" exists... yes
  Checking if "/boot/grub/e2fs_stage1_5" exists... yes
  Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 26 sectors are embedded. successed.
  Running "install /boot/grub/stage1 (hd0) (hd0)1+26 p (hd0,0)/boot/grub/stage2 /boot/grub/grub.conf "... succeeded
Done.
grub> quit
#

After above task, basically the VM is ready for reboot and it should be able to boot. However, before rebooting, you may also want to reset the root password so that you could get into the VM once it is rebooted.

Allow Linux VM running on KVM paravirtualized environment to support disk hotplugging

In case you have a Linux OS VM running on KVM paravirtualized environment and found that it doesn't response to the new virtual disk you added (you dont see a new disk on dmesg or nothing coming up from fdisk -l), you will need to make sure virtio_blk driver is in place.


[root@vm ~]# lsmod | grep -i virtio
virtio_net             15665  0
virtio_balloon          4281  0
virtio_blk              5087  3
virtio_pci              6733  0
virtio_ring             7169  4 virtio_net,virtio_balloon,virtio_blk,virtio_pci
virtio                  4824  4 virtio_net,virtio_balloon,virtio_blk,virtio_pci


Indeed, you should be seeing /dev/vd* instead of /dev/sd* if you have picked correct OS type during VM initialization. In case you are seeing /dev/sd*, probably you didnt pick correct OS type and that caused virtio associated driver not being loaded.

[root@vm ~]# fdisk -l /dev/vda

Disk /dev/vda: 21.5 GB, 21474836480 bytes
16 heads, 63 sectors/track, 41610 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006fe0a

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3         409      204800   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2             409       41611    20765696   8e  Linux LVM
Partition 2 does not end on cylinder boundary.

Garbage / Missing fonts in RHEL Virt-manager

Virt-Manager is a X application which is capable to manage RHEL's XEN, QEMU/KVM, LXC service. Similar to other X application, it could be launched remotely via X11 Forwarding.

However, it happens that garbage fonts will be shown if the host is not installed with appropriate font package. To fix this, package dejavu-lgc-sans-fonts have to be installed.

[root@server ~]# yum -y install dejavu-lgc-sans-fonts

Install QEMU-KVM in RHEL / CentOS

Here is the command to install QEMU-KVM packages in RHEL / CentOS, 

yum -y install qemu-kvm qemu-kvm-tools libvirt

or these command will install the packages for associated virtualization stuff.

yum -y groupinstall "Virtualization"
yum -y groupinstall "Virtualization Client"
yum -y groupinstall "Virtualization Platform"
yum -y groupinstall "Virtualization Tools"