A quick and dirty munin plugins to count number of VM running on RHEL/CentOS based KVM host.

So recently I was configuring munin to monitor some QEMU/KVM hosts which based on generic RHEL servers (Noted, not RHEV) which run libvirtd and QEMU/KVM.

So here is a plugins that I created, it is quick and dirty but this should work as expected. Just copy and paste the plugins file into /etc/munin/plugins/ directory and make sure it is executable (755, ideally), then you should be good.

So here is the content of the file.

[root@localhost plugins]# cat /etc/munin/plugins/vm_count
#!/bin/sh

case $1 in
   config)
        cat <<'EOM'
graph_title Number of VMs
graph_vlabel VMcount
vmcount.label VMcount
vmcount.graph_category Vserver
EOM
        exit 0;;
esac

i=`ps auxww | grep [/]usr/libexec/qemu-kvm | wc -l`
echo -n "vmcount.value "
echo $i

And it is how it would work.

# You should be able to execute it directly from system shell. In this example I had 17 VMs running on the host.

[root@localhost plugins]# pwd
/etc/munin/plugins
[root@localhost plugins]# ./vm_count
vmcount.value 17

# Alternatively, you can test it with munin-run. This is how the script will look like when it is being loaded

[root@localhost plugins]# munin-run vm_count
vmcount.value 17




# And here is the parameters of this plugins.

[root@localhost plugins]# munin-run vm_count  config
graph_title Number of VMs
graph_vlabel VMcount
vmcount.label VMcount
vmcount.graph_category Vserver

EX436: Add iptables rule to allow fence_xvmd

Assuming you want to make sure of fence_xvmd to do the VM fencing and you have iptables enabled, you may see issue while fence_xvm (client) send request to fence_xvmd (server). Here is an example,

Here is an example of fence_xvmd (server side)running on dom0 and the multicast address is on 225.0.0.12 (which is the default if option "-a" is not defined when you start fence_xvmd)

[root@dom0 images]# fence_xvmd -L -X -fd -I eth0
-- args @ 0x7fff92fb76d0 --
  args->addr = 225.0.0.12
  args->domain = (null)
  args->key_file = /etc/cluster/fence_xvm.key
  args->op = 2
  args->hash = 2
  args->auth = 2
  args->port = 1229
  args->ifindex = 5
  args->family = 2
  args->timeout = 30
  args->retr_time = 20
  args->flags = 259
  args->debug = 1
-- end args --
My Node ID = 1
Domain                   UUID                                 Owner State
------                   ----                                 ----- -----
Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
iscsitgt                 743affaf-eae7-6e40-0d1d-e3a3bb1b1eaf 00001 00002
lab1                     20a6e8b6-26a6-a700-b656-63b72b0a407e 00001 00002
lab2                     cb3f49a8-9841-d917-50ab-97425d900da4 00001 00002
Storing iscsitgt
Storing lab1
Storing lab2


So assuming you now fence the vm "lab1" from vm 'lab2" with fence_xvm (the client), you will be seeing something like this. Apparently, the fence_xvm request doesn't seem to connect to fence_xvmd (the fence server) and it keep complaining "Waiting for connection from XVM host daemon."

[root@lab2 ~]# fence_xvm -ddd -H lab1
Debugging threshold is now 3
-- args @ 0x7fffebce4540 --
  args->addr = 225.0.0.12
  args->domain = lab1
  args->key_file = /etc/cluster/fence_xvm.key
  args->op = 2
  args->hash = 2
  args->auth = 2
  args->port = 1229
  args->ifindex = 0
  args->family = 2
  args->timeout = 30
  args->retr_time = 20
  args->flags = 0
  args->debug = 3
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fffebce34f0 (4096 max size)
Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 192.168.0.202
Sending to 225.0.0.12 via 192.168.0.199
Sending to 225.0.0.12 via 10.0.0.202
Sending to 225.0.0.12 via 172.16.0.202
Sending to 225.0.0.12 via 172.16.1.202
Waiting for connection from XVM host daemon.
Sending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 192.168.0.202
Sending to 225.0.0.12 via 192.168.0.199
Sending to 225.0.0.12 via 10.0.0.202
Sending to 225.0.0.12 via 172.16.0.202
Sending to 225.0.0.12 via 172.16.1.202
Waiting for connection from XVM host daemon.

In my scenario, the xen host is enabled with iptables and looking at xvmd side, there is no fence request coming in too. That seems like the fence request was filtered.

To allow the fence request to get in via multicast traffic, we can add below rule to allow the traffic.

# iptables -I INPUT -d 225.0.0.12 -p udp -m udp --dport 1229 -j ACCEPT

Given that the fence_xvmd listen on default ip (225.0.0.12) and port (udp 1229).

Once the rule is added, you can retry fencing and now you would see something similar to this.

[root@lab2 ~]# fence_xvm -ddd -H lab1
Debugging threshold is now 3
-- args @ 0x7fffb74cc2a0 --
  args->addr = 225.0.0.12
  args->domain = lab1
  args->key_file = /etc/cluster/fence_xvm.key
  args->op = 2
  args->hash = 2
  args->auth = 2
  args->port = 1229
  args->ifindex = 0
  args->family = 2
  args->timeout = 30
  args->retr_time = 20
  args->flags = 0
  args->debug = 3
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fffb74cb250 (4096 max size)
Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 192.168.0.202
Sending to 225.0.0.12 via 192.168.0.199
Sending to 225.0.0.12 via 10.0.0.202
Sending to 225.0.0.12 via 172.16.0.202
Sending to 225.0.0.12 via 172.16.1.202
Waiting for connection from XVM host daemon.
Issuing TCP challenge
Responding to TCP challenge
TCP Exchange + Authentication done...
Waiting for return value from XVM host
Remote: Operation was successful

virtio_nic or e1000 on Linux KVM?

If you are RHEV / QEMU KVM user, you probably launched some virtual machines and yeah it is just as easy as couples of clicks. However, did you ever notice the reason why we have to fill in OS Type and Version ? And, actually VM still can boot up even you didnt fill in the exact OS type and Version for your VM.

The reason is that OS type and Version are used to define whether para-virtualized device will be used for the particular VM. Linux para-virtualized driver starts to run on kernel 2.6.25 or later and if you are running with a VM guest based on older kernel, chances that you will not be able to take advantages from it. For e.g. if you have a VM based on CentOS 4 (running with old 2.6.9 kernel) and you selected OS Type as CentOS 4, the VM will be configured to use simulated block device (/dev/sdX) and Intel e1000 emulated network card and these devices are actually not doing any better to paravirtualized device. However, if you start a VM by defining OS type Ubuntu 10.04 (i.e. kernel version 2.6.32 which does come with para-virtualized driver) and then you really load a Ubuntu 10.04 to it, it would recognize the para-virtualized device and make use of para-virtualized driver. In situation like this, your VM will be presented with para-virtualized block device (/dev/vdX) and virtio network card (virtio_nic) which would take full advantages of para-virtualized kernel.

Garbage / Missing fonts in RHEL Virt-manager

Virt-Manager is a X application which is capable to manage RHEL's XEN, QEMU/KVM, LXC service. Similar to other X application, it could be launched remotely via X11 Forwarding.

However, it happens that garbage fonts will be shown if the host is not installed with appropriate font package. To fix this, package dejavu-lgc-sans-fonts have to be installed.

[root@server ~]# yum -y install dejavu-lgc-sans-fonts