Getting UUID depedency failure when installing Vmware-Vsphere-CLI for Linux

Trying to install Vmware Vsphere CLI for Linux to allow some automation scripts to work and I am hitting below failure

++++++++++++++++++++++++++++++++++++++++++++++++++++++
CPAN is downloading and installing pre-requisite Perl module "UUID" .

CPAN not able to install following Perl modules on the system. These must be
installed manually for use by vSphere CLI:

UUID 0.03 or newer ++++++++++++++++++++++++++++++++++++++++++++++++++++++

So I try to manually install the UUID CPAN modules but I am hitting the same failure too.

++++++++++++++++++++++++++++++++++++++++++++++++++++++
Writing Makefile for UUID
Could not read metadata file. Falling back to other methods to determine prerequisites
cp UUID.pm blib/lib/UUID.pm
/usr/bin/perl /usr/share/perl5/ExtUtils/xsubpp  -typemap /usr/share/perl5/ExtUtils/typemap  UUID.xs > UUID.xsc && mv UUID.xsc UUID.c
gcc -c   -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic   -DVERSION=\"0.05\" -DXS_VERSION=\"0.05\" -fPIC "-I/usr/lib64/perl5/CORE"   UUID.c
UUID.xs:5:23: error: uuid/uuid.h: No such file or directory
UUID.xs: In function ‘do_generate’:
UUID.xs:13: error: ‘uuid_t’ undeclared (first use in this function)
UUID.xs:13: error: (Each undeclared identifier is reported only once
UUID.xs:13: error: for each function it appears in.)
UUID.xs:13: error: expected ‘;’ before ‘uuid’
UUID.xs:14: warning: implicit declaration of function ‘uuid_generate’
UUID.xs:14: error: ‘uuid’ undeclared (first use in this function)
UUID.xs: In function ‘do_unparse’:
UUID.xs:20: error: ‘uuid_t’ undeclared (first use in this function)
UUID.xs:20: error: expected ‘;’ before ‘uuid’
UUID.xs:23: warning: implicit declaration of function ‘uuid_unparse’
UUID.xs: In function ‘do_parse’:
UUID.xs:29: error: ‘uuid_t’ undeclared (first use in this function)
UUID.xs:29: error: expected ‘;’ before ‘uuid’
UUID.xs:33: warning: implicit declaration of function ‘uuid_parse’
UUID.xs:33: error: ‘uuid’ undeclared (first use in this function)
UUID.xs:30: warning: unused variable ‘str’
make: *** [UUID.o] Error 1
  LZAP/UUID-0.05.tar.gz
  /usr/bin/make -- NOT OK
Running make test
  Can't test without successful make
Running make install
  Make had returned bad status, install seems impossible
Failed during this command:
 LZAP/UUID-0.05.tar.gz                        : make NO
++++++++++++++++++++++++++++++++++++++++++++++++++++++

As I was on a CentOS 6 box, at first glance I was thinking it missed uuid-devel so I go ahead and install the package via yum but the same failure occurred during modules complication.


UUID.xs:5:23: error: uuid/uuid.h: No such file or directory

With deeper search, looks like uuid/uuid.h is on libuuid-devel package instead of uuid-devel (stupid me). So I go ahead to install that and now I could proceed with Vmware Vsphere CLI package install.

Could not load host key: /etc/ssh/ssh_host_rsa_key, SSH Connection closed by x.x.x.x

Before a ssh host to accept a ssh client request, the host have to be installed with SSH host key. Usually if you are installing the ssh servers (which usually come as default package in most Linux distro) from a package (e.g. rpm or deb), the installation script will take care the ssh host key generation for you.

However, in any cases you removed the ssh host key (usually sit under /etc/ssh/), you will no longer be able to get into the machine via ssh until the host key is generated.

So this is what you get when your ssh host is gone.

$ ssh root@x.x.x.x
Connection closed by x.x.x.x

And if you have a console access to your machine, you will be seeing something like this on /var/log/auth.log (or /var/log/messages ... depends on your syslog configuration).

Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 17 07:42:38 localhost sshd[26509]: fatal: No supported key exchange algorithms [preauth]

In order to fix this, ssh host key have to be regenerated

ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_rsa_key -N ‘’
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ‘’
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ‘’

The option -N refers to the new passphrase to be used. In above example '' means empty. Should you want a passphrase be assigned to the key, you may want to put it like -N 'your_pass_phrase'

Nginx X-Forwarded-Protocol and X-Forwarded-For

I have a client that had multiples apache web servers sit behind the Nginx web load balancer. Currently both http and https requests are terminated on Nginx and the requests will then be proxied to the backend apache web servers at port 80 (i.e. http).

From backend web servers perspective, all traffic coming in are sort of masqueraded by the Nginx , web server could only see the requests are made by Nginx and the protocol was http. So my client would interest to know which protocol the original request was, whether it is http or the ssl-encrypted https.

Nginx do allow customization on proxy header via proxy_set_header attributes. So I added below parameter to the location block so that extra header will be passed to the backend web server.


Here is the reverse proxy configuration
    upstream backend_web_server_pool {
       server 1.2.3.4:80;
       server 1.2.3.5:80;
    }

Here is the http site configuration

server {
    listen       80;  # The http server
    ....     
   location / {
       proxy_pass http://backend_web_server_pool;
       proxy_set_header X-Forwarded-Protocol "http" ;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}




Here is the https site configuration
server {
    listen       443;  # The https server
    ssl                  on;
    ....

    location / {
       proxy_pass http://backend_web_server_pool;
       proxy_set_header X-Forwarded-Protocol "https" ;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}


So the line proxy_set_header X-Forwarded-Protocol "http" will pass a header named "X-Forwarded-Protocol" and its value "http" to the backend web server. You can replace this header value to any arbitrary value, e.g. "xyz123". After all it is just a placeholder or symbol to let you know where the request came from. The same logic applies to the HTTPS block however you may want to replace the value from "http" to "https" to avoid confusion. The line proxy_set_header X-Forwarded-For $remote_addr pass the variable remote_addr (i.e. the remote client IP address) to the backend web server.

Once the above configuration applied, restart nginx and then we can head to reconfigure the log format configuration on apache web server. We will now modify the combined log format to capture the X-Forwarded-For and X-Forwarded-Protocol.

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %{X-Forwarded-For}i %{X-Forwarded-Protocol}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

I added the %{X-Forwarded-For}i and %{X-Forwarded-Protocol}i to the combined log format followed by apache restart and now apache log could capture the client IP address and the original protocol.

dnsmasq, assign duplicate DHCP subnet via single interface

I was troubleshooting a dnsmasq DHCP server back in few days and notice special settings have to be applied if the particular dnsmasq DHCP server is configured to serve multiple subnet.

In my scenario, the DHCP server is having one nic, eth0. For some reason, there are 2 subnets being served, 192.168.0.0/24 and 192.168.1.0/24. The eth0 is configured as 192.168.0.1/24 while an additional ip 192.168.1.1 is added to eth0 too.

So DHCP clients that connect to the eth0 of DHCP server (through switches) could retrieve IP from same subnet without any issue, however it seems like the DHCP gateway of the 2nd subnet is acting weird. For example, here is the DHCP lease file i got from one of the DHCP client.

# cat /var/lib/dhcp/pump.lease
Device eth0
    IP: 192.168.1.10
    Netmask: 255.255.255.0
    Broadcast: 192.168.1.255
    Network: 192.168.1.0
    Boot server 192.168.1.1
    Next server 192.168.1.1
    Gateways: 192.168.0.1
    Hostname: test-dhcp-client
    Domain: test.internal
    Renewal time: Fri Nov 23 17:38:13 2012
    Expiration time: Fri Nov 23 19:08:13 2012

Interestingly, this DHCP client is assigned with the 2nd DHCP subnet (i.e. 192.168.1.0/24) however gateway of default subnet (192.168.0.0/24) is being assigned.

To fix this, I tried couples of approach but eventually it looks like configuring the option though dhcp-range tagging would do the best. Below are the configuration snippet that fixed the problem.

listen-address=192.168.0.1
listen-address=192.168.1.1
dhcp-range=set:1stblock,192.168.1.10,192.168.0.50,255.255.255.0
dhcp-range=set:2ndblock,192.168.1.1,192.168.1.50,255.255.255.0
dhcp-option=tag:1stblock,option:router,192.168.0.1
dhcp-option=tag:2ndblock,option:router,192.168.1.1

So in above example, I assign a tag to each individual subnet (i.e 1stblock -> 192.168.0.0/24, 2ndblock -> 192.168.1.0/24) and then assign individual router ip with each associated tag.

QEMU/KVM atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)

So I am getting this annoying warning message when I press Enter on my KVM VM.

# 
atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)
atkbd.c Use 'setkeycodes 00 <keycode>' to make it known.

So now whenever I hit enter, the annoying warning will be coming up and this can be easily cleaned up by using the showkey and setkeycodes command.

So from the tty console (it may not work on x-windows), i executed showkey

# showkey
kb mode was UNICODE
[ if you are trying this under X, it might not work
since the X server is also reading /dev/console ]

press any key (program terminates 10s after last keypress)...

Now you can press 'Enter' (or whatever key that could cause the annoying message), it will show the keycode

keycode  28 press

It means that the Enter key is associated to keycode 28. Now you can use setkeycodes to map the Enter key.

# setkeycodes 0x00 28

0x00 here is the scancode, if you are interested on what it means, you could google it and there is some other articles out there explaining this.

Now, right after you typed the command, you should now be safed from this annoying warning message.

Should you want this be sustained across reboot, you may want to add it to /etc/rc.local

# echo 'setykeycodes 0x00 28' >> /etc/rc.local

Using gawk (GNU awk) to monitor /var/log/messages

I have a requirement to periodically scan through /var/log/messages to catch specific error message. The error itself is very time-sensitive and I want to be informed as soon as the message is shown on the log, ideally in a minute interval.

Usually people would suggest to periodically run grep against the log which is the simplest way to facilitate the need but it doesn't work for my scenario. The problem to simply run grep is that it may also capture unnecessary information. For example, I have below logs on my /var/log/messages.

Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal

If I simply use grep to catch above log entry in a minute interval, I will probably be informed at 12:40.06 and then all the way till the end of the day (i.e. when the /var/log/messages be rotated and cleaned up). Not to mention starting from 15:05:28, the grep script will catch 2 errors from the log, the lines being recorded at 12:40:05 and 15:05:28. However, the fact is that I actually need the one on 15:05:28.

After some research, it looks like gawk (GNU awk) would be the perfect tool to solve the problem. Eventually I come up with below gawk script to read /var/log/messages.


$ cat gawk-script
BEGIN {
    # Declare field seperator.
    FS="[- :.]";

    
    # Generate the timestamp
    NOW=systime();
    
    # The time interval that I want this script to read from
    # As long as I will run this script with cron in every minutes
    # I set the time interval to 1sec*60
    # So, for example when the script is run at 12:05, only log entries between 12:04 ~ 12:05 will be read.
    PAST=NOW-(60);

    
    # /var/log/messages start with the line in format of "Month date" (e.g Mar 23)
    # Ideally we can convert it to something like 23/3 but I want to be lazy, 
    # so I simply make use of the system time library and 
    # picked %Y %m to represent the Year and Month attributes.
    format="%Y %m";
    # LOGMTH will be something like "2012 10" (Oct 2012)
    # This will be used later to generate the timestamp of the log entry.
    LOGMTH=strftime(format, THEN)
}
{
    # Read the line of the log and convert it to a timestamp
    LOGTIME=mktime(LOGMTH " " $2 " " $3 " " $4 " " $5);
    # Below 3 lines can be commented out to debug with the value being read.
    #{print $2, $3, $4, $5};
    #{print LOGMTH};
    #{print LOGTIME};
    
    # print the log if the timestamp of the line is between PAST <> NOW.
    if(PAST<LOGTIME){print}
}

So here is a demo of the script.

$ cat /var/log/messages  ### totally 4 lines here
Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal Aug 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28
Aug 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices

$ date  #Lets check the current time
Wed Oct 3 18:04:24 HKT 2012

$ gawk -f gawk-script /var/log/messages
Oct 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28
Oct 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices

So, the script now only print everything captured within last minute. You can then use the script together with grep utility to catch the string you are interested.

Natting TCP port 2000 behind Cisco device.

Recently I am mudding on a storage appliance named NexentaStor. NexentaStor is based on OpenSolaris and make use of ZFS implementation which looks pretty promising. It has a clean and easy to use GUI, hence it supports quite a lot of storage protocol like NFS, CIFS, ISCSI and even support Link aggregation on network layer too.

Everything is going smooth so far just one minor obstacles, that is its Web GUI by default listen on tcp port 2000. Basically tcp port 2000 is a valid port but somehow I am not able to access the Web GUI and the connection towards the port keep timing out from outside, though the port is working on the same subnet.

I started to suspect there is something to do with NAT and yeah it is. I put my NexentaStor server behind a Cisco ASA firewall with NAT enabled. However, it looks like the port 2000 traffic of NexentaStor overlapped with the Cisco SCCP (http://en.wikipedia.org/wiki/Skinny_Call_Control_Protocol) on port 2000 too. Eventually I have to change the port of the GUI to a non-2000 port.

Just an additional notes, to reconfigure NexentaStor Web GUI port, I have to get into the console and execute below command.

nmc@myhost:/$ setup appliance init