Openstack nova Instance failed to push static IP through config drive. openstack/content/0000 is missing

So I need to bootstrap an instance that is configured as static IP through config drive, instead of the generic DHCP.  So with nova boot, everything seems to be happy but the instance just failed to get the static IP for some reason. Doing a nova console-log, the IP address shows like this in cloud-init.

cloud-init[775]: Cloud-init v. 0.7.5 running 'init' at Fri, 11 Sep 2015 17:56:46 +0000. Up 29.44 seconds.
cloud-init[775]: ci-info: +++++++++++++++++++++++Net device info+++++++++++++++++++++++
cloud-init[775]: ci-info: +--------+------+-----------+-----------+-------------------+
cloud-init[775]: ci-info: | Device |  Up  |  Address  |    Mask   |     Hw-Address    |
cloud-init[775]: ci-info: +--------+------+-----------+-----------+-------------------+
cloud-init[775]: ci-info: |  lo:   | True | 127.0.0.1 | 255.0.0.0 |         .         |
cloud-init[775]: ci-info: | eth0:  | True |     .     |     .     | fa:16:3e:4d:35:34 |
cloud-init[775]: ci-info: +--------+------+-----------+-----------+-------------------+

Obviously the eth0 is not getting the static IP. Once again I mounted the config-drive and then try to find the /openstack/content/0000 file but I can't find it.

[root@nova ~]# find /mnt/openstack | grep content
[root@nova ~]#

This is pretty weird as I remember the file should be there to allow static ip assignment to work through config-drive. So taking further looks, it seems the problem go to the subnet configuration. The subnet that the instance is on is having "enable_dhcp" equal to true, and that prohibited the config-drive to create the openstack/content/0000 file.

To disable DHCP for a subnet, run this.

# neutron  subnet-update  $SUBNET_UUID --enable-dhcp=False

Openstack broken metadata interfaces template

Was my 2nd time hitting the same issue and it wasted my whole day so I think I should document this as a note just in case it happens again.

In case you are seeing something like this from your metadata (be it metadata server, or config-drive), in this example I mounted the config drive as /mnt.

# cat /mnt/openstack/content/0000
DEVICE="{{ name }}"
NM_CONTROLLED="no"
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=static
IPADDR={{ address }}
NETMASK={{ netmask }}
BROADCAST={{ broadcast }}
GATEWAY={{ gateway }}
DNS1={{ dns }}

#if $use_ipv6
IPV6INIT=yes
IPV6ADDR={{ address_v6 }}
#end if

Chances that you are hitting this bug.

As cloud-init doesnt really recognize format like that, to fix the issue you will need to update the template /usr/share/nova/interfaces.template (assuming you are on CentOS/RHEL7) with something like this which is a debian-ish template

# Injected by Nova on instance boot
#
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
{% for ifc in interfaces -%}
auto {{ ifc.name }}
iface {{ ifc.name }} inet static
address {{ ifc.address }}
netmask {{ ifc.netmask }}
broadcast {{ ifc.broadcast }}
{%- if ifc.gateway %}
gateway {{ ifc.gateway }}
{%- endif %}
{%- if ifc.dns %}
dns-nameservers {{ ifc.dns }}
{%- endif %}
{% if use_ipv6 -%}
iface {{ ifc.name }} inet6 static
address {{ ifc.address_v6 }}
netmask {{ ifc.netmask_v6 }}
{%- if ifc.gateway_v6 %}
gateway {{ ifc.gateway_v6 }}
{%- endif %}
{%- endif %}
{%- endfor %}

Cloudera Manager agent failed to connect to previous supervisor

Continue to another article written earlier, I am hitting another road block while installing Cloudera manager agent.

[10/Sep/2015 09:10:54 +0000] 19017 MainThread agent        ERROR    Failed to connect to previous supervisor.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1524, in find_or_start_supervisor
    self.get_supervisor_process_info()
  File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1725, in get_supervisor_process_info
    self.identifier = self.supervisor_client.supervisor.getIdentification()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/xmlrpc.py", line 460, in request
    self.connection.request('POST', handler, request_body, self.headers)
  File "/usr/lib64/python2.6/httplib.py", line 914, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
    self.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 720, in connect
    self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
error: [Errno 111] Connection refused

So if you are seeing something like this which complain failure of Cloudera manager agent to provide the heartbeat and such, your probably run into a hostname issue. You may want to fix the hostname entry by referencing this link Check the hostname on the server and compare it with the one shown in the installer web GUI, if they are different then you probably want to follow below procedure to refresh the cached hostname (ref: link). It took me like an hour to figure out this painful workaround.

Installing on AWS, you must use private EC2 hostnames.
When installing on an AWS instance, and adding hosts using their public names, the installation will fail when the hosts fail to heartbeat.

Severity: Med

Workaround:

Use the Back button in the wizard to return to the original screen, where it prompts for a license.

Rerun the wizard, but choose "Use existing hosts" instead of searching for hosts. Now those hosts show up with their internal EC2 names.

Continue through the wizard and the installation should succeed.

Cloudera manager agent installation fail due to missing of ntp package

So was trying to explore Cloudera Hadoop by following this installation guide and run into issue in bringing up cloudera manager agent (cloudera-scm-agent).

The installation GUI was complaining with below message

  Installation failed. Failed to receive heartbeat from agent.

Ensure that the host's hostname is configured properly.
Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
Ensure that ports 9000 and 9001 are free on the host being added.
Check agent logs in /var/log/cloudera-scm-agent/ on the host being added (some of the logs can be found in the installation details).

So looking at /var/log/cloudera-scm-agent/cloudera-scm-agent.log

1 2 3 4 5 6 7 8 9 10 11 12 13

[10/Sep/2015 07:57:16 +0000] 2366 Monitor-HostMonitor throttling_logger ERROR Failed to collect NTP metrics Traceback (most recent call last): File "/usr/lib64/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 37, in collect result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) File "/usr/lib64/cmf/agent/src/cmf/monitor/host/ntp_monitor.py", line 30, in _subprocess_with_timeout return subprocess_with_timeout(args, timeout) File "/usr/lib64/cmf/agent/src/cmf/subprocess_timeout.py", line 49, in subprocess_with_timeout p = subprocess.Popen(**kwargs) File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__ errread, errwrite) File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory

So looking at /usr/lib64/cmf/agent/src/cmf/monitor/host/ntp_monitor.py, the culprit is there

     35     try:
     36       args = ["ntpdc", "-np"]
     37       result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout)

As a quick fix, do a yum install ntp should help getting rid of this error.

Passed EX436 Clustering and Storage Management last sunday.

For sake of self achievement, I planned to get RHCA for some time since 2011 and hopefully this could be done before end of 2013, if everything go smooth.

Last sunday, I just finished EX436 which is my 2nd RHCA exam out of a series of 5. The score was 268 out of 300 which is pretty good enough for me. The major focus of EX436 is clustering and storage management which is an area that I have some experience on but would definitely love to improve. The exam itself is pretty interesting, challenging and fun. Like other famous RHCA blogger out there,  I have the comment that the exam is not really tough given that you are well prepared with the topics.

For those that would like to take this exam, no doubt the best bet would certainly be joining the training class provided by Redhat :-) . But for those that want to save some bucks, you gotta work on your own and a good start would be to revisit the Course Outline here (link). For me, I basically stick with the official Redhat guide of clustering, GFS/GFS2, Multipath, Fencing, LVM and CLVM(all can be found here) and keep practice on my own lab. And, base on latest outline, you may also want to check with XFS and Gluster ( I am not sure how can you find resources from redhat site though). For me, I didn't practise XFS and Gluster in my own lab as I was sticking with previous Course Outline which didn't include XFS and Gluster at all (!!!). So it is a big surprise when I saw those questions in the exam. However, due to my job duties, I did have few exposure on them and luckily those exposure helped me to survive in the exam.
 
People may interest to know the distro or exact version in question, but due to N.D.A agreement I can't say which version it is here :-) . What I would say is , clustering, GFS/GFS2 don't have a major different on RHEL5/6, at least from exam perspective. For me, my lab was based on RHEL 5 and i didn't subscribe to any Redhat subscription service (I copied all rpm based off the ISO and create my own repository tree as well as yum repository configuration files to allow the lab machine to fetch the required packages).

My another advise is to stay calm during the exam. During my exam, I was stucked on a particular task and was going a bit nervous, this lead me mistakenly reboot my host (!!) at the middle of the exam. For those that already took RHCSA/RHCE exam, you may know that your exam system is on a VM that sit on a physical machines dedicated to you. So in my situation I was rebooting my host that caused all my VM suspended and then at that point I didn't know what could happen. You know, the worse case would be a re-image of all exam VM and I have to rebuild everything in the remaining 2 hours out of 4 hours exam duration.  Luckily all my VMs are still there after the host reboot and the only impact was a 15 minutes downtime on my exam environment (million thanks to the examiner who helped recovering my environment though).

The upcoming exam for me would be

EX442    Red Hat Enterprise System Monitoring and Performance Tuning Expertise Exam
EX333    Red Hat Enterprise Security: Network Services Expertise Exam
EX401    Red Hat Enterprise Deployment and Systems Management Expertise Exam

Hopefully I would be taking EX442 on Sept if everything go smooth.  EX442 was well known among RHCA-er for its complexity so I would look forward to give a try on it.

A quick and dirty munin plugins to count number of VM running on RHEL/CentOS based KVM host.

So recently I was configuring munin to monitor some QEMU/KVM hosts which based on generic RHEL servers (Noted, not RHEV) which run libvirtd and QEMU/KVM.

So here is a plugins that I created, it is quick and dirty but this should work as expected. Just copy and paste the plugins file into /etc/munin/plugins/ directory and make sure it is executable (755, ideally), then you should be good.

So here is the content of the file.

[root@localhost plugins]# cat /etc/munin/plugins/vm_count
#!/bin/sh

case $1 in
   config)
        cat <<'EOM'
graph_title Number of VMs
graph_vlabel VMcount
vmcount.label VMcount
vmcount.graph_category Vserver
EOM
        exit 0;;
esac

i=`ps auxww | grep [/]usr/libexec/qemu-kvm | wc -l`
echo -n "vmcount.value "
echo $i

And it is how it would work.

# You should be able to execute it directly from system shell. In this example I had 17 VMs running on the host.

[root@localhost plugins]# pwd
/etc/munin/plugins
[root@localhost plugins]# ./vm_count
vmcount.value 17

# Alternatively, you can test it with munin-run. This is how the script will look like when it is being loaded

[root@localhost plugins]# munin-run vm_count
vmcount.value 17




# And here is the parameters of this plugins.

[root@localhost plugins]# munin-run vm_count  config
graph_title Number of VMs
graph_vlabel VMcount
vmcount.label VMcount
vmcount.graph_category Vserver

Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N

So,  because of my job duty and I have to deal with Citrix Cloudstack day by day. Recently we are deploying a new advanced zone and for some reason we are seeing errors like this during deploy of our first VM instance.


2013-07-22 22:09:27,625 WARN  [api.commands.DeployVMCmd] (Job-Executor-50:job-534828) Exception:
com.cloud.exception.AgentUnavailableException: Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N
................
Caused by: com.cloud.utils.exception.CloudRuntimeException: Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N
................
2013-07-22 22:09:27,626 WARN  [cloud.api.ApiDispatcher] (Job-Executor-50:job-534828) class com.cloud.api.ServerApiException : Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N

So, basically, what Cloudstack doing is to
1. check if there is any valid systemvm template (in this case systemvm-kvm-3.0.0) deployed to the zone.
2. If things works as it should, you should be able to find the installed/downloaded template from table cloud.vm_template, cloud.template_zone_ref and template_host_ref. Hence, if you scan through the template list from the Web GUI, you should be able to see the template be downloaded.

In my case, the template was not downloaded as it should (or marked as downloaded at DB layer), and if you look at the table cloud.template_host_ref, there is some abnormality here.

mysql> select * from  template_host_ref where id=11111\G
*************************** 1. row ***************************
            id: 11111
       host_id: *masked*
   template_id: *masked*
       created: 2013-07-18 17:50:43
  last_updated: 2013-07-22 20:04:52
        job_id: 75a75e55-5280-4ba5-b823-cadbcbe2cc7a
  download_pct: 0
          size: 0
 physical_size: 0
download_state: DOWNLOAD_ERROR
     error_str: No route to host
    local_path: /mnt/SecStorage/04ab8f0b-c4e0-34a4-80b3-457c433acde3/template/tmpl/2/1686/dnld6951269530983090325tmp_
  install_path: NULL
           url: http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2
     destroyed: 0
       is_copy: 0

So, basically the things are 1) download_pct is 0 (while it should be 100 if download succeed), 2) download_state is DOWNLOAD_ERROR (while it should be DOWNLOADED if download successed and 3) error_str is "No route to host".

In my case, the template installation procedures was not completed (though I have completed the cloud-install-sys-tmplt script per official installation guide), at least at DB layer.

So I double checked the secondary storage to make sure the template file is completely downloaded (IMPORTANT!!!, if the file is not there, go through installation guide and re-run cloud-install-sys-tmplt script) and hacked the DB by updating the cloud.template_host_ref table. (Replace "N" with the correct account id and template id respectively)

mysql> updated template_host_ref set download_pct=100, download_state='DOWNLOADED', error_str=NULL, localpath='template/tmpl/N/N' where id=11111\G
*************************** 1. row ***************************
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Now cloudstack could launch VM as it should.

EX436: Add iptables rule to allow fence_xvmd

Assuming you want to make sure of fence_xvmd to do the VM fencing and you have iptables enabled, you may see issue while fence_xvm (client) send request to fence_xvmd (server). Here is an example,

Here is an example of fence_xvmd (server side)running on dom0 and the multicast address is on 225.0.0.12 (which is the default if option "-a" is not defined when you start fence_xvmd)

[root@dom0 images]# fence_xvmd -L -X -fd -I eth0
-- args @ 0x7fff92fb76d0 --
  args->addr = 225.0.0.12
  args->domain = (null)
  args->key_file = /etc/cluster/fence_xvm.key
  args->op = 2
  args->hash = 2
  args->auth = 2
  args->port = 1229
  args->ifindex = 5
  args->family = 2
  args->timeout = 30
  args->retr_time = 20
  args->flags = 259
  args->debug = 1
-- end args --
My Node ID = 1
Domain                   UUID                                 Owner State
------                   ----                                 ----- -----
Domain-0                 00000000-0000-0000-0000-000000000000 00001 00001
iscsitgt                 743affaf-eae7-6e40-0d1d-e3a3bb1b1eaf 00001 00002
lab1                     20a6e8b6-26a6-a700-b656-63b72b0a407e 00001 00002
lab2                     cb3f49a8-9841-d917-50ab-97425d900da4 00001 00002
Storing iscsitgt
Storing lab1
Storing lab2


So assuming you now fence the vm "lab1" from vm 'lab2" with fence_xvm (the client), you will be seeing something like this. Apparently, the fence_xvm request doesn't seem to connect to fence_xvmd (the fence server) and it keep complaining "Waiting for connection from XVM host daemon."

[root@lab2 ~]# fence_xvm -ddd -H lab1
Debugging threshold is now 3
-- args @ 0x7fffebce4540 --
  args->addr = 225.0.0.12
  args->domain = lab1
  args->key_file = /etc/cluster/fence_xvm.key
  args->op = 2
  args->hash = 2
  args->auth = 2
  args->port = 1229
  args->ifindex = 0
  args->family = 2
  args->timeout = 30
  args->retr_time = 20
  args->flags = 0
  args->debug = 3
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fffebce34f0 (4096 max size)
Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 192.168.0.202
Sending to 225.0.0.12 via 192.168.0.199
Sending to 225.0.0.12 via 10.0.0.202
Sending to 225.0.0.12 via 172.16.0.202
Sending to 225.0.0.12 via 172.16.1.202
Waiting for connection from XVM host daemon.
Sending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 192.168.0.202
Sending to 225.0.0.12 via 192.168.0.199
Sending to 225.0.0.12 via 10.0.0.202
Sending to 225.0.0.12 via 172.16.0.202
Sending to 225.0.0.12 via 172.16.1.202
Waiting for connection from XVM host daemon.

In my scenario, the xen host is enabled with iptables and looking at xvmd side, there is no fence request coming in too. That seems like the fence request was filtered.

To allow the fence request to get in via multicast traffic, we can add below rule to allow the traffic.

# iptables -I INPUT -d 225.0.0.12 -p udp -m udp --dport 1229 -j ACCEPT

Given that the fence_xvmd listen on default ip (225.0.0.12) and port (udp 1229).

Once the rule is added, you can retry fencing and now you would see something similar to this.

[root@lab2 ~]# fence_xvm -ddd -H lab1
Debugging threshold is now 3
-- args @ 0x7fffb74cc2a0 --
  args->addr = 225.0.0.12
  args->domain = lab1
  args->key_file = /etc/cluster/fence_xvm.key
  args->op = 2
  args->hash = 2
  args->auth = 2
  args->port = 1229
  args->ifindex = 0
  args->family = 2
  args->timeout = 30
  args->retr_time = 20
  args->flags = 0
  args->debug = 3
-- end args --
Reading in key file /etc/cluster/fence_xvm.key into 0x7fffb74cb250 (4096 max size)
Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1
Sending to 225.0.0.12 via 192.168.0.202
Sending to 225.0.0.12 via 192.168.0.199
Sending to 225.0.0.12 via 10.0.0.202
Sending to 225.0.0.12 via 172.16.0.202
Sending to 225.0.0.12 via 172.16.1.202
Waiting for connection from XVM host daemon.
Issuing TCP challenge
Responding to TCP challenge
TCP Exchange + Authentication done...
Waiting for return value from XVM host
Remote: Operation was successful

Getting UUID depedency failure when installing Vmware-Vsphere-CLI for Linux

Trying to install Vmware Vsphere CLI for Linux to allow some automation scripts to work and I am hitting below failure

++++++++++++++++++++++++++++++++++++++++++++++++++++++
CPAN is downloading and installing pre-requisite Perl module "UUID" .

CPAN not able to install following Perl modules on the system. These must be
installed manually for use by vSphere CLI:

UUID 0.03 or newer ++++++++++++++++++++++++++++++++++++++++++++++++++++++

So I try to manually install the UUID CPAN modules but I am hitting the same failure too.

++++++++++++++++++++++++++++++++++++++++++++++++++++++
Writing Makefile for UUID
Could not read metadata file. Falling back to other methods to determine prerequisites
cp UUID.pm blib/lib/UUID.pm
/usr/bin/perl /usr/share/perl5/ExtUtils/xsubpp  -typemap /usr/share/perl5/ExtUtils/typemap  UUID.xs > UUID.xsc && mv UUID.xsc UUID.c
gcc -c   -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic   -DVERSION=\"0.05\" -DXS_VERSION=\"0.05\" -fPIC "-I/usr/lib64/perl5/CORE"   UUID.c
UUID.xs:5:23: error: uuid/uuid.h: No such file or directory
UUID.xs: In function ‘do_generate’:
UUID.xs:13: error: ‘uuid_t’ undeclared (first use in this function)
UUID.xs:13: error: (Each undeclared identifier is reported only once
UUID.xs:13: error: for each function it appears in.)
UUID.xs:13: error: expected ‘;’ before ‘uuid’
UUID.xs:14: warning: implicit declaration of function ‘uuid_generate’
UUID.xs:14: error: ‘uuid’ undeclared (first use in this function)
UUID.xs: In function ‘do_unparse’:
UUID.xs:20: error: ‘uuid_t’ undeclared (first use in this function)
UUID.xs:20: error: expected ‘;’ before ‘uuid’
UUID.xs:23: warning: implicit declaration of function ‘uuid_unparse’
UUID.xs: In function ‘do_parse’:
UUID.xs:29: error: ‘uuid_t’ undeclared (first use in this function)
UUID.xs:29: error: expected ‘;’ before ‘uuid’
UUID.xs:33: warning: implicit declaration of function ‘uuid_parse’
UUID.xs:33: error: ‘uuid’ undeclared (first use in this function)
UUID.xs:30: warning: unused variable ‘str’
make: *** [UUID.o] Error 1
  LZAP/UUID-0.05.tar.gz
  /usr/bin/make -- NOT OK
Running make test
  Can't test without successful make
Running make install
  Make had returned bad status, install seems impossible
Failed during this command:
 LZAP/UUID-0.05.tar.gz                        : make NO
++++++++++++++++++++++++++++++++++++++++++++++++++++++

As I was on a CentOS 6 box, at first glance I was thinking it missed uuid-devel so I go ahead and install the package via yum but the same failure occurred during modules complication.


UUID.xs:5:23: error: uuid/uuid.h: No such file or directory

With deeper search, looks like uuid/uuid.h is on libuuid-devel package instead of uuid-devel (stupid me). So I go ahead to install that and now I could proceed with Vmware Vsphere CLI package install.

Could not load host key: /etc/ssh/ssh_host_rsa_key, SSH Connection closed by x.x.x.x

Before a ssh host to accept a ssh client request, the host have to be installed with SSH host key. Usually if you are installing the ssh servers (which usually come as default package in most Linux distro) from a package (e.g. rpm or deb), the installation script will take care the ssh host key generation for you.

However, in any cases you removed the ssh host key (usually sit under /etc/ssh/), you will no longer be able to get into the machine via ssh until the host key is generated.

So this is what you get when your ssh host is gone.

$ ssh root@x.x.x.x
Connection closed by x.x.x.x

And if you have a console access to your machine, you will be seeing something like this on /var/log/auth.log (or /var/log/messages ... depends on your syslog configuration).

Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jan 17 07:42:38 localhost sshd[26509]: fatal: No supported key exchange algorithms [preauth]

In order to fix this, ssh host key have to be regenerated

ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_rsa_key -N ‘’
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ‘’
ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ‘’

The option -N refers to the new passphrase to be used. In above example '' means empty. Should you want a passphrase be assigned to the key, you may want to put it like -N 'your_pass_phrase'

Nginx X-Forwarded-Protocol and X-Forwarded-For

I have a client that had multiples apache web servers sit behind the Nginx web load balancer. Currently both http and https requests are terminated on Nginx and the requests will then be proxied to the backend apache web servers at port 80 (i.e. http).

From backend web servers perspective, all traffic coming in are sort of masqueraded by the Nginx , web server could only see the requests are made by Nginx and the protocol was http. So my client would interest to know which protocol the original request was, whether it is http or the ssl-encrypted https.

Nginx do allow customization on proxy header via proxy_set_header attributes. So I added below parameter to the location block so that extra header will be passed to the backend web server.


Here is the reverse proxy configuration
    upstream backend_web_server_pool {
       server 1.2.3.4:80;
       server 1.2.3.5:80;
    }

Here is the http site configuration

server {
    listen       80;  # The http server
    ....     
   location / {
       proxy_pass http://backend_web_server_pool;
       proxy_set_header X-Forwarded-Protocol "http" ;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}




Here is the https site configuration
server {
    listen       443;  # The https server
    ssl                  on;
    ....

    location / {
       proxy_pass http://backend_web_server_pool;
       proxy_set_header X-Forwarded-Protocol "https" ;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}


So the line proxy_set_header X-Forwarded-Protocol "http" will pass a header named "X-Forwarded-Protocol" and its value "http" to the backend web server. You can replace this header value to any arbitrary value, e.g. "xyz123". After all it is just a placeholder or symbol to let you know where the request came from. The same logic applies to the HTTPS block however you may want to replace the value from "http" to "https" to avoid confusion. The line proxy_set_header X-Forwarded-For $remote_addr pass the variable remote_addr (i.e. the remote client IP address) to the backend web server.

Once the above configuration applied, restart nginx and then we can head to reconfigure the log format configuration on apache web server. We will now modify the combined log format to capture the X-Forwarded-For and X-Forwarded-Protocol.

#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %{X-Forwarded-For}i %{X-Forwarded-Protocol}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

I added the %{X-Forwarded-For}i and %{X-Forwarded-Protocol}i to the combined log format followed by apache restart and now apache log could capture the client IP address and the original protocol.

dnsmasq, assign duplicate DHCP subnet via single interface

I was troubleshooting a dnsmasq DHCP server back in few days and notice special settings have to be applied if the particular dnsmasq DHCP server is configured to serve multiple subnet.

In my scenario, the DHCP server is having one nic, eth0. For some reason, there are 2 subnets being served, 192.168.0.0/24 and 192.168.1.0/24. The eth0 is configured as 192.168.0.1/24 while an additional ip 192.168.1.1 is added to eth0 too.

So DHCP clients that connect to the eth0 of DHCP server (through switches) could retrieve IP from same subnet without any issue, however it seems like the DHCP gateway of the 2nd subnet is acting weird. For example, here is the DHCP lease file i got from one of the DHCP client.

# cat /var/lib/dhcp/pump.lease
Device eth0
    IP: 192.168.1.10
    Netmask: 255.255.255.0
    Broadcast: 192.168.1.255
    Network: 192.168.1.0
    Boot server 192.168.1.1
    Next server 192.168.1.1
    Gateways: 192.168.0.1
    Hostname: test-dhcp-client
    Domain: test.internal
    Renewal time: Fri Nov 23 17:38:13 2012
    Expiration time: Fri Nov 23 19:08:13 2012

Interestingly, this DHCP client is assigned with the 2nd DHCP subnet (i.e. 192.168.1.0/24) however gateway of default subnet (192.168.0.0/24) is being assigned.

To fix this, I tried couples of approach but eventually it looks like configuring the option though dhcp-range tagging would do the best. Below are the configuration snippet that fixed the problem.

listen-address=192.168.0.1
listen-address=192.168.1.1
dhcp-range=set:1stblock,192.168.1.10,192.168.0.50,255.255.255.0
dhcp-range=set:2ndblock,192.168.1.1,192.168.1.50,255.255.255.0
dhcp-option=tag:1stblock,option:router,192.168.0.1
dhcp-option=tag:2ndblock,option:router,192.168.1.1

So in above example, I assign a tag to each individual subnet (i.e 1stblock -> 192.168.0.0/24, 2ndblock -> 192.168.1.0/24) and then assign individual router ip with each associated tag.

QEMU/KVM atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)

So I am getting this annoying warning message when I press Enter on my KVM VM.

# 
atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)
atkbd.c Use 'setkeycodes 00 <keycode>' to make it known.

So now whenever I hit enter, the annoying warning will be coming up and this can be easily cleaned up by using the showkey and setkeycodes command.

So from the tty console (it may not work on x-windows), i executed showkey

# showkey
kb mode was UNICODE
[ if you are trying this under X, it might not work
since the X server is also reading /dev/console ]

press any key (program terminates 10s after last keypress)...

Now you can press 'Enter' (or whatever key that could cause the annoying message), it will show the keycode

keycode  28 press

It means that the Enter key is associated to keycode 28. Now you can use setkeycodes to map the Enter key.

# setkeycodes 0x00 28

0x00 here is the scancode, if you are interested on what it means, you could google it and there is some other articles out there explaining this.

Now, right after you typed the command, you should now be safed from this annoying warning message.

Should you want this be sustained across reboot, you may want to add it to /etc/rc.local

# echo 'setykeycodes 0x00 28' >> /etc/rc.local

Using gawk (GNU awk) to monitor /var/log/messages

I have a requirement to periodically scan through /var/log/messages to catch specific error message. The error itself is very time-sensitive and I want to be informed as soon as the message is shown on the log, ideally in a minute interval.

Usually people would suggest to periodically run grep against the log which is the simplest way to facilitate the need but it doesn't work for my scenario. The problem to simply run grep is that it may also capture unnecessary information. For example, I have below logs on my /var/log/messages.

Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal

If I simply use grep to catch above log entry in a minute interval, I will probably be informed at 12:40.06 and then all the way till the end of the day (i.e. when the /var/log/messages be rotated and cleaned up). Not to mention starting from 15:05:28, the grep script will catch 2 errors from the log, the lines being recorded at 12:40:05 and 15:05:28. However, the fact is that I actually need the one on 15:05:28.

After some research, it looks like gawk (GNU awk) would be the perfect tool to solve the problem. Eventually I come up with below gawk script to read /var/log/messages.


$ cat gawk-script
BEGIN {
    # Declare field seperator.
    FS="[- :.]";

    
    # Generate the timestamp
    NOW=systime();
    
    # The time interval that I want this script to read from
    # As long as I will run this script with cron in every minutes
    # I set the time interval to 1sec*60
    # So, for example when the script is run at 12:05, only log entries between 12:04 ~ 12:05 will be read.
    PAST=NOW-(60);

    
    # /var/log/messages start with the line in format of "Month date" (e.g Mar 23)
    # Ideally we can convert it to something like 23/3 but I want to be lazy, 
    # so I simply make use of the system time library and 
    # picked %Y %m to represent the Year and Month attributes.
    format="%Y %m";
    # LOGMTH will be something like "2012 10" (Oct 2012)
    # This will be used later to generate the timestamp of the log entry.
    LOGMTH=strftime(format, THEN)
}
{
    # Read the line of the log and convert it to a timestamp
    LOGTIME=mktime(LOGMTH " " $2 " " $3 " " $4 " " $5);
    # Below 3 lines can be commented out to debug with the value being read.
    #{print $2, $3, $4, $5};
    #{print LOGMTH};
    #{print LOGTIME};
    
    # print the log if the timestamp of the line is between PAST <> NOW.
    if(PAST<LOGTIME){print}
}

So here is a demo of the script.

$ cat /var/log/messages  ### totally 4 lines here
Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal Aug 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28
Aug 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices

$ date  #Lets check the current time
Wed Oct 3 18:04:24 HKT 2012

$ gawk -f gawk-script /var/log/messages
Oct 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28
Oct 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices

So, the script now only print everything captured within last minute. You can then use the script together with grep utility to catch the string you are interested.

Natting TCP port 2000 behind Cisco device.

Recently I am mudding on a storage appliance named NexentaStor. NexentaStor is based on OpenSolaris and make use of ZFS implementation which looks pretty promising. It has a clean and easy to use GUI, hence it supports quite a lot of storage protocol like NFS, CIFS, ISCSI and even support Link aggregation on network layer too.

Everything is going smooth so far just one minor obstacles, that is its Web GUI by default listen on tcp port 2000. Basically tcp port 2000 is a valid port but somehow I am not able to access the Web GUI and the connection towards the port keep timing out from outside, though the port is working on the same subnet.

I started to suspect there is something to do with NAT and yeah it is. I put my NexentaStor server behind a Cisco ASA firewall with NAT enabled. However, it looks like the port 2000 traffic of NexentaStor overlapped with the Cisco SCCP (http://en.wikipedia.org/wiki/Skinny_Call_Control_Protocol) on port 2000 too. Eventually I have to change the port of the GUI to a non-2000 port.

Just an additional notes, to reconfigure NexentaStor Web GUI port, I have to get into the console and execute below command.

nmc@myhost:/$ setup appliance init

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 6

Resources Management 
Below examples show how could one mange the HA resources in between the nodes.

- Check Cluster status

[root@dbmaster-01 ~]# crm_mon -1

============

Last updated: Fri Dec 30 00:43:51 2011

Last change: Fri Dec 30 00:20:38 2011 via crm_attribute on dbmaster-01.localdomain

Stack: openais

Current DC: dbmaster-01.localdomain - partition with quorum

Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558

2 Nodes configured, 2 expected votes

5 Resources configured.

============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

Resource Group: dbGroup

ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-01.localdomain

DBstore (ocf::heartbeat:Filesystem): Started dbmaster-01.localdomain

MySQL (ocf::heartbeat:mysql): Started dbmaster-01.localdomain

Clone Set: pingclone [check-ext-conn]

Started: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

-  Put node to offline mode

[root@dbmaster-01 ~]# crm node standby

[root@dbmaster-01 ~]# crm_mon -1

============

Last updated: Fri Dec 30 00:44:45 2011

Last change: Fri Dec 30 00:44:39 2011 via crm_attribute on dbmaster-01.localdomain

Stack: openais

Current DC: dbmaster-01.localdomain - partition with quorum

Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558

2 Nodes configured, 2 expected votes

5 Resources configured.

============

Node dbmaster-01.localdomain: standby

Online: [ dbmaster-02.localdomain ]

Resource Group: dbGroup

ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-02.localdomain

DBstore (ocf::heartbeat:Filesystem): Started dbmaster-02.localdomain

MySQL (ocf::heartbeat:mysql): Started dbmaster-02.localdomain

Clone Set: pingclone [check-ext-conn]

Started: [ dbmaster-02.localdomain ]

Stopped: [ check-ext-conn:0 ]

- Put node to online mode

[root@dbmaster-01 ~]# crm node online

[root@dbmaster-01 ~]# crm_mon -1

============

Last updated: Fri Dec 30 00:45:12 2011

Last change: Fri Dec 30 00:45:10 2011 via crm_attribute on dbmaster-01.localdomain

Stack: openais

Current DC: dbmaster-01.localdomain - partition with quorum

Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558

2 Nodes configured, 2 expected votes

5 Resources configured.

============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

Resource Group: dbGroup

ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-02.localdomain

DBstore (ocf::heartbeat:Filesystem): Started dbmaster-02.localdomain

MySQL (ocf::heartbeat:mysql): Started dbmaster-02.localdomain

Clone Set: pingclone [check-ext-conn]

Started: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

- Migrate resources to neighbor node

[root@dbmaster-01 ~]# crm resource migrate dbGroup dbmaster-02.localdomain

[root@dbmaster-01 ~]# crm_mon -1

============

Last updated: Fri Dec 30 00:47:50 2011

Last change: Fri Dec 30 00:47:35 2011 via crm_resource on dbmaster-01.localdomain

Stack: openais

Current DC: dbmaster-01.localdomain - partition with quorum

Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558

2 Nodes configured, 2 expected votes

5 Resources configured.

============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

Resource Group: dbGroup

ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-01.localdomain

DBstore (ocf::heartbeat:Filesystem): Started dbmaster-01.localdomain

MySQL (ocf::heartbeat:mysql): Started dbmaster-01.localdomain

Clone Set: pingclone [check-ext-conn]

Started: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

- Start / Stop / Restart specific resouce on node

[root@dbmaster-01 ~]# crm resource status MySQL

resource MySQL is running on: dbmaster-01.localdomain

...

[root@dbmaster-01 ~]# crm resource stop MySQL

....

[root@dbmaster-01 ~]# crm resource start MySQL

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 5

Cluster management

Corosync service is responsible for Cluster management while pacemaker is responsible for resource on top of the clustering service. The dependency of startup sequence will be 1) corosync and then 2) pacemaker. The shutdown sequence will be 1) pacemaker and then 2) corosync

- Check service status

[root@dbmaster-02 ~]# /etc/init.d/corosync status

corosync (pid 23118) is running...

[root@dbmaster-02 ~]# /etc/init.d/pacemaker status

pacemakerd (pid 8714) is running...

- Stop pacemaker and corosync

 

If the subject node is in active state, resources will be failed over to standby node. Alternatively if it is standby node, no changes will be made on active node. 

[root@dbmaster-02 ~]# /etc/init.d/pacemaker stop

Signaling Pacemaker Cluster Manager to terminate: [ OK ]

Waiting for cluster services to unload:....... [ OK ]

[root@dbmaster-02 ~]# /etc/init.d/corosync stop

Signaling Corosync Cluster Engine (corosync) to terminate: [ OK ]

Waiting for corosync services to unload:. [ OK ]

- Start corosync and pacemaker

If there isn’t any node running in the cluster, the first up shown up in the cluster will be the active node. If there is one active node in the cluster, the 2^nd node will automatically become the standby

[root@dbmaster-02 ~]# /etc/init.d/corosync start

Starting Corosync Cluster Engine (corosync): [ OK ]

[root@dbmaster-02 ~]# /etc/init.d/pacemaker start

Starting Pacemaker Cluster Manager: [ OK ]

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 4

- Configure Cluster Resources

Now the cluster is up, and we will have to add cluster resources (e.g. VIP, MySQL DB store, MySQL DB service) on top of the cluster. We only need to run this once on dbmaster-01 as the configuration changes will be written to cluster configuration file and being replicated to dbmaster-02.

- Configure misc cluster parameter

[root@dbmaster-01 ~]# crm configure property stonith-enabled=false

[root@dbmaster-01 ~]# crm configure property no-quorum-policy=ignore

[root@dbmaster-01 ~]# crm configure property start-failure-is-fatal="false"

[root@dbmaster-01 ~]# crm configure rsc_defaults resource-stickiness=100

- Configure VIP

[root@dbmaster-01 ~]# crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.0.10 cidr_netmask=32 op monitor interval=10s meta migration-threshold="10"

- Configure MySQL DB store, i.e. the shared-disk

[root@dbmaster-01 ~]# crm configure primitive DBstore ocf:heartbeat:Filesystem params device="/dev/sdb" directory="/mysql" fstype="ext4" meta migration-threshold="10"

WARNING: DBstore: default timeout 20s for start is smaller than the advised 60

WARNING: DBstore: default timeout 20s for stop is smaller than the advised 60

- Configure MySQL services

[root@dbmaster-01 ~]# crm configure primitive MySQL ocf:heartbeat:mysql params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" user="mysql" group="mysql" datadir="/mysql" log="/mysql/mysqld.log" \

> op start interval="0" timeout="60s" \

> op stop interval="0" timeout="60s" \

> op monitor interval="1min" timeout="60s" \

> meta migration-threshold="10" target-role="Started"

WARNING: MySQL: specified timeout 60s for start is smaller than the advised 120

WARNING: MySQL: specified timeout 60s for stop is smaller than the advised 120

- Configure all resources as a resource group for failover
If we don't configure them as a resource group, indivdual resources will be failovered seperately so eventually you may be seeing VIP on dbmaster01 while DB store on dbmaster02 which is something we don't want to see.

[root@dbmaster-01 ~]# crm configure group dbGroup ClusterIP DBstore MySQL

-  Define external ping monitoring and failover policy

This part of configuration will be a complicated, basically it means it will try to ping the gateway. In case the active node failed to ping gateway (e.g. internet connectivity down), it will fail over all services to standby node

[root@dbmaster-01 ~]# crm configure primitive check-ext-conn ocf:pacemaker:ping \

> params host_list="192.168.0.1" multiplier="100" attempts="3" \

> op monitor interval="10s" timeout="5s" start stop \

> meta migration-threshold="10"

WARNING: check-ext-conn: default timeout 20s for start is smaller than the advised 60

WARNING: check-ext-conn: specified timeout 5s for monitor is smaller than the advised 60

[root@dbmaster-01 ~]# crm configure clone pingclone check-ext-conn meta globally-unique="false"

[root@dbmaster-01 ~]# crm

crm(live)# configure

crm(live)configure# location dbnode dbGroup \

> rule $id="dbnode-rule" pingd: defined pingd \

> rule $id="dbnode-rule-0" -inf: not_defined pingd or pingd lte 10 \

> rule $id="dbnode-rule-1" 20: uname eq dbmaster-01.localdomain \

> rule $id="dbnode-rule-2" 20: uname eq dbmaster-01

crm(live)configure# end

There are changes pending. Do you want to commit them? Yes

crm(live)configure# exit

bye

- Review all configuration details.

[root@dbmaster-01 ~]# crm configure show

node dbmaster-01.localdomain

node dbmaster-02.localdomain

primitive ClusterIP ocf:heartbeat:IPaddr2 \

params ip="192.168.0.10" cidr_netmask="32" \

op monitor interval="10s" \

meta migration-threshold="10"

primitive DBstore ocf:heartbeat:Filesystem \

params device="/dev/sdb" directory="/mysql" fstype="ext4" \

meta migration-threshold="10"

primitive MySQL ocf:heartbeat:mysql \

params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" user="mysql" group="mysql" datadir="/mysql" log="/mysql/mysqld.log" \

op start interval="0" timeout="60s" \

op stop interval="0" timeout="60s" \

op monitor interval="1min" timeout="60s" \

meta migration-threshold="10" target-role="Started"

primitive check-ext-conn ocf:pacemaker:ping \

params host_list="192.168.0.1" multiplier="100" attempts="3" \

op monitor interval="10s" timeout="5s" start stop \

meta migration-threshold="10"

group dbGroup ClusterIP DBstore MySQL

clone pingclone check-ext-conn \

meta globally-unique="false"

location dbnode dbGroup \

rule $id="dbnode-rule" pingd: defined pingd \

rule $id="dbnode-rule-0" -inf: not_defined pingd or pingd lte 10 \

rule $id="dbnode-rule-1" 20: uname eq dbmaster-01.localdomain \

rule $id="dbnode-rule-2" 20: uname eq dbmaster-01

property $id="cib-bootstrap-options" \

dc-version="1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558" \

cluster-infrastructure="openais" \

expected-quorum-votes="2" \

stonith-enabled="false" \

no-quorum-policy="ignore" \

start-failure-is-fatal="false"

rsc_defaults $id="rsc-options" \

resource-stickiness="100"

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 3

- Cluster software installation and configuration

Now it is time to proceed with cluster software installation and configuration. If you are installing on CentOS, the packages could be fetched from default yum repository but if you are doing it on a RHEL6, you will probably need to add CentOS repository.


Below packages have to be installed on both nodes.

[root@dbmaster-01 yum.repos.d]# yum -y install pacemaker corosync

Loaded plugins: product-id, rhnplugin, subscription-manager

Updating certificate-based repositories.

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package corosync.x86_64 0:1.4.1-4.el6 will be installed

**********************

***** detail skipped ****

**********************

Installed:

corosync.x86_64 0:1.4.1-4.el6 pacemaker.x86_64 0:1.1.6-3.el6

Dependency Installed:

cifs-utils.x86_64 0:4.8.1-5.el6

cluster-glue.x86_64 0:1.0.5-2.el6

cluster-glue-libs.x86_64 0:1.0.5-2.el6

clusterlib.x86_64 0:3.0.12.1-23.el6

corosynclib.x86_64 0:1.4.1-4.el6

keyutils.x86_64 0:1.4-3.el6

libevent.x86_64 0:1.4.13-1.el6

libgssglue.x86_64 0:0.1-11.el6

libibverbs.x86_64 0:1.1.5-3.el6

librdmacm.x86_64 0:1.0.14.1-3.el6

libtalloc.x86_64 0:2.0.1-1.1.el6

libtirpc.x86_64 0:0.2.1-5.el6

nfs-utils.x86_64 1:1.2.3-15.el6

nfs-utils-lib.x86_64 0:1.1.5-4.el6

pacemaker-cli.x86_64 0:1.1.6-3.el6

pacemaker-cluster-libs.x86_64 0:1.1.6-3.el6

pacemaker-libs.x86_64 0:1.1.6-3.el6

resource-agents.x86_64 0:3.9.2-7.el6

rpcbind.x86_64 0:0.2.0-8.el6

Complete!

- Configure Corosync and Pacemaker
Create configuration file /etc/corosync/corosync.conf. We only need to run this on dbmaster-01 as we will replicate the file over to dbmaster-02

[root@dbmaster-01 ~]# export ais_port=5405

[root@dbmaster-01 ~]# export ais_mcast=226.94.1.1

[root@dbmaster-01 ~]# export ais_addr=`ip addr | grep "inet " | grep eth0 | awk '{print $4}' | sed s/255/0/`

[root@dbmaster-01 ~]# env | grep ais_

ais_mcast=226.94.1.1

ais_port=5405

ais_addr=192.168.0.255

[root@dbmaster-01 ~]# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf

[root@dbmaster-01 ~]# sed -i.bak "s/.*mcastaddr:.*/mcastaddr:\ $ais_mcast/g" /etc/corosync/corosync.conf

[root@dbmaster-01 ~]# sed -i.bak "s/.*mcastport:.*/mcastport:\ $ais_port/g" /etc/corosync/corosync.conf

[root@dbmaster-01 ~]# sed -i.bak "s/.*bindnetaddr:.*/bindnetaddr:\ $ais_addr/g" /etc/corosync/corosync.conf

[root@dbmaster-01 ~]# cat <<-END >>/etc/corosync/service.d/pcmk

> service {

> # Load the Pacemaker Cluster Resource Manager

> name: pacemaker

> ver: 1

> }

> END

- Review the configuration file /etc/corosync/corosync.conf

[root@dbmaster-01 ~]# cd /etc/corosync

[root@dbmaster-01 corosync]# cat corosync.conf

# Please read the corosync.conf.5 manual page

compatibility: whitetank

totem {

version: 2

secauth: off

threads: 0

interface {

ringnumber: 0

bindnetaddr: 192.168.114.127

mcastaddr: 226.94.1.1

mcastport: 5405

ttl: 1

}

}

logging {

fileline: off

to_stderr: no

to_logfile: yes

to_syslog: yes

logfile: /var/log/cluster/corosync.log

debug: off

timestamp: on

logger_subsys {

subsys: AMF

debug: off

}

}

amf {

mode: disabled

}

- Replicate the configuration to neighbor node (dbmaster-02) and start corosync service.

[root@dbmaster-01 corosync]# for f in /etc/corosync/corosync.conf /etc/corosync/service.d/pcmk /etc/hosts; do scp $f dbmaster-02:$f ; done

[root@dbmaster-01 corosync]# /etc/init.d/corosync start

Starting Corosync Cluster Engine (corosync): [ OK ]

[root@dbmaster-01 corosync]# grep -e "corosync.*network interface" -e "Corosync Cluster Engine" -e "Successfully read main configuration file" /var/log/messages

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [TOTEM ] The network interface [192.168.0.11] is now up.

[root@dbmaster-01 corosync]# grep TOTEM /var/log/messages

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [TOTEM ] Initializing transport (UDP/IP Multicast).

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [TOTEM ] The network interface [192.168.0.11] is now up.

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [TOTEM ] A processor joined or left the membership and a new membership was formed.

[root@dbmaster-01 ~]# ssh dbmaster-02 -- /etc/init.d/corosync start

Starting Corosync Cluster Engine (corosync): [ OK ]

- Monitoring startup status of corosync
Make sure pacemaker module is loaded successfully.

[root@dbmaster-01 corosync]# grep pcmk_startup /var/log/messages

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: CRM: Initialized

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [pcmk ] Logging: Initialized pcmk_startup

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: Service: 10

Dec 29 03:08:39 dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: Local hostname: dbmaster-01.localdomain

- Startup pacemaker on both nodes

[root@dbmaster-01 ~]# chown -R hacluster:haclient /var/log/cluster

[root@dbmaster-01 ~]# /etc/init.d/pacemaker start

Starting Pacemaker Cluster Manager: [ OK ]

[root@dbmaster-01 ~]# grep -e pacemakerd.*get_config_opt -e pacemakerd.*start_child -e "Starting Pacemaker" /var/log/messages

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'pacemaker' for option: name

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found '1' for option: ver

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'pacemaker' for option: name

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found '1' for option: ver

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Defaulting to 'no' for option: use_logd

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Defaulting to 'no' for option: use_mgmtd

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'off' for option: debug

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'yes' for option: to_logfile

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'yes' for option: to_syslog

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: main: Starting Pacemaker 1.1.6-3.el6 (Build: a02c0f19a00c1eb2527ad38f146ebc0834814558): generated-manpages agent-manpages ascii-docs publican-docs ncurses trace-logging cman corosync-quorum corosync

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child 31341 for process stonith-ng

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child 31342 for process cib

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child 31343 for process lrmd

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child 31344 for process attrd

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child 31345 for process pengine

Dec 29 03:29:05 dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child 31346 for process crmd

[root@dbmaster-01 ~]# ssh dbmaster-02 -- chown -R hacluster:haclient /var/log/cluster

[root@dbmaster-01 ~]# ssh dbmaster-02 -- /etc/init.d/pacemaker start

Starting Pacemaker Cluster Manager: [ OK ]

- Verify if heartbeat processes are started

[root@dbmaster-01 ~]# ps axf

PID TTY STAT TIME COMMAND

2 ? S 0:00 [kthreadd]

... lots of processes ....

27718 ? Ssl 0:00 corosync

31337 pts/0 S 0:00 pacemakerd

31341 ? Ss 0:00 \_ /usr/lib64/heartbeat/stonithd

31342 ? Ss 0:00 \_ /usr/lib64/heartbeat/cib

31343 ? Ss 0:00 \_ /usr/lib64/heartbeat/lrmd

31344 ? Ss 0:00 \_ /usr/lib64/heartbeat/attrd

31345 ? Ss 0:00 \_ /usr/lib64/heartbeat/pengine

31346 ? Ss 0:00 \_ /usr/lib64/heartbeat/crmd

[root@test-db1 corosync]# grep ERROR: /var/log/messages | grep -v unpack_resources

[root@test-db1 corosync]#

- Verify the HA cluster is running now

[root@dbmaster-01 ~]# crm_mon

============

Last updated: Thu Dec 29 05:19:52 2011

Last change: Thu Dec 29 05:07:59 2011 via crmd on dbmaster-01.localdomain

Stack: openais

Current DC: dbmaster-01.localdomain - partition with quorum

Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558

2 Nodes configured, 2 expected votes

0 Resources configured.

============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]