2012年11月23日 星期五

dnsmasq, assign duplicate DHCP subnet via single interface

I was troubleshooting a dnsmasq DHCP server back in few days and notice special settings have to be applied if the particular dnsmasq DHCP server is configured to serve multiple subnet.

In my scenario, the DHCP server is having one nic, eth0. For some reason, there are 2 subnets being served, 192.168.0.0/24 and 192.168.1.0/24. The eth0 is configured as 192.168.0.1/24 while an additional ip 192.168.1.1 is added to eth0 too.

So DHCP clients that connect to the eth0 of DHCP server (through switches) could retrieve IP from same subnet without any issue, however it seems like the DHCP gateway of the 2nd subnet is acting weird. For example, here is the DHCP lease file i got from one of the DHCP client.

# cat /var/lib/dhcp/pump.lease
Device eth0
    IP: 192.168.1.10
    Netmask: 255.255.255.0
    Broadcast: 192.168.1.255
    Network: 192.168.1.0
    Boot server 192.168.1.1
    Next server 192.168.1.1
    Gateways: 192.168.0.1
    Hostname: test-dhcp-client
    Domain: test.internal
    Renewal time: Fri Nov 23 17:38:13 2012
    Expiration time: Fri Nov 23 19:08:13 2012


Interestingly, this DHCP client is assigned with the 2nd DHCP subnet (i.e. 192.168.1.0/24) however gateway of default subnet (192.168.0.0/24) is being assigned.

To fix this, I tried couples of approach but eventually it looks like configuring the option though dhcp-range tagging would do the best. Below are the configuration snippet that fixed the problem.

listen-address=192.168.0.1
listen-address=192.168.1.1
dhcp-range=set:1stblock,192.168.1.10,192.168.0.50,255.255.255.0
dhcp-range=set:2ndblock,192.168.1.1,192.168.1.50,255.255.255.0
dhcp-option=tag:1stblock,option:router,192.168.0.1
dhcp-option=tag:2ndblock,option:router,192.168.1.1

So in above example, I assign a tag to each individual subnet (i.e 1stblock -> 192.168.0.0/24, 2ndblock -> 192.168.1.0/24) and then assign individual router ip with each associated tag.

2012年10月25日 星期四

QEMU/KVM atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)

So I am getting this annoying warning message when I press Enter on my KVM VM.

# 
atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)
atkbd.c Use 'setkeycodes 00 <keycode>' to make it known.

So now whenever I hit enter, the annoying warning will be coming up and this can be easily cleaned up by using the showkey and setkeycodes command.

So from the tty console (it may not work on x-windows), i executed showkey

# showkey
kb mode was UNICODE
[ if you are trying this under X, it might not work
since the X server is also reading /dev/console ]

press any key (program terminates 10s after last keypress)...

Now you can press 'Enter' (or whatever key that could cause the annoying message), it will show the keycode

keycode  28 press

It means that the Enter key is associated to keycode 28. Now you can use setkeycodes to map the Enter key.

# setkeycodes 0x00 28

0x00 here is the scancode, if you are interested on what it means, you could google it and there is some other articles out there explaining this.

Now, right after you typed the command, you should now be safed from this annoying warning message.

Should you want this be sustained across reboot, you may want to add it to /etc/rc.local

# echo 'setykeycodes 0x00 28' >> /etc/rc.local


2012年8月16日 星期四

Using gawk (GNU awk) to monitor /var/log/messages

I have a requirement to periodically scan through /var/log/messages to catch specific error message. The error itself is very time-sensitive and I want to be informed as soon as the message is shown on the log, ideally in a minute interval.

Usually people would suggest to periodically run grep against the log which is the simplest way to facilitate the need but it doesn't work for my scenario. The problem to simply run grep is that it may also capture unnecessary information. For example, I have below logs on my /var/log/messages.

Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal

If I simply use grep to catch above log entry in a minute interval, I will probably be informed at 12:40.06 and then all the way till the end of the day (i.e. when the /var/log/messages be rotated and cleaned up). Not to mention starting from 15:05:28, the grep script will catch 2 errors from the log, the lines being recorded at 12:40:05 and 15:05:28. However, the fact is that I actually need the one on 15:05:28.

After some research, it looks like gawk (GNU awk) would be the perfect tool to solve the problem. Eventually I come up with below gawk script to read /var/log/messages.


$ cat gawk-script
BEGIN {
    # Declare field seperator.
    FS="[- :.]";


    
    # Generate the timestamp
    NOW=systime();

    
    # The time interval that I want this script to read from
    # As long as I will run this script with cron in every minutes
    # I set the time interval to 1sec*60
    # So, for example when the script is run at 12:05, only log entries between 12:04 ~ 12:05 will be read.
    PAST=NOW-(60);


    
    # /var/log/messages start with the line in format of "Month date" (e.g Mar 23)
    # Ideally we can convert it to something like 23/3 but I want to be lazy, 
    # so I simply make use of the system time library and 
    # picked %Y %m to represent the Year and Month attributes.
    format="%Y %m";

    # LOGMTH will be something like "2012 10" (Oct 2012)
    # This will be used later to generate the timestamp of the log entry.
    LOGMTH=strftime(format, THEN)
}
{

    # Read the line of the log and convert it to a timestamp
    LOGTIME=mktime(LOGMTH " " $2 " " $3 " " $4 " " $5);

    # Below 3 lines can be commented out to debug with the value being read.
    #{print $2, $3, $4, $5};
    #{print LOGMTH};
    #{print LOGTIME};

    
    # print the log if the timestamp of the line is between PAST <> NOW.
    if(PAST<LOGTIME){print}
}


So here is a demo of the script.

$ cat /var/log/messages  ### totally 4 lines here
Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal Aug 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28
Aug 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices


$ date  #Lets check the current time
Wed Oct 3 18:04:24 HKT 2012


$ gawk -f gawk-script /var/log/messages
Oct 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28
Oct 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices


So, the script now only print everything captured within last minute. You can then use the script together with grep utility to catch the string you are interested.

2012年7月24日 星期二

Natting TCP port 2000 behind Cisco device.

Recently I am mudding on a storage appliance named NexentaStor. NexentaStor is based on OpenSolaris and make use of ZFS implementation which looks pretty promising. It has a clean and easy to use GUI, hence it supports quite a lot of storage protocol like NFS, CIFS, ISCSI and even support Link aggregation on network layer too.

Everything is going smooth so far just one minor obstacles, that is its Web GUI by default listen on tcp port 2000. Basically tcp port 2000 is a valid port but somehow I am not able to access the Web GUI and the connection towards the port keep timing out from outside, though the port is working on the same subnet.

I started to suspect there is something to do with NAT and yeah it is. I put my NexentaStor server behind a Cisco ASA firewall with NAT enabled. However, it looks like the port 2000 traffic of NexentaStor overlapped with the Cisco SCCP (http://en.wikipedia.org/wiki/Skinny_Call_Control_Protocol) on port 2000 too. Eventually I have to change the port of the GUI to a non-2000 port.

Just an additional notes, to reconfigure NexentaStor Web GUI port, I have to get into the console and execute below command.

nmc@myhost:/$ setup appliance init

2012年6月24日 星期日

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 6

Resources Management 
Below examples show how could one mange the HA resources in between the nodes.

- Check Cluster status

[root@dbmaster-01 ~]# crm_mon -1
============
Last updated: Fri Dec 30 00:43:51 2011
Last change: Fri Dec 30 00:20:38 2011 via crm_attribute on dbmaster-01.localdomain
Stack: openais
Current DC: dbmaster-01.localdomain - partition with quorum
Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

Resource Group: dbGroup
ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-01.localdomain
DBstore (ocf::heartbeat:Filesystem): Started dbmaster-01.localdomain
MySQL (ocf::heartbeat:mysql): Started dbmaster-01.localdomain
Clone Set: pingclone [check-ext-conn]
Started: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

-  Put node to offline mode
[root@dbmaster-01 ~]# crm node standby
[root@dbmaster-01 ~]# crm_mon -1
============
Last updated: Fri Dec 30 00:44:45 2011
Last change: Fri Dec 30 00:44:39 2011 via crm_attribute on dbmaster-01.localdomain
Stack: openais
Current DC: dbmaster-01.localdomain - partition with quorum
Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Node dbmaster-01.localdomain: standby
Online: [ dbmaster-02.localdomain ]

Resource Group: dbGroup
ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-02.localdomain
DBstore (ocf::heartbeat:Filesystem): Started dbmaster-02.localdomain
MySQL (ocf::heartbeat:mysql): Started dbmaster-02.localdomain
Clone Set: pingclone [check-ext-conn]
Started: [ dbmaster-02.localdomain ]
Stopped: [ check-ext-conn:0 ]

- Put node to online mode

[root@dbmaster-01 ~]# crm node online
[root@dbmaster-01 ~]# crm_mon -1
============
Last updated: Fri Dec 30 00:45:12 2011
Last change: Fri Dec 30 00:45:10 2011 via crm_attribute on dbmaster-01.localdomain
Stack: openais
Current DC: dbmaster-01.localdomain - partition with quorum
Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

Resource Group: dbGroup
ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-02.localdomain
DBstore (ocf::heartbeat:Filesystem): Started dbmaster-02.localdomain
MySQL (ocf::heartbeat:mysql): Started dbmaster-02.localdomain
Clone Set: pingclone [check-ext-conn]
Started: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

- Migrate resources to neighbor node

[root@dbmaster-01 ~]# crm resource migrate dbGroup dbmaster-02.localdomain
[root@dbmaster-01 ~]# crm_mon -1
============
Last updated: Fri Dec 30 00:47:50 2011
Last change: Fri Dec 30 00:47:35 2011 via crm_resource on dbmaster-01.localdomain
Stack: openais
Current DC: dbmaster-01.localdomain - partition with quorum
Version: 1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558
2 Nodes configured, 2 expected votes
5 Resources configured.
============

Online: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

Resource Group: dbGroup
ClusterIP (ocf::heartbeat:IPaddr2): Started dbmaster-01.localdomain
DBstore (ocf::heartbeat:Filesystem): Started dbmaster-01.localdomain
MySQL (ocf::heartbeat:mysql): Started dbmaster-01.localdomain
Clone Set: pingclone [check-ext-conn]
Started: [ dbmaster-01.localdomain dbmaster-02.localdomain ]

- Start / Stop / Restart specific resouce on node

[root@dbmaster-01 ~]# crm resource status MySQL
resource MySQL is running on: dbmaster-01.localdomain
...
[root@dbmaster-01 ~]# crm resource stop MySQL

....
[root@dbmaster-01 ~]# crm resource start MySQL

2012年6月21日 星期四

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 5

Cluster management

Corosync service is responsible for Cluster management while pacemaker is responsible for resource on top of the clustering service. The dependency of startup sequence will be 1) corosync and then 2) pacemaker. The shutdown sequence will be 1) pacemaker and then 2) corosync

- Check service status
[root@dbmaster-02 ~]# /etc/init.d/corosync status
corosync (pid 23118) is running...
[root@dbmaster-02 ~]# /etc/init.d/pacemaker status
pacemakerd (pid 8714) is running...

- Stop pacemaker and corosync
 
If the subject node is in active state, resources will be failed over to standby node. Alternatively if it is standby node, no changes will be made on active node. 
[root@dbmaster-02 ~]# /etc/init.d/pacemaker stop
Signaling Pacemaker Cluster Manager to terminate: [ OK ]
Waiting for cluster services to unload:....... [ OK ]
[root@dbmaster-02 ~]# /etc/init.d/corosync stop
Signaling Corosync Cluster Engine (corosync) to terminate: [ OK ]
Waiting for corosync services to unload:. [ OK ]

- Start corosync and pacemaker
If there isn’t any node running in the cluster, the first up shown up in the cluster will be the active node. If there is one active node in the cluster, the 2nd node will automatically become the standby

[root@dbmaster-02 ~]# /etc/init.d/corosync start
Starting Corosync Cluster Engine (corosync): [ OK ]
[root@dbmaster-02 ~]# /etc/init.d/pacemaker start
Starting Pacemaker Cluster Manager: [ OK ]


2012年6月18日 星期一

HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 4

- Configure Cluster Resources

Now the cluster is up, and we will have to add cluster resources (e.g. VIP, MySQL DB store, MySQL DB service) on top of the cluster. We only need to run this once on dbmaster-01 as the configuration changes will be written to cluster configuration file and being replicated to dbmaster-02.

- Configure misc cluster parameter

[root@dbmaster-01 ~]# crm configure property stonith-enabled=false
[root@dbmaster-01 ~]# crm configure property no-quorum-policy=ignore
[root@dbmaster-01 ~]# crm configure property start-failure-is-fatal="false"
[root@dbmaster-01 ~]# crm configure rsc_defaults resource-stickiness=100

- Configure VIP

[root@dbmaster-01 ~]# crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.0.10 cidr_netmask=32 op monitor interval=10s meta migration-threshold="10"

- Configure MySQL DB store, i.e. the shared-disk
[root@dbmaster-01 ~]# crm configure primitive DBstore ocf:heartbeat:Filesystem params device="/dev/sdb" directory="/mysql" fstype="ext4" meta migration-threshold="10"
WARNING: DBstore: default timeout 20s for start is smaller than the advised 60
WARNING: DBstore: default timeout 20s for stop is smaller than the advised 60

- Configure MySQL services
[root@dbmaster-01 ~]# crm configure primitive MySQL ocf:heartbeat:mysql params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" user="mysql" group="mysql" datadir="/mysql" log="/mysql/mysqld.log" \
> op start interval="0" timeout="60s" \
> op stop interval="0" timeout="60s" \
> op monitor interval="1min" timeout="60s" \
> meta migration-threshold="10" target-role="Started"
WARNING: MySQL: specified timeout 60s for start is smaller than the advised 120
WARNING: MySQL: specified timeout 60s for stop is smaller than the advised 120

- Configure all resources as a resource group for failover
If we don't configure them as a resource group, indivdual resources will be failovered seperately so eventually you may be seeing VIP on dbmaster01 while DB store on dbmaster02 which is something we don't want to see.

[root@dbmaster-01 ~]# crm configure group dbGroup ClusterIP DBstore MySQL

-  Define external ping monitoring and failover policy
This part of configuration will be a complicated, basically it means it will try to ping the gateway. In case the active node failed to ping gateway (e.g. internet connectivity down), it will fail over all services to standby node

[root@dbmaster-01 ~]# crm configure primitive check-ext-conn ocf:pacemaker:ping \
> params host_list="192.168.0.1" multiplier="100" attempts="3" \
> op monitor interval="10s" timeout="5s" start stop \
> meta migration-threshold="10"
WARNING: check-ext-conn: default timeout 20s for start is smaller than the advised 60
WARNING: check-ext-conn: specified timeout 5s for monitor is smaller than the advised 60
[root@dbmaster-01 ~]# crm configure clone pingclone check-ext-conn meta globally-unique="false"
[root@dbmaster-01 ~]# crm
crm(live)# configure
crm(live)configure# location dbnode dbGroup \
> rule $id="dbnode-rule" pingd: defined pingd \
> rule $id="dbnode-rule-0" -inf: not_defined pingd or pingd lte 10 \
> rule $id="dbnode-rule-1" 20: uname eq dbmaster-01.localdomain \
> rule $id="dbnode-rule-2" 20: uname eq dbmaster-01
crm(live)configure# end
There are changes pending. Do you want to commit them? Yes
crm(live)configure# exit
bye

- Review all configuration details.
[root@dbmaster-01 ~]# crm configure show
node dbmaster-01.localdomain
node dbmaster-02.localdomain
primitive ClusterIP ocf:heartbeat:IPaddr2 \
params ip="192.168.0.10" cidr_netmask="32" \
op monitor interval="10s" \
meta migration-threshold="10"
primitive DBstore ocf:heartbeat:Filesystem \
params device="/dev/sdb" directory="/mysql" fstype="ext4" \
meta migration-threshold="10"
primitive MySQL ocf:heartbeat:mysql \
params binary="/usr/bin/mysqld_safe" config="/etc/my.cnf" user="mysql" group="mysql" datadir="/mysql" log="/mysql/mysqld.log" \
op start interval="0" timeout="60s" \
op stop interval="0" timeout="60s" \
op monitor interval="1min" timeout="60s" \
meta migration-threshold="10" target-role="Started"
primitive check-ext-conn ocf:pacemaker:ping \
params host_list="192.168.0.1" multiplier="100" attempts="3" \
op monitor interval="10s" timeout="5s" start stop \
meta migration-threshold="10"
group dbGroup ClusterIP DBstore MySQL
clone pingclone check-ext-conn \
meta globally-unique="false"
location dbnode dbGroup \
rule $id="dbnode-rule" pingd: defined pingd \
rule $id="dbnode-rule-0" -inf: not_defined pingd or pingd lte 10 \
rule $id="dbnode-rule-1" 20: uname eq dbmaster-01.localdomain \
rule $id="dbnode-rule-2" 20: uname eq dbmaster-01
property $id="cib-bootstrap-options" \
dc-version="1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="false" \
no-quorum-policy="ignore" \
start-failure-is-fatal="false"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"