tag:blogger.com,1999:blog-83920211811982389172024-03-05T23:03:19.101-08:00Philip's Tech BlogI am a Linux Administrator in Hong Kong, specialized in RHEL administration as well as IAAS cloud deployment. :-)Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.comBlogger63125tag:blogger.com,1999:blog-8392021181198238917.post-79516909621615032902015-09-11T11:08:00.002-07:002015-09-11T11:14:10.241-07:00Openstack nova Instance failed to push static IP through config drive. openstack/content/0000 is missingSo I need to bootstrap an instance that is configured as static IP through config drive, instead of the generic DHCP. So with nova boot, everything seems to be happy but the instance just failed to get the static IP for some reason. Doing a nova console-log, the IP address shows like this in cloud-init.<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">Cloud-init</span> <span style="color: #d0d0d0;">v.</span> <span style="color: #3677a9;">0.7</span><span style="color: #d0d0d0;">.</span><span style="color: #3677a9;">5</span> <span style="color: #d0d0d0;">running</span> <span style="color: #ed9d13;">'init'</span> <span style="color: #d0d0d0;">at</span> <span style="color: #d0d0d0;">Fri,</span> <span style="color: #3677a9;">11</span> <span style="color: #d0d0d0;">Sep</span> <span style="color: #3677a9;">2015</span> <span style="color: #3677a9;">17</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">56</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">46</span> <span style="color: #d0d0d0;">+</span><span style="color: #3677a9;">0000.</span> <span style="color: #d0d0d0;">Up</span> <span style="color: #3677a9;">29.44</span> <span style="color: #d0d0d0;">seconds.</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">+++++++++++++++++++++++Net</span> <span style="color: #d0d0d0;">device</span> <span style="color: #d0d0d0;">info+++++++++++++++++++++++</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">+--------+------+-----------+-----------+-------------------+</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">Device</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">Up</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">Address</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">Mask</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">Hw-Address</span> <span style="color: #d0d0d0;">|</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">+--------+------+-----------+-----------+-------------------+</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">lo:</span> <span style="color: #d0d0d0;">|</span> <span style="color: #24909d;">True</span> <span style="color: #d0d0d0;">|</span> <span style="color: #3677a9;">127.0</span><span style="color: #d0d0d0;">.</span><span style="color: #3677a9;">0.1</span> <span style="color: #d0d0d0;">|</span> <span style="color: #3677a9;">255.0</span><span style="color: #d0d0d0;">.</span><span style="color: #3677a9;">0.0</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">.</span> <span style="color: #d0d0d0;">|</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">eth0:</span> <span style="color: #d0d0d0;">|</span> <span style="color: #24909d;">True</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">.</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">.</span> <span style="color: #d0d0d0;">|</span> <span style="color: #d0d0d0;">fa:</span><span style="color: #3677a9;">16</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">3</span><span style="color: #d0d0d0;">e:</span><span style="color: #3677a9;">4</span><span style="color: #d0d0d0;">d:</span><span style="color: #3677a9;">35</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">34</span> <span style="color: #d0d0d0;">|</span>
<span style="color: #d0d0d0;">cloud-init[</span><span style="color: #3677a9;">775</span><span style="color: #d0d0d0;">]:</span> <span style="color: #d0d0d0;">ci-info:</span> <span style="color: #d0d0d0;">+--------+------+-----------+-----------+-------------------+</span>
</pre>
</div>
<br />
Obviously the eth0 is not getting the static IP. Once again I mounted the config-drive and then try to find the /openstack/content/0000 file but I can't find it.<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;">[root</span><span style="color: orange;">@nova</span> <span style="color: #d0d0d0;">~]</span><span style="color: #999999; font-style: italic;"># find /mnt/openstack | grep content</span>
<span style="color: #d0d0d0;">[root</span><span style="color: orange;">@nova</span> <span style="color: #d0d0d0;">~]</span><span style="color: #999999; font-style: italic;">#</span>
</pre>
</div>
<br />
This is pretty weird as I remember the file should be there to allow static ip assignment to work through config-drive. So taking further looks, it seems the problem go to the subnet configuration. The subnet that the instance is on is having "enable_dhcp" equal to true, and that prohibited the config-drive to create the openstack/content/0000 file.<br />
<br />
To disable DHCP for a subnet, run this.<br />
<div style="background: #202020; overflow:auto;width:auto;border:solid gray;border-width:.1em .1em .1em .8em;padding:.2em .6em;"><pre style="margin: 0; line-height: 125%"><span style="color: #999999; font-style: italic"># neutron subnet-update $SUBNET_UUID --enable-dhcp=False</span>
</pre></div>
Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-85765791910161829532015-09-10T10:37:00.000-07:002015-09-11T11:08:57.036-07:00Openstack broken metadata interfaces templateWas my 2nd time hitting the same issue and it wasted my whole day so I think I should document this as a note just in case it happens again.<br />
<br />
In case you are seeing something like this from your metadata (be it metadata server, or config-drive), in this example I mounted the config drive as /mnt.<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;"># cat</span> <span style="color: #d0d0d0;">/mnt/openstack/content/</span><span style="color: #3677a9;">0000</span>
<span style="color: #d0d0d0;">DEVICE=</span><span style="color: #ed9d13;">"{{ name }}"</span>
<span style="color: #d0d0d0;">NM_CONTROLLED=</span><span style="color: #ed9d13;">"no"</span>
<span style="color: #d0d0d0;">ONBOOT=yes</span>
<span style="color: #d0d0d0;">TYPE=Ethernet</span>
<span style="color: #d0d0d0;">BOOTPROTO=static</span>
<span style="color: #d0d0d0;">IPADDR={{</span> <span style="color: #d0d0d0;">address</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">NETMASK={{</span> <span style="color: #d0d0d0;">netmask</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">BROADCAST={{</span> <span style="color: #d0d0d0;">broadcast</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">GATEWAY={{</span> <span style="color: #d0d0d0;">gateway</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">DNS1={{</span> <span style="color: #d0d0d0;">dns</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #999999; font-style: italic;">#if $use_ipv6</span>
<span style="color: #d0d0d0;">IPV6INIT=yes</span>
<span style="color: #d0d0d0;">IPV6ADDR={{</span> <span style="color: #d0d0d0;">address_v6</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #999999; font-style: italic;">#end if</span>
</pre>
</div>
<br />
Chances that you are hitting this <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1236219" target="_blank">bug</a>.<br />
<br />
As cloud-init doesnt really recognize format like that, to fix the issue you will need to update the template /usr/share/nova/interfaces.template (assuming you are on CentOS/RHEL7) with something like this which is a debian-ish template<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #999999; font-style: italic;"># Injected by Nova on instance boot</span>
<span style="color: #999999; font-style: italic;">#</span>
<span style="color: #999999; font-style: italic;"># This file describes the network interfaces available on your system</span>
<span style="color: #999999; font-style: italic;"># and how to activate them. For more information, see interfaces(5).</span>
<span style="color: #999999; font-style: italic;"># The loopback network interface</span>
<span style="color: #d0d0d0;">auto</span> <span style="color: #d0d0d0;">lo</span>
<span style="color: #d0d0d0;">iface</span> <span style="color: #d0d0d0;">lo</span> <span style="color: #d0d0d0;">inet</span> <span style="color: #d0d0d0;">loopback</span>
<span style="color: #d0d0d0;">{%</span> <span style="color: #6ab825; font-weight: bold;">for</span> <span style="color: #d0d0d0;">ifc</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">interfaces</span> <span style="color: #d0d0d0;">-%}</span>
<span style="color: #d0d0d0;">auto</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.name</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">iface</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.name</span> <span style="color: #d0d0d0;">}}</span> <span style="color: #d0d0d0;">inet</span> <span style="color: #d0d0d0;">static</span>
<span style="color: #d0d0d0;">address</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.address</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">netmask</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.netmask</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">broadcast</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.broadcast</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #6ab825; font-weight: bold;">if</span> <span style="color: #d0d0d0;">ifc.gateway</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">gateway</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.gateway</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #d0d0d0;">endif</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #6ab825; font-weight: bold;">if</span> <span style="color: #d0d0d0;">ifc.dns</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">dns-nameservers</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.dns</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #d0d0d0;">endif</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">{%</span> <span style="color: #6ab825; font-weight: bold;">if</span> <span style="color: #d0d0d0;">use_ipv6</span> <span style="color: #d0d0d0;">-%}</span>
<span style="color: #d0d0d0;">iface</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.name</span> <span style="color: #d0d0d0;">}}</span> <span style="color: #d0d0d0;">inet6</span> <span style="color: #d0d0d0;">static</span>
<span style="color: #d0d0d0;">address</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.address_v6</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">netmask</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.netmask_v6</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #6ab825; font-weight: bold;">if</span> <span style="color: #d0d0d0;">ifc.gateway_v6</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">gateway</span> <span style="color: #d0d0d0;">{{</span> <span style="color: #d0d0d0;">ifc.gateway_v6</span> <span style="color: #d0d0d0;">}}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #d0d0d0;">endif</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #d0d0d0;">endif</span> <span style="color: #d0d0d0;">%}</span>
<span style="color: #d0d0d0;">{%-</span> <span style="color: #d0d0d0;">endfor</span> <span style="color: #d0d0d0;">%}</span>
</pre>
</div>
Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-86025346607311843032015-09-02T07:10:00.000-07:002015-09-10T07:28:23.173-07:00Cloudera Manager agent failed to connect to previous supervisorContinue to <a href="http://philipsilly.blogspot.com/2015/09/cloudera-manager-agent-installation.html">another article</a> written earlier, I am hitting another road block while installing Cloudera manager agent.<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;">[</span><span style="color: #3677a9;">10</span><span style="color: #d0d0d0;">/Sep/</span><span style="color: #3677a9;">2015</span> <span style="color: #3677a9;">09</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">10</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">54</span> <span style="color: #d0d0d0;">+</span><span style="color: #3677a9;">0000</span><span style="color: #d0d0d0;">]</span> <span style="color: #3677a9;">19017</span> <span style="color: #d0d0d0;">MainThread</span> <span style="color: #d0d0d0;">agent</span> <span style="color: #d0d0d0;">ERROR</span> <span style="color: #d0d0d0;">Failed</span> <span style="color: #d0d0d0;">to</span> <span style="color: #d0d0d0;">connect</span> <span style="color: #d0d0d0;">to</span> <span style="color: #d0d0d0;">previous</span> <span style="color: #d0d0d0;">supervisor.</span>
<span style="color: #d0d0d0;">Traceback</span> <span style="color: #d0d0d0;">(most</span> <span style="color: #d0d0d0;">recent</span> <span style="color: #d0d0d0;">call</span> <span style="color: #d0d0d0;">last):</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/cmf/agent/src/cmf/agent.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">1524</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">find_or_start_supervisor</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.get_supervisor_process_info()</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/cmf/agent/src/cmf/agent.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">1725</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">get_supervisor_process_info</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.identifier</span> <span style="color: #d0d0d0;">=</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.supervisor_client.supervisor.getIdentification()</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/xmlrpclib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">1199</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">__call__</span>
<span style="color: #6ab825; font-weight: bold;">return</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.__send(</span><span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.__name,</span> <span style="color: #d0d0d0;">args)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/xmlrpclib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">1489</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">__request</span>
<span style="color: #d0d0d0;">verbose=</span><span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.__verbose</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/xmlrpc.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">460</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">request</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.connection.request(</span><span style="color: #ed9d13;">'POST'</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">handler,</span> <span style="color: #d0d0d0;">request_body,</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.headers)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/httplib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">914</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">request</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">._send_request(method,</span> <span style="color: #d0d0d0;">url,</span> <span style="color: #d0d0d0;">body,</span> <span style="color: #d0d0d0;">headers)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/httplib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">951</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">_send_request</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.endheaders()</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/httplib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">908</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">endheaders</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">._send_output()</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/httplib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">780</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">_send_output</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.send(msg)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/httplib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">739</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">send</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.connect()</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/httplib.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">720</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">connect</span>
<span style="color: #24909d;">self</span><span style="color: #d0d0d0;">.timeout)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/socket.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">567</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">create_connection</span>
<span style="color: #6ab825; font-weight: bold;">raise</span> <span style="color: #d0d0d0;">error,</span> <span style="color: #d0d0d0;">msg</span>
<span style="color: #d0d0d0;">error:</span> <span style="color: #d0d0d0;">[Errno</span> <span style="color: #3677a9;">111</span><span style="color: #d0d0d0;">]</span> <span style="color: #d0d0d0;">Connection</span> <span style="color: #d0d0d0;">refused</span>
</pre>
</div>
<br />
So if you are seeing something like this which complain failure of Cloudera manager agent to provide the heartbeat and such, your probably run into a hostname issue. You may want to fix the hostname entry by referencing <a href="https://community.cloudera.com/t5/Cloudera-Manager-Installation/Cloudera-5-installation-on-CentOS-Installation-failed-Failed-to/td-p/8692">this link</a> Check the hostname on the server and compare it with the one shown in the installer web GUI, if they are different then you probably want to follow below procedure to refresh the cached hostname (ref: <a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_rn_known_issues.html">link</a>). It took me like an hour to figure out this painful workaround.<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;">Installing</span> <span style="color: #d0d0d0;">on</span> <span style="color: #d0d0d0;">AWS,</span> <span style="color: #d0d0d0;">you</span> <span style="color: #d0d0d0;">must</span> <span style="color: #d0d0d0;">use</span> <span style="color: #d0d0d0;">private</span> <span style="color: #d0d0d0;">EC2</span> <span style="color: #d0d0d0;">hostnames.</span>
<span style="color: #d0d0d0;">When</span> <span style="color: #d0d0d0;">installing</span> <span style="color: #d0d0d0;">on</span> <span style="color: #d0d0d0;">an</span> <span style="color: #d0d0d0;">AWS</span> <span style="color: #d0d0d0;">instance,</span> <span style="color: #6ab825; font-weight: bold;">and</span> <span style="color: #d0d0d0;">adding</span> <span style="color: #d0d0d0;">hosts</span> <span style="color: #d0d0d0;">using</span> <span style="color: #d0d0d0;">their</span> <span style="color: #d0d0d0;">public</span> <span style="color: #d0d0d0;">names,</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">installation</span> <span style="color: #d0d0d0;">will</span> <span style="color: #d0d0d0;">fail</span> <span style="color: #d0d0d0;">when</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">hosts</span> <span style="color: #d0d0d0;">fail</span> <span style="color: #d0d0d0;">to</span> <span style="color: #d0d0d0;">heartbeat.</span>
<span style="color: #d0d0d0;">Severity:</span> <span style="color: #d0d0d0;">Med</span>
<span style="color: #d0d0d0;">Workaround:</span>
<span style="color: #d0d0d0;">Use</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">Back</span> <span style="color: #d0d0d0;">button</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">wizard</span> <span style="color: #d0d0d0;">to</span> <span style="color: #6ab825; font-weight: bold;">return</span> <span style="color: #d0d0d0;">to</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">original</span> <span style="color: #d0d0d0;">screen,</span> <span style="color: #d0d0d0;">where</span> <span style="color: #d0d0d0;">it</span> <span style="color: #d0d0d0;">prompts</span> <span style="color: #6ab825; font-weight: bold;">for</span> <span style="color: #d0d0d0;">a</span> <span style="color: #d0d0d0;">license.</span>
<span style="color: #d0d0d0;">Rerun</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">wizard,</span> <span style="color: #d0d0d0;">but</span> <span style="color: #d0d0d0;">choose</span> <span style="color: #ed9d13;">"Use existing hosts"</span> <span style="color: #d0d0d0;">instead</span> <span style="color: #d0d0d0;">of</span> <span style="color: #d0d0d0;">searching</span> <span style="color: #6ab825; font-weight: bold;">for</span> <span style="color: #d0d0d0;">hosts.</span> <span style="color: #d0d0d0;">Now</span> <span style="color: #d0d0d0;">those</span> <span style="color: #d0d0d0;">hosts</span> <span style="color: #d0d0d0;">show</span> <span style="color: #d0d0d0;">up</span> <span style="color: #6ab825; font-weight: bold;">with</span> <span style="color: #d0d0d0;">their</span> <span style="color: #d0d0d0;">internal</span> <span style="color: #d0d0d0;">EC2</span> <span style="color: #d0d0d0;">names.</span>
<span style="color: #d0d0d0;">Continue</span> <span style="color: #d0d0d0;">through</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">wizard</span> <span style="color: #6ab825; font-weight: bold;">and</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">installation</span> <span style="color: #d0d0d0;">should</span> <span style="color: #d0d0d0;">succeed.</span>
</pre>
</div>
Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-36201431536108390982015-09-01T07:08:00.000-07:002015-09-10T07:09:20.122-07:00Cloudera manager agent installation fail due to missing of ntp packageSo was trying to explore Cloudera Hadoop by following <a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_install_path_b.html">this installation guide</a> and run into issue in bringing up cloudera manager agent (cloudera-scm-agent).<br />
<br />
The installation GUI was complaining with below message<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"> <span style="color: #d0d0d0;">Installation</span> <span style="color: #d0d0d0;">failed.</span> <span style="color: #d0d0d0;">Failed</span> <span style="color: #d0d0d0;">to</span> <span style="color: #d0d0d0;">receive</span> <span style="color: #d0d0d0;">heartbeat</span> <span style="color: #6ab825; font-weight: bold;">from</span> <span style="color: #447fcf; text-decoration: underline;">agent.</span>
<span style="color: #d0d0d0;">Ensure</span> <span style="color: #d0d0d0;">that</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">host</span><span style="color: #ed9d13;">'s hostname is configured properly.</span>
<span style="color: #d0d0d0;">Ensure</span> <span style="color: #d0d0d0;">that</span> <span style="color: #d0d0d0;">port</span> <span style="color: #3677a9;">7182</span> <span style="color: #6ab825; font-weight: bold;">is</span> <span style="color: #d0d0d0;">accessible</span> <span style="color: #d0d0d0;">on</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">Cloudera</span> <span style="color: #d0d0d0;">Manager</span> <span style="color: #d0d0d0;">Server</span> <span style="color: #d0d0d0;">(check</span> <span style="color: #d0d0d0;">firewall</span> <span style="color: #d0d0d0;">rules).</span>
<span style="color: #d0d0d0;">Ensure</span> <span style="color: #d0d0d0;">that</span> <span style="color: #d0d0d0;">ports</span> <span style="color: #3677a9;">9000</span> <span style="color: #6ab825; font-weight: bold;">and</span> <span style="color: #3677a9;">9001</span> <span style="color: #d0d0d0;">are</span> <span style="color: #d0d0d0;">free</span> <span style="color: #d0d0d0;">on</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">host</span> <span style="color: #d0d0d0;">being</span> <span style="color: #d0d0d0;">added.</span>
<span style="color: #d0d0d0;">Check</span> <span style="color: #d0d0d0;">agent</span> <span style="color: #d0d0d0;">logs</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">/var/log/cloudera-scm-agent/</span> <span style="color: #d0d0d0;">on</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">host</span> <span style="color: #d0d0d0;">being</span> <span style="color: #d0d0d0;">added</span> <span style="color: #d0d0d0;">(some</span> <span style="color: #d0d0d0;">of</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">logs</span> <span style="color: #d0d0d0;">can</span> <span style="color: #d0d0d0;">be</span> <span style="color: #d0d0d0;">found</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">the</span> <span style="color: #d0d0d0;">installation</span> <span style="color: #d0d0d0;">details).</span>
</pre>
</div>
<br />
<ul style="margin: 0px 0px 10px 25px; padding: 0px;"><span class="InstallStatus error" id="configurator2-failureHints" style="background-color: #f9f9f9; color: #b94a48; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; widows: 1;">
</span></ul>
<span class="InstallStatus error" id="configurator2-failureHints" style="background-color: #f9f9f9; color: #b94a48; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; widows: 1;">
</span>
<br />
<ul style="margin: 0px 0px 10px 25px; padding: 0px;">
</ul>
So looking at /var/log/cloudera-scm-agent/cloudera-scm-agent.log<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7
8
9
10
11
12
13</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;">[</span><span style="color: #3677a9;">10</span><span style="color: #d0d0d0;">/Sep/</span><span style="color: #3677a9;">2015</span> <span style="color: #3677a9;">07</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">57</span><span style="color: #d0d0d0;">:</span><span style="color: #3677a9;">16</span> <span style="color: #d0d0d0;">+</span><span style="color: #3677a9;">0000</span><span style="color: #d0d0d0;">]</span> <span style="color: #3677a9;">2366</span> <span style="color: #d0d0d0;">Monitor-HostMonitor</span> <span style="color: #d0d0d0;">throttling_logger</span> <span style="color: #d0d0d0;">ERROR</span> <span style="color: #d0d0d0;">Failed</span> <span style="color: #d0d0d0;">to</span> <span style="color: #d0d0d0;">collect</span> <span style="color: #d0d0d0;">NTP</span> <span style="color: #d0d0d0;">metrics</span>
<span style="color: #d0d0d0;">Traceback</span> <span style="color: #d0d0d0;">(most</span> <span style="color: #d0d0d0;">recent</span> <span style="color: #d0d0d0;">call</span> <span style="color: #d0d0d0;">last):</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/cmf/agent/src/cmf/monitor/host/ntp_monitor.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">37</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">collect</span>
<span style="color: #d0d0d0;">result,</span> <span style="color: #d0d0d0;">stdout,</span> <span style="color: #d0d0d0;">stderr</span> <span style="color: #d0d0d0;">=</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">._subprocess_with_timeout(args,</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">._timeout)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/cmf/agent/src/cmf/monitor/host/ntp_monitor.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">30</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">_subprocess_with_timeout</span>
<span style="color: #6ab825; font-weight: bold;">return</span> <span style="color: #d0d0d0;">subprocess_with_timeout(args,</span> <span style="color: #d0d0d0;">timeout)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/cmf/agent/src/cmf/subprocess_timeout.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">49</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">subprocess_with_timeout</span>
<span style="color: #d0d0d0;">p</span> <span style="color: #d0d0d0;">=</span> <span style="color: #d0d0d0;">subprocess.Popen(**kwargs)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/subprocess.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">642</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">__init__</span>
<span style="color: #d0d0d0;">errread,</span> <span style="color: #d0d0d0;">errwrite)</span>
<span style="color: #d0d0d0;">File</span> <span style="color: #ed9d13;">"/usr/lib64/python2.6/subprocess.py"</span><span style="color: #d0d0d0;">,</span> <span style="color: #d0d0d0;">line</span> <span style="color: #3677a9;">1234</span><span style="color: #d0d0d0;">,</span> <span style="color: #6ab825; font-weight: bold;">in</span> <span style="color: #d0d0d0;">_execute_child</span>
<span style="color: #6ab825; font-weight: bold;">raise</span> <span style="color: #d0d0d0;">child_exception</span>
<span style="color: #bbbbbb;">OSError</span><span style="color: #d0d0d0;">:</span> <span style="color: #d0d0d0;">[Errno</span> <span style="color: #3677a9;">2</span><span style="color: #d0d0d0;">]</span> <span style="color: #d0d0d0;">No</span> <span style="color: #d0d0d0;">such</span> <span style="color: #24909d;">file</span> <span style="color: #6ab825; font-weight: bold;">or</span> <span style="color: #d0d0d0;">directory</span>
</pre>
</td></tr>
</tbody></table>
</div>
<br />
So looking at /usr/lib64/cmf/agent/src/cmf/monitor/host/ntp_monitor.py, the culprit is there<br />
<br />
<div style="background: #202020; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"> <span style="color: #3677a9;">35</span> <span style="color: #6ab825; font-weight: bold;">try</span><span style="color: #d0d0d0;">:</span>
<span style="color: #3677a9;">36</span> <span style="color: #d0d0d0;">args</span> <span style="color: #d0d0d0;">=</span> <span style="color: #d0d0d0;">[</span><span style="color: #ed9d13;">"ntpdc"</span><span style="color: #d0d0d0;">,</span> <span style="color: #ed9d13;">"-np"</span><span style="color: #d0d0d0;">]</span>
<span style="color: #3677a9;">37</span> <span style="color: #d0d0d0;">result,</span> <span style="color: #d0d0d0;">stdout,</span> <span style="color: #d0d0d0;">stderr</span> <span style="color: #d0d0d0;">=</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">._subprocess_with_timeout(args,</span> <span style="color: #24909d;">self</span><span style="color: #d0d0d0;">._timeout)</span>
</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #d0d0d0;">
</span></pre>
</div>
As a quick fix, do a yum install ntp should help getting rid of this error.Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-58701103742658466032013-07-30T23:37:00.001-07:002013-07-30T23:46:51.914-07:00Passed EX436 Clustering and Storage Management last sunday.For sake of self achievement, I planned to get RHCA for some time since 2011 and hopefully this could be done before end of 2013, if everything go smooth.<br />
<br />
Last sunday, I just finished EX436 which is my 2nd RHCA exam out of a series of 5. The score was 268 out of 300 which is pretty good enough for me. The major focus of EX436 is clustering and storage management which is an area that I have some experience on but would definitely love to improve. The exam itself is pretty interesting, challenging and fun. Like other famous RHCA blogger out there, I have the comment that the exam is not really tough given that you are well prepared with the topics.<br />
<br />
For those that would like to take this exam, no doubt the best bet would certainly be joining the training class provided by Redhat :-) . But for those that want to save some bucks, you gotta work on your own and a good start would be to revisit the Course Outline here (<a href="http://www.redhat.com/training/courses/rh436/course-exam-outline" target="_blank">link</a>). For me, I basically stick with the official Redhat guide of clustering, GFS/GFS2, Multipath, Fencing, LVM and CLVM(<a href="https://access.redhat.com/site/documentation/Red_Hat_Enterprise_Linux/?locale=en-US" target="_blank">all can be found here</a>) and keep practice on my own lab. And, base on latest outline, you may also want to check with XFS and Gluster ( I am not sure how can you find resources from redhat site though). For me, I didn't practise XFS and Gluster in my own lab as I was sticking with previous Course Outline which didn't include XFS and Gluster at all (!!!). So it is a big surprise when I saw those questions in the exam. However, due to my job duties, I did have few exposure on them and luckily those exposure helped me to survive in the exam.<br />
<br />
People may interest to know the distro or exact version in question, but due to N.D.A agreement I can't say which version it is here :-) . What I would say is , clustering, GFS/GFS2 don't have a major different on RHEL5/6, at least from exam perspective. For me, my lab was based on RHEL 5 and i didn't subscribe to any Redhat subscription service (I copied all rpm based off the ISO and create my own repository tree as well as yum repository configuration files to allow the lab machine to fetch the required packages). <br />
<br />
My another advise is to stay calm during the exam. During my exam, I was stucked on a particular task and was going a bit nervous, this lead me mistakenly reboot my host (!!) at the middle of the exam. For those that already took RHCSA/RHCE exam, you may know that your exam system is on a VM that sit on a physical machines dedicated to you. So in my situation I was rebooting my host that caused all my VM suspended and then at that point I didn't know what could happen. You know, the worse case would be a re-image of all exam VM and I have to rebuild everything in the remaining 2 hours out of 4 hours exam duration. Luckily all my VMs are still there after the host reboot and the only impact was a 15 minutes downtime on my exam environment (million thanks to the examiner who helped recovering my environment though).<br />
<br />
The upcoming exam for me would be<br />
<br />
EX442 Red Hat Enterprise System Monitoring and Performance Tuning Expertise Exam <br />
EX333 Red Hat Enterprise Security: Network Services Expertise Exam<br />
EX401 Red Hat Enterprise Deployment and Systems Management Expertise Exam<br />
<br />
Hopefully I would be taking EX442 on Sept if everything go smooth. EX442 was well known among RHCA-er for its complexity so I would look forward to give a try on it.Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-58449684663376998752013-07-24T23:59:00.000-07:002013-07-31T00:03:38.161-07:00A quick and dirty munin plugins to count number of VM running on RHEL/CentOS based KVM host.So recently I was configuring munin to monitor some QEMU/KVM hosts which based on generic RHEL servers (Noted, not RHEV) which run libvirtd and QEMU/KVM.<br />
<br />
So here is a plugins that I created, it is quick and dirty but this should work as expected. Just copy and paste the plugins file into /etc/munin/plugins/ directory and make sure it is executable (755, ideally), then you should be good.<br />
<br />
So here is the content of the file.<br />
<br />
<i>[root@localhost plugins]# cat /etc/munin/plugins/vm_count <br />#!/bin/sh<br /><br />case $1 in<br /> config)<br /> cat <<'EOM'<br />graph_title Number of VMs<br />graph_vlabel VMcount<br />vmcount.label VMcount<br />vmcount.graph_category Vserver<br />EOM<br /> exit 0;;<br />esac<br /><br />i=`ps auxww | grep [/]usr/libexec/qemu-kvm | wc -l`<br />echo -n "vmcount.value "<br />echo $i</i><br />
<br />
And it is how it would work.<br />
<br />
# You should be able to execute it directly from system shell. In this example I had 17 VMs running on the host.<br />
<br />
<i>[root@localhost plugins]# pwd<br />/etc/munin/plugins</i><br />
<i>[root@localhost plugins]# ./vm_count <br />vmcount.value 17</i><br />
<br />
# Alternatively, you can test it with munin-run. This is how the script will look like when it is being loaded<br />
<br />
<i>[root@localhost plugins]# munin-run vm_count <br />vmcount.value 17</i><br />
<br />
<br />
<br />
<br />
# And here is the parameters of this plugins.<br />
<br />
<i>[root@localhost plugins]# munin-run vm_count config<br />graph_title Number of VMs<br />graph_vlabel VMcount<br />vmcount.label VMcount<br />vmcount.graph_category Vserver</i>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-83458855827673407892013-07-22T21:01:00.000-07:002013-07-22T21:01:57.885-07:00Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone NSo, because of my job duty and I have to deal with Citrix Cloudstack day by day. Recently we are deploying a new advanced zone and for some reason we are seeing errors like this during deploy of our first VM instance.<br />
<br />
<br />
<i>2013-07-22 22:09:27,625 WARN [api.commands.DeployVMCmd] (Job-Executor-50:job-534828) Exception:<br />com.cloud.exception.AgentUnavailableException: Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N</i><br />
<i>................<br />Caused by: com.cloud.utils.exception.CloudRuntimeException: Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N<br />................<br />2013-07-22 22:09:27,626 WARN [cloud.api.ApiDispatcher] (Job-Executor-50:job-534828) class com.cloud.api.ServerApiException : Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N</i><br />
<br />
So, basically, what Cloudstack doing is to<br />
1. check if there is any valid systemvm template (in this case systemvm-kvm-3.0.0) deployed to the zone. <br />
2. If things works as it should, you should be able to find the installed/downloaded template from table cloud.vm_template, cloud.template_zone_ref and template_host_ref. Hence, if you scan through the template list from the Web GUI, you should be able to see the template be downloaded.<br />
<br />
In my case, the template was not downloaded as it should (or marked as downloaded at DB layer), and if you look at the table cloud.template_host_ref, there is some abnormality here.<br />
<br />
<i>mysql> select * from template_host_ref where id=11111\G<br />*************************** 1. row ***************************<br /> id: 11111<br /> host_id: *masked*<br /> template_id: *masked*<br /> created: 2013-07-18 17:50:43<br /> last_updated: 2013-07-22 20:04:52<br /> job_id: 75a75e55-5280-4ba5-b823-cadbcbe2cc7a<br /> download_pct: 0<br /> size: 0<br /> physical_size: 0<br /><span style="color: red;">download_state: DOWNLOAD_ERROR<br /> error_str: No route to host</span><br /> local_path: /mnt/SecStorage/04ab8f0b-c4e0-34a4-80b3-457c433acde3/template/tmpl/2/1686/dnld6951269530983090325tmp_<br /> install_path: NULL<br /> url: http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2<br /> destroyed: 0<br /> is_copy: 0</i><br />
<br />
So, basically the things are 1) download_pct is 0 (while it should be 100 if download succeed), 2) download_state is DOWNLOAD_ERROR (while it should be DOWNLOADED if download successed and 3) error_str is "No route to host".<br />
<br />
In my case, the template installation procedures was not completed (though I have completed the cloud-install-sys-tmplt script per official installation guide), at least at DB layer.<br />
<br />
So I double checked the secondary storage to make sure the template file is completely downloaded (IMPORTANT!!!, if the file is not there, go through installation guide and re-run cloud-install-sys-tmplt script) and hacked the DB by updating the cloud.template_host_ref table. (Replace "N" with the correct account id and template id respectively)<br />
<br />
<i>mysql> updated template_host_ref set download_pct=100, download_state='DOWNLOADED', error_str=NULL, localpath='template/tmpl/N/N' where id=11111\G</i><br />
<i>*************************** 1. row ***************************<br />Query OK, 1 row affected (0.00 sec)<br />Rows matched: 1 Changed: 1 Warnings: 0</i><br />
<br />
Now cloudstack could launch VM as it should.Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-88348111050998722112013-07-20T21:01:00.003-07:002013-07-30T23:38:06.984-07:00EX436: Add iptables rule to allow fence_xvmdAssuming you want to make sure of fence_xvmd to do the VM fencing and you have iptables enabled, you may see issue while fence_xvm (client) send request to fence_xvmd (server). Here is an example,<br />
<br />
Here is an example of fence_xvmd (server side)running on dom0 and the multicast address is on 225.0.0.12 (which is the default if option "-a" is not defined when you start fence_xvmd)<br />
<br />
<i>[root@dom0 images]# fence_xvmd -L -X -fd -I eth0<br />-- args @ 0x7fff92fb76d0 --<br /> args->addr = 225.0.0.12<br /> args->domain = (null)<br /> args->key_file = /etc/cluster/fence_xvm.key<br /> args->op = 2<br /> args->hash = 2<br /> args->auth = 2<br /> args->port = 1229<br /> args->ifindex = 5<br /> args->family = 2<br /> args->timeout = 30<br /> args->retr_time = 20<br /> args->flags = 259<br /> args->debug = 1<br />-- end args --<br />My Node ID = 1<br />Domain UUID Owner State<br />------ ---- ----- -----<br />Domain-0 00000000-0000-0000-0000-000000000000 00001 00001<br />iscsitgt 743affaf-eae7-6e40-0d1d-e3a3bb1b1eaf 00001 00002<br />lab1 20a6e8b6-26a6-a700-b656-63b72b0a407e 00001 00002<br />lab2 cb3f49a8-9841-d917-50ab-97425d900da4 00001 00002<br />Storing iscsitgt<br />Storing lab1<br />Storing lab2</i><br />
<br />
<br />
So assuming you now fence the vm "lab1" from vm 'lab2" with fence_xvm (the client), you will be seeing something like this. Apparently, the fence_xvm request doesn't seem to connect to fence_xvmd (the fence server) and it keep complaining "Waiting for connection from XVM host daemon."<br />
<i><br /></i>
<i>[root@lab2 ~]# fence_xvm -ddd -H lab1<br />Debugging threshold is now 3<br />-- args @ 0x7fffebce4540 --<br /> args->addr = 225.0.0.12<br /> args->domain = lab1<br /> args->key_file = /etc/cluster/fence_xvm.key<br /> args->op = 2<br /> args->hash = 2<br /> args->auth = 2<br /> args->port = 1229<br /> args->ifindex = 0<br /> args->family = 2<br /> args->timeout = 30<br /> args->retr_time = 20<br /> args->flags = 0<br /> args->debug = 3<br />-- end args --<br />Reading in key file /etc/cluster/fence_xvm.key into 0x7fffebce34f0 (4096 max size)<br />Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1<br />Sending to 225.0.0.12 via 192.168.0.202<br />Sending to 225.0.0.12 via 192.168.0.199<br />Sending to 225.0.0.12 via 10.0.0.202<br />Sending to 225.0.0.12 via 172.16.0.202<br />Sending to 225.0.0.12 via 172.16.1.202<br />Waiting for connection from XVM host daemon.<br />Sending to 225.0.0.12 via 127.0.0.1<br />Sending to 225.0.0.12 via 192.168.0.202<br />Sending to 225.0.0.12 via 192.168.0.199<br />Sending to 225.0.0.12 via 10.0.0.202<br />Sending to 225.0.0.12 via 172.16.0.202<br />Sending to 225.0.0.12 via 172.16.1.202<br />Waiting for connection from XVM host daemon.</i><br />
<br />
In my scenario, the xen host is enabled with iptables and looking at xvmd side, there is no fence request coming in too. That seems like the fence request was filtered.<br />
<br />
To allow the fence request to get in via multicast traffic, we can add below rule to allow the traffic. <br />
<br />
<i># iptables -I INPUT -d 225.0.0.12 -p udp -m udp --dport 1229 -j ACCEPT </i><br />
<br />
Given that the fence_xvmd listen on default ip (225.0.0.12) and port (udp 1229).<br />
<br />
Once the rule is added, you can retry fencing and now you would see something similar to this.<br />
<br />
<i>[root@lab2 ~]# fence_xvm -ddd -H lab1<br />Debugging threshold is now 3<br />-- args @ 0x7fffb74cc2a0 --<br /> args->addr = 225.0.0.12<br /> args->domain = lab1<br /> args->key_file = /etc/cluster/fence_xvm.key<br /> args->op = 2<br /> args->hash = 2<br /> args->auth = 2<br /> args->port = 1229<br /> args->ifindex = 0<br /> args->family = 2<br /> args->timeout = 30<br /> args->retr_time = 20<br /> args->flags = 0<br /> args->debug = 3<br />-- end args --<br />Reading in key file /etc/cluster/fence_xvm.key into 0x7fffb74cb250 (4096 max size)<br />Actual key length = 4096 bytesSending to 225.0.0.12 via 127.0.0.1<br />Sending to 225.0.0.12 via 192.168.0.202<br />Sending to 225.0.0.12 via 192.168.0.199<br />Sending to 225.0.0.12 via 10.0.0.202<br />Sending to 225.0.0.12 via 172.16.0.202<br />Sending to 225.0.0.12 via 172.16.1.202<br />Waiting for connection from XVM host daemon.<br />Issuing TCP challenge<br />Responding to TCP challenge<br />TCP Exchange + Authentication done... <br />Waiting for return value from XVM host<br />Remote: Operation was successful</i>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-71316521632648679482013-06-30T20:12:00.000-07:002013-08-01T20:12:23.458-07:00Getting UUID depedency failure when installing Vmware-Vsphere-CLI for LinuxTrying to install Vmware Vsphere CLI for Linux to allow some automation scripts to work and I am hitting below failure<br />
<br />
++++++++++++++++++++++++++++++++++++++++++++++++++++++ <br />
<i>CPAN is downloading and installing pre-requisite Perl module "UUID" .<br /><br />CPAN not able to install following Perl modules on the system. These must be <br />installed manually for use by vSphere CLI:<br /><br />UUID 0.03 or newer </i>++++++++++++++++++++++++++++++++++++++++++++++++++++++<br />
<br />
So I try to manually install the UUID CPAN modules but I am hitting the same failure too.<br />
<br />
++++++++++++++++++++++++++++++++++++++++++++++++++++++<br />
<i>Writing Makefile for UUID<br />Could not read metadata file. Falling back to other methods to determine prerequisites<br />cp UUID.pm blib/lib/UUID.pm<br />/usr/bin/perl /usr/share/perl5/ExtUtils/xsubpp -typemap /usr/share/perl5/ExtUtils/typemap UUID.xs > UUID.xsc && mv UUID.xsc UUID.c<br />gcc -c -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DVERSION=\"0.05\" -DXS_VERSION=\"0.05\" -fPIC "-I/usr/lib64/perl5/CORE" UUID.c<br />UUID.xs:5:23: error: uuid/uuid.h: No such file or directory<br />UUID.xs: In function ‘do_generate’:<br />UUID.xs:13: error: ‘uuid_t’ undeclared (first use in this function)<br />UUID.xs:13: error: (Each undeclared identifier is reported only once<br />UUID.xs:13: error: for each function it appears in.)<br />UUID.xs:13: error: expected ‘;’ before ‘uuid’<br />UUID.xs:14: warning: implicit declaration of function ‘uuid_generate’<br />UUID.xs:14: error: ‘uuid’ undeclared (first use in this function)<br />UUID.xs: In function ‘do_unparse’:<br />UUID.xs:20: error: ‘uuid_t’ undeclared (first use in this function)<br />UUID.xs:20: error: expected ‘;’ before ‘uuid’<br />UUID.xs:23: warning: implicit declaration of function ‘uuid_unparse’<br />UUID.xs: In function ‘do_parse’:<br />UUID.xs:29: error: ‘uuid_t’ undeclared (first use in this function)<br />UUID.xs:29: error: expected ‘;’ before ‘uuid’<br />UUID.xs:33: warning: implicit declaration of function ‘uuid_parse’<br />UUID.xs:33: error: ‘uuid’ undeclared (first use in this function)<br />UUID.xs:30: warning: unused variable ‘str’<br />make: *** [UUID.o] Error 1<br /> LZAP/UUID-0.05.tar.gz<br /> /usr/bin/make -- NOT OK<br />Running make test<br /> Can't test without successful make<br />Running make install<br /> Make had returned bad status, install seems impossible<br />Failed during this command:<br /> LZAP/UUID-0.05.tar.gz : make NO</i><br />
++++++++++++++++++++++++++++++++++++++++++++++++++++++<br />
<br />
As I was on a CentOS 6 box, at first glance I was thinking it missed uuid-devel so I go ahead and install the package via yum but the same failure occurred during modules complication.<br />
<br />
<br />
<i>UUID.xs:5:23: error: uuid/uuid.h: No such file or directory</i><br />
<br />
With deeper search, looks like uuid/uuid.h is on libuuid-devel package instead of uuid-devel (stupid me). So I go ahead to install that and now I could proceed with Vmware Vsphere CLI package install.Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-17310803027220116622013-01-16T23:53:00.002-08:002013-01-16T23:53:43.643-08:00Could not load host key: /etc/ssh/ssh_host_rsa_key, SSH Connection closed by x.x.x.xBefore a ssh host to accept a ssh client request, the host have to be installed with SSH host key. Usually if you are installing the ssh servers (which usually come as default package in most Linux distro) from a package (e.g. rpm or deb), the installation script will take care the ssh host key generation for you.<br />
<br />
However, in any cases you removed the ssh host key (usually sit under /etc/ssh/), you will no longer be able to get into the machine via ssh until the host key is generated.<br />
<br />
So this is what you get when your ssh host is gone.<br />
<br />
<i>$ ssh root@x.x.x.x<br />Connection closed by x.x.x.x</i><br />
<br />
And if you have a console access to your machine, you will be seeing something like this on /var/log/auth.log (or /var/log/messages ... depends on your syslog configuration).<br />
<i><br /></i>
<i>Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key<br />Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key<br />Jan 17 07:42:38 localhost sshd[26509]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key<br />Jan 17 07:42:38 localhost sshd[26509]: fatal: No supported key exchange algorithms [preauth]</i><br />
<br />
In order to fix this, ssh host key have to be regenerated <br />
<i><br /></i>
<i>ssh-keygen -t rsa1 -f /etc/ssh/ssh_host_rsa_key -N ‘’<br />ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ‘’<br />ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ‘’</i><br />
<br />
The option -N refers to the new passphrase to be used. In above example '' means empty. Should you want a passphrase be assigned to the key, you may want to put it like <i>-N 'your_pass_phrase'</i>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-33276980151776444792012-12-26T00:38:00.000-08:002013-01-17T00:41:04.631-08:00Nginx X-Forwarded-Protocol and X-Forwarded-ForI have a client that had multiples apache web servers sit behind the Nginx web load balancer. Currently both http and https requests are terminated on Nginx and the requests will then be proxied to the backend apache web servers at port 80 (i.e. http).<br />
<br />
From backend web servers perspective, all traffic coming in are sort of masqueraded by the Nginx , web server could only see the requests are made by Nginx and the protocol was http. So my client would interest to know which protocol the original request was, whether it is http or the ssl-encrypted https. <br />
<br />
Nginx do allow customization on proxy header via proxy_set_header attributes. So I added below parameter to the location block so that extra header will be passed to the backend web server.<br />
<br />
<br />
Here is the reverse proxy configuration<br />
<i> upstream </i><i>backend_web_server_pool</i><i> {<br /> server 1.2.3.4:80;</i><br />
<i> server 1.2.3.5:80;</i><br />
<i> }</i><br />
<i><br /></i>
Here is the http site configuration<i><br /></i><br />
<i>server {<br /> listen 80; # The http server</i><br />
<i> ....</i>
<i> </i><br />
<i> location / {<br /> proxy_pass http://backend_web_server_pool;<br /><span style="color: red;"> proxy_set_header X-Forwarded-Protocol "http" ;</span><br style="color: red;" /><span style="color: red;"> proxy_set_header X-Forwarded-For $remote_addr;</span><br /> }</i><br />
<i>} </i><br />
<br />
<br />
<br />
<br />
Here is the https site configuration<i>
</i><br />
<i>server {<br />
listen 443; </i><i># The https server</i><br />
<i>
ssl on;</i><br />
<i> ....<br />
</i><br />
<i> location / {<br />
proxy_pass http://backend_web_server_pool;<br /><span style="color: red;">
proxy_set_header X-Forwarded-Protocol "https" ;</span><br style="color: red;" /><span style="color: red;">
proxy_set_header X-Forwarded-For $remote_addr;</span><br style="color: red;" /><span style="color: red;">
</span> }</i><br />
<i>} </i><br />
<br />
<br />
So the line <i>proxy_set_header X-Forwarded-Protocol "http"</i> will pass a header named "X-Forwarded-Protocol" and its value "http" to the backend web server. You can replace this header value to any arbitrary value, e.g. "xyz123". After all it is just a placeholder or symbol to let you know where the request came from. The same logic applies to the HTTPS block however you may want to replace the value from "http" to "https" to avoid confusion. The line <i>proxy_set_header X-Forwarded-For $remote_addr </i>pass the variable remote_addr (i.e. the remote client IP address) to the backend web server.<br />
<br />
Once the above configuration applied, restart nginx and then we can head to reconfigure the log format configuration on apache web server. We will now modify the combined log format to capture the X-Forwarded-For and X-Forwarded-Protocol.<br />
<br />
<i>#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined<br />LogFormat "%h <span style="color: red;">%{X-Forwarded-For}i %{X-Forwarded-Protocol}i</span> %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined</i><br />
<br />
I added the %{X-Forwarded-For}i and %{X-Forwarded-Protocol}i to the combined log format followed by apache restart and now apache log could capture the client IP address and the original protocol.<br />
<br />
<br />Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-8705900636004714882012-11-23T01:20:00.001-08:002012-11-23T01:20:09.705-08:00dnsmasq, assign duplicate DHCP subnet via single interfaceI was troubleshooting a dnsmasq DHCP server back in few days and notice special settings have to be applied if the particular dnsmasq DHCP server is configured to serve multiple subnet.<br />
<br />
In my scenario, the DHCP server is having one nic, eth0. For some reason, there are 2 subnets being served, 192.168.0.0/24 and 192.168.1.0/24. The eth0 is configured as 192.168.0.1/24 while an additional ip 192.168.1.1 is added to eth0 too.<br />
<br />
So DHCP clients that connect to the eth0 of DHCP server (through switches) could retrieve IP from same subnet without any issue, however it seems like the DHCP gateway of the 2nd subnet is acting weird. For example, here is the DHCP lease file i got from one of the DHCP client.<br />
<i><br /></i>
<i># cat /var/lib/dhcp/pump.lease <br />Device eth0<br /><span style="color: blue;"> IP: 192.168.1.10</span><br style="color: blue;" /><span style="color: blue;"> Netmask: 255.255.255.0</span><br style="color: blue;" /><span style="color: blue;"> Broadcast: 192.168.1.255</span><br style="color: blue;" /><span style="color: blue;"> Network: 192.168.1.0</span><br style="color: blue;" /><span style="color: blue;"> Boot server 192.168.1.1</span><br style="color: blue;" /><span style="color: blue;"> Next server 192.168.1.1</span><br /><b style="color: red;"> Gateways: 192.168.0.1</b><br /> Hostname: test-dhcp-client<br /> Domain: test.internal<br /> Renewal time: Fri Nov 23 17:38:13 2012<br /> Expiration time: Fri Nov 23 19:08:13 2012</i><br /><br />
Interestingly, this DHCP client is assigned with the 2nd DHCP subnet (i.e. 192.168.1.0/24) however gateway of default subnet (192.168.0.0/24) is being assigned.<br />
<br />
To fix this, I tried couples of approach but eventually it looks like configuring the option though dhcp-range tagging would do the best. Below are the configuration snippet that fixed the problem.<br />
<br />
listen-address=192.168.0.1<br />listen-address=192.168.1.1<br />dhcp-range=set:1stblock,192.168.1.10,192.168.0.50,255.255.255.0<br />
dhcp-range=set:2ndblock,192.168.1.1,192.168.1.50,255.255.255.0<br />dhcp-option=tag:1stblock,option:router,192.168.0.1<br />dhcp-option=tag:2ndblock,option:router,192.168.1.1<br />
<br />
So in above example, I assign a tag to each individual subnet (i.e 1stblock -> 192.168.0.0/24, 2ndblock -> 192.168.1.0/24) and then assign individual router ip with each associated tag.Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-73829420547719830472012-10-25T02:55:00.001-07:002012-10-25T02:55:57.540-07:00QEMU/KVM atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)So I am getting this annoying warning message when I press Enter on my KVM VM.<br />
<br />
<i># </i><br />
<i>atkbd.c: Unknown key pressed (translated set 2, code 0xa0 on isa0060/serio0)</i><br />
<i>atkbd.c Use 'setkeycodes 00 <keycode>' to make it known.</i><br />
<br />
So now whenever I hit enter, the annoying warning will be coming up and this can be easily cleaned up by using the showkey and setkeycodes command.<br />
<br />
So from the tty console (it may not work on x-windows), i executed showkey<br />
<br />
<i># showkey</i><br />
<i>kb mode was UNICODE</i><br />
<i>[ if you are trying this under X, it might not work</i><br />
<i>since the X server is also reading /dev/console ]</i><br />
<br />
press any key (program terminates 10s after last keypress)...<br />
<br />
Now you can press 'Enter' (or whatever key that could cause the annoying message), it will show the keycode<br />
<i><br /></i>
<i>keycode 28 press</i><br />
<br />
It means that the Enter key is associated to keycode 28. Now you can use setkeycodes to map the Enter key.<br />
<br />
<i># setkeycodes 0x00 28</i><br />
<br />
0x00 here is the scancode, if you are interested on what it means, you could google it and there is some other articles out there explaining this.<br />
<br />
Now, right after you typed the command, you should now be safed from this annoying warning message.<br />
<br />
Should you want this be sustained across reboot, you may want to add it to /etc/rc.local<br />
<br />
<i># echo 'setykeycodes 0x00 28' >> /etc/rc.local</i><br />
<br />
<br />Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-67384631628755928902012-08-16T03:09:00.000-07:002012-10-10T03:10:20.106-07:00Using gawk (GNU awk) to monitor /var/log/messagesI have a requirement to periodically scan through /var/log/messages to catch specific error message. The error itself is very time-sensitive and I want to be informed as soon as the message is shown on the log, ideally in a minute interval.<br />
<br />
Usually people would suggest to periodically run grep against the log which is the simplest way to facilitate the need but it doesn't work for my scenario. The problem to simply run grep is that it may also capture unnecessary information. For example, I have below logs on my /var/log/messages.<br />
<br />
Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal<br />
Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal<br />
<br />
If I simply use grep to catch above log entry in a minute interval, I will probably be informed at 12:40.06 and then all the way till the end of the day (i.e. when the /var/log/messages be rotated and cleaned up). Not to mention starting from 15:05:28, the grep script will catch 2 errors from the log, the lines being recorded at 12:40:05 and 15:05:28. However, the fact is that I actually need the one on 15:05:28. <br />
<br />
After some research, it looks like gawk (GNU awk) would be the perfect tool to solve the problem. Eventually I come up with below gawk script to read /var/log/messages.<br />
<br />
<br />
<i>$ cat gawk-script</i><br />
<i>BEGIN {</i><br />
<i> # Declare field seperator.<br /> FS="[- :.]";</i><br />
<br />
<i> </i><br />
<i> # Generate the timestamp<br /> NOW=systime();</i><br />
<i> </i><br />
<i> # The time interval that I want this script to read from</i><br />
<i> # As long as I will run this script with cron in every minutes</i><br />
<i> # I set the time interval to 1sec*60</i><br />
<i> # So, for example when the script is run at 12:05, only log entries between 12:04 ~ 12:05 will be read.<br /> PAST=NOW-(60);</i><br />
<br />
<i> </i><br />
<i> # /var/log/messages start with the line in format of "Month date" (e.g Mar 23)</i><br />
<i> # Ideally we can convert it to something like 23/3 but I want to be lazy, </i><br />
<i> # so I simply make use of the system time library and </i><br />
<i> # picked %Y %m to represent the Year and Month attributes.<br /> format="%Y %m";</i><br />
<i> # LOGMTH will be something like "2012 10" (Oct 2012)</i><br />
<i> # This will be used later to generate the timestamp of the log entry.<br /> LOGMTH=strftime(format, THEN)<br />}<br />{</i><br />
<i> # Read the line of the log and convert it to a timestamp<br /> LOGTIME=mktime(LOGMTH " " $2 " " $3 " " $4 " " $5);</i><br />
<i> # Below 3 lines can be commented out to debug with the value being read. <br /> #{print $2, $3, $4, $5};<br /> #{print LOGMTH};<br /> #{print LOGTIME};</i><br />
<i> </i><br />
<i> # print the log if the timestamp of the line is between PAST <> NOW.<br /> if(PAST<LOGTIME){print}<br />} </i><br />
<br />
So here is a demo of the script.<br />
<br />
<i>$ cat /var/log/messages ### totally 4 lines here</i><br />
<i>Aug 3 12:40:06 test-box kernel: [1645652.295156] CPU0: Temperature/speed normal</i><br />
<i>Aug 3 15:05:28 test-box kernel: [1645673.980296] CPU1: Temperature/speed normal </i><i>Aug 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28<br />Aug 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices</i><br />
<br />
<i>$ date #Lets check the current time<br />Wed Oct 3 18:04:24 HKT 2012</i><br />
<i><br /></i>
<i>$ gawk -f gawk-script /var/log/messages<br />Oct 3 18:03:31 test-box kernel: [1640277.321612] usb 2-2: new high speed USB device using ehci_hcd and address 28<br />Oct 3 18:03:32 test-box kernel: [1640277.474483] usb 2-2: configuration #1 chosen from 4 choices</i><br />
<br />
So, the script now only print everything captured within last minute. You can then use the script together with grep utility to catch the string you are interested.Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-47907349359429280472012-07-24T20:34:00.000-07:002012-09-20T20:35:22.108-07:00Natting TCP port 2000 behind Cisco device.Recently I am mudding on a storage appliance named NexentaStor. NexentaStor is based on OpenSolaris and make use of ZFS implementation which looks pretty promising. It has a clean and easy to use GUI, hence it supports quite a lot of storage protocol like NFS, CIFS, ISCSI and even support Link aggregation on network layer too.<br />
<br />
Everything is going smooth so far just one minor obstacles, that is its Web GUI by default listen on tcp port 2000. Basically tcp port 2000 is a valid port but somehow I am not able to access the Web GUI and the connection towards the port keep timing out from outside, though the port is working on the same subnet. <br />
<br />
I started to suspect there is something to do with NAT and yeah it is. I put my NexentaStor server behind a Cisco ASA firewall with NAT enabled. However, it looks like the port 2000 traffic of NexentaStor overlapped with the Cisco SCCP (http://en.wikipedia.org/wiki/Skinny_Call_Control_Protocol) on port 2000 too. Eventually I have to change the port of the GUI to a non-2000 port. <br />
<br />
Just an additional notes, to reconfigure NexentaStor Web GUI port, I have to get into the console and execute below command.<br />
<i><br />nmc@myhost:/$ setup appliance init </i>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-74273399664273062192012-06-24T22:31:00.000-07:002012-08-05T22:31:20.967-07:00HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 6<b>Resources Management </b><br />
Below examples show how could one mange the HA resources in between the nodes.<br />
<br />
<b>- Check Cluster status</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm_mon -1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last updated: Fri
Dec 30 00:43:51 2011</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last change: Fri Dec
30 00:20:38 2011 via crm_attribute on dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Stack: openais</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Current DC:
dbmaster-01.localdomain - partition with quorum</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Version:
1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>2 Nodes configured,
2 expected votes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>5 Resources
configured.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Online: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Resource Group:
dbGroup</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> ClusterIP
(ocf::heartbeat:IPaddr2): Started dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> DBstore
(ocf::heartbeat:Filesystem): Started dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> MySQL
(ocf::heartbeat:mysql): Started dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Clone Set:
pingclone [check-ext-conn]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Started: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Put node to offline mode</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm node standby</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm_mon -1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last updated: Fri
Dec 30 00:44:45 2011</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last change: Fri Dec
30 00:44:39 2011 via crm_attribute on dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Stack: openais</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Current DC:
dbmaster-01.localdomain - partition with quorum</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Version:
1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>2 Nodes configured,
2 expected votes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>5 Resources
configured.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Node
dbmaster-01.localdomain: standby</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Online: [
dbmaster-02.localdomain ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Resource Group:
dbGroup</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> ClusterIP
(ocf::heartbeat:IPaddr2): Started dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> DBstore
(ocf::heartbeat:Filesystem): Started dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> MySQL
(ocf::heartbeat:mysql): Started dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Clone Set:
pingclone [check-ext-conn]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Started: [
dbmaster-02.localdomain ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Stopped: [
check-ext-conn:0 ]</i></div>
<br />
<b>- Put node to online mode</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm node online</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm_mon -1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last updated: Fri
Dec 30 00:45:12 2011</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last change: Fri Dec
30 00:45:10 2011 via crm_attribute on dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Stack: openais</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Current DC:
dbmaster-01.localdomain - partition with quorum</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Version:
1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>2 Nodes configured,
2 expected votes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>5 Resources
configured.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Online: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Resource Group:
dbGroup</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> ClusterIP
(ocf::heartbeat:IPaddr2): Started dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> DBstore
(ocf::heartbeat:Filesystem): Started dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> MySQL
(ocf::heartbeat:mysql): Started dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Clone Set:
pingclone [check-ext-conn]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Started: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>
<br />
<b>- Migrate resources to neighbor node</b><br />
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm resource migrate dbGroup dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm_mon -1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last updated: Fri
Dec 30 00:47:50 2011</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last change: Fri Dec
30 00:47:35 2011 via crm_resource on dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Stack: openais</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Current DC:
dbmaster-01.localdomain - partition with quorum</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Version:
1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>2 Nodes configured,
2 expected votes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>5 Resources
configured.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Online: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Resource Group:
dbGroup</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> ClusterIP
(ocf::heartbeat:IPaddr2): Started dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> DBstore
(ocf::heartbeat:Filesystem): Started dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> MySQL
(ocf::heartbeat:mysql): Started dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Clone Set:
pingclone [check-ext-conn]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> Started: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>
<br />
<b>- Start / Stop / Restart specific resouce on node</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm resource status MySQL</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>resource MySQL is
running on: dbmaster-01.localdomain
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><a href="http://www.blogger.com/blogger.g?blogID=8392021181198238917" name="_GoBack"></a>
</i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>...</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm resource stop MySQL</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>....</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm resource start MySQL</i></div>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-16764268609225510092012-06-21T22:26:00.000-07:002012-08-05T22:26:35.108-07:00HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 5<b>Cluster management</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="margin-bottom: 0in;">
Corosync service is responsible for
Cluster management while pacemaker is responsible for resource on top
of the clustering service. The dependency of startup sequence will
be 1) corosync and then 2) pacemaker. The shutdown sequence will be
1) pacemaker and then 2) corosync</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>- Check service status</b></div>
<div style="margin-bottom: 0in;">
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-02
~]# /etc/init.d/corosync status</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>corosync (pid
23118) is running...</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-02
~]# /etc/init.d/pacemaker status</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>pacemakerd (pid
8714) is running...</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Stop pacemaker and corosync</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="margin-bottom: 0in;">
If the subject node is in active state,
resources will be failed over to standby node. Alternatively if it is
standby node, no changes will be made on active node. </div>
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-02
~]# /etc/init.d/pacemaker stop</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Signaling Pacemaker
Cluster Manager to terminate: [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Waiting for cluster
services to unload:....... [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-02
~]# /etc/init.d/corosync stop</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Signaling Corosync
Cluster Engine (corosync) to terminate: [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Waiting for corosync
services to unload:. [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Start corosync and pacemaker</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="margin-bottom: 0in;">
If there isn’t any node running in
the cluster, the first up shown up in the cluster will be the active
node. If there is one active node in the cluster, the 2<sup>nd</sup>
node will automatically become the standby</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-02
~]# /etc/init.d/corosync start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting Corosync
Cluster Engine (corosync): [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-02
~]# /etc/init.d/pacemaker start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting Pacemaker
Cluster Manager: [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-16986215126516764902012-06-18T22:15:00.000-07:002012-08-05T22:22:43.984-07:00HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 4<b>- Configure Cluster Resources</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="margin-bottom: 0in;">
Now the cluster is up, and we will have
to add cluster resources (e.g. VIP, MySQL DB store, MySQL DB service)
on top of the cluster. We only need to run this once on dbmaster-01 as the configuration changes will be written to cluster configuration file and being replicated to dbmaster-02.</div>
<br />
<b>- Configure misc cluster parameter</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure property stonith-enabled=false</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure property no-quorum-policy=ignore</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure property start-failure-is-fatal="false"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure rsc_defaults resource-stickiness=100</i></div>
<br />
<b>- Configure VIP </b><br />
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params
ip=192.168.0.10 cidr_netmask=32 op monitor interval=10s meta
migration-threshold="10"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Configure MySQL DB store, i.e. the shared-disk</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure primitive DBstore ocf:heartbeat:Filesystem params
device="/dev/sdb" directory="/mysql"
fstype="ext4" meta migration-threshold="10"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>WARNING: DBstore:
default timeout 20s for start is smaller than the advised 60</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>WARNING: DBstore:
default timeout 20s for stop is smaller than the advised 60</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Configure MySQL services</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure primitive MySQL ocf:heartbeat:mysql params
binary="/usr/bin/mysqld_safe" config="/etc/my.cnf"
user="mysql" group="mysql" datadir="/mysql"
log="/mysql/mysqld.log" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> op start
interval="0" timeout="60s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> op stop
interval="0" timeout="60s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> op monitor
interval="1min" timeout="60s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> meta
migration-threshold="10" target-role="Started"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>WARNING: MySQL:
specified timeout 60s for start is smaller than the advised 120</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>WARNING: MySQL:
specified timeout 60s for stop is smaller than the advised 120</i></div>
<br />
<b>- Configure all resources as a resource group for failover</b><br />
If we don't configure them as a resource group, indivdual resources will be failovered seperately so eventually you may be seeing VIP on dbmaster01 while DB store on dbmaster02 which is something we don't want to see.<br />
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure group dbGroup ClusterIP DBstore MySQL</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Define external ping monitoring and failover policy</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="margin-bottom: 0in;">
This part of configuration will be a
complicated, basically it means it will try to ping the gateway. In
case the active node failed to ping gateway (e.g. internet
connectivity down), it will fail over all services to standby node</div>
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure primitive check-ext-conn ocf:pacemaker:ping \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> params
host_list="192.168.0.1" multiplier="100"
attempts="3" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> op monitor
interval="10s" timeout="5s" start stop \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> meta
migration-threshold="10"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>WARNING:
check-ext-conn: default timeout 20s for start is smaller than the
advised 60</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>WARNING:
check-ext-conn: specified timeout 5s for monitor is smaller than the
advised 60</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure clone pingclone check-ext-conn meta
globally-unique="false"
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>crm(live)# configure</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>crm(live)configure#
location dbnode dbGroup \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> rule
$id="dbnode-rule" pingd: defined pingd \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> rule
$id="dbnode-rule-0" -inf: not_defined pingd or pingd lte 10
\</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> rule
$id="dbnode-rule-1" 20: uname eq dbmaster-01.localdomain \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> rule
$id="dbnode-rule-2" 20: uname eq dbmaster-01</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>crm(live)configure#
end</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>There are changes
pending. Do you want to commit them? Yes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>crm(live)configure#
exit</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>bye</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Review all configuration details.</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm configure show</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>node
dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>node
dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>primitive ClusterIP
ocf:heartbeat:IPaddr2 \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> params
ip="192.168.0.10" cidr_netmask="32" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> op monitor
interval="10s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> meta
migration-threshold="10"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>primitive DBstore
ocf:heartbeat:Filesystem \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> params
device="/dev/sdb" directory="/mysql"
fstype="ext4" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> meta
migration-threshold="10"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>primitive MySQL
ocf:heartbeat:mysql \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> params
binary="/usr/bin/mysqld_safe" config="/etc/my.cnf"
user="mysql" group="mysql" datadir="/mysql"
log="/mysql/mysqld.log" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> op start
interval="0" timeout="60s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> op stop
interval="0" timeout="60s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> op monitor
interval="1min" timeout="60s" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> meta
migration-threshold="10" target-role="Started"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>primitive
check-ext-conn ocf:pacemaker:ping \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> params
host_list="192.168.0.1" multiplier="100"
attempts="3" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> op monitor
interval="10s" timeout="5s" start stop \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> meta
migration-threshold="10"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>group dbGroup
ClusterIP DBstore MySQL</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>clone pingclone
check-ext-conn \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> meta
globally-unique="false"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>location dbnode
dbGroup \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> rule
$id="dbnode-rule" pingd: defined pingd \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> rule
$id="dbnode-rule-0" -inf: not_defined pingd or pingd lte 10
\</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> rule
$id="dbnode-rule-1" 20: uname eq dbmaster-01.localdomain \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> rule
$id="dbnode-rule-2" 20: uname eq dbmaster-01</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>property
$id="cib-bootstrap-options" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
dc-version="1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558"
\</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
cluster-infrastructure="openais" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
expected-quorum-votes="2" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
stonith-enabled="false" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
no-quorum-policy="ignore" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
start-failure-is-fatal="false"</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>rsc_defaults
$id="rsc-options" \</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
resource-stickiness="100"</i></div>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-70331359102104980692012-06-16T22:14:00.000-07:002012-08-05T22:15:13.623-07:00HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 3<b>- Cluster software installation and configuration</b><br />
<br />
Now it is time to proceed with cluster software installation and configuration. If you are installing on CentOS, the packages could be fetched from default yum repository but if you are doing it on a RHEL6, you will probably need to add CentOS repository.<br />
<br />
<br />
Below packages have to be installed on both nodes.<br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
yum.repos.d]# yum -y install pacemaker corosync</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Loaded plugins:
product-id, rhnplugin, subscription-manager</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Updating
certificate-based repositories.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Setting up Install
Process</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Resolving
Dependencies</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>--> Running
transaction check</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>---> Package
corosync.x86_64 0:1.4.1-4.el6 will be installed</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>**********************</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>***** detail skipped
****</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>**********************</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Installed:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> corosync.x86_64
0:1.4.1-4.el6 pacemaker.x86_64 0:1.1.6-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dependency
Installed:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> cifs-utils.x86_64
0:4.8.1-5.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
cluster-glue.x86_64 0:1.0.5-2.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
cluster-glue-libs.x86_64 0:1.0.5-2.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> clusterlib.x86_64
0:3.0.12.1-23.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> corosynclib.x86_64
0:1.4.1-4.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> keyutils.x86_64
0:1.4-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> libevent.x86_64
0:1.4.13-1.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> libgssglue.x86_64
0:0.1-11.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> libibverbs.x86_64
0:1.1.5-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> librdmacm.x86_64
0:1.0.14.1-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> libtalloc.x86_64
0:2.0.1-1.1.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> libtirpc.x86_64
0:0.2.1-5.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> nfs-utils.x86_64
1:1.2.3-15.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
nfs-utils-lib.x86_64 0:1.1.5-4.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
pacemaker-cli.x86_64 0:1.1.6-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
pacemaker-cluster-libs.x86_64 0:1.1.6-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
pacemaker-libs.x86_64 0:1.1.6-3.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
resource-agents.x86_64 0:3.9.2-7.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> rpcbind.x86_64
0:0.2.0-8.el6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Complete!</i></div>
<br />
<b>- Configure Corosync and Pacemaker</b><br />
Create configuration file /etc/corosync/corosync.conf. We only need to run this on dbmaster-01 as we will replicate the file over to dbmaster-02<br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# export ais_port=5405</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# export ais_mcast=226.94.1.1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# export ais_addr=`ip addr | grep "inet " | grep eth0 |
awk '{print $4}' | sed s/255/0/`</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# env | grep ais_</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>ais_mcast=226.94.1.1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>ais_port=5405</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>ais_addr=192.168.0.255
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# cp /etc/corosync/corosync.conf.example
/etc/corosync/corosync.conf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# sed -i.bak "s/.*mcastaddr:.*/mcastaddr:\ $ais_mcast/g"
/etc/corosync/corosync.conf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# sed -i.bak "s/.*mcastport:.*/mcastport:\ $ais_port/g"
/etc/corosync/corosync.conf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# sed -i.bak "s/.*bindnetaddr:.*/bindnetaddr:\ $ais_addr/g"
/etc/corosync/corosync.conf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# cat <<-END >>/etc/corosync/service.d/pcmk</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> service {</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> # Load
the Pacemaker Cluster Resource Manager</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> name:
pacemaker</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> ver: 1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> }</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>> END</i></div>
<br />
<b>- Review the configuration file /etc/corosync/corosync.conf</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# cd /etc/corosync</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
corosync]# cat corosync.conf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i># Please read the
corosync.conf.5 manual page</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>compatibility:
whitetank</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>totem {</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> version: 2</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> secauth: off</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> threads: 0</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> interface {</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
ringnumber: 0</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>bindnetaddr:
192.168.114.127</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>mcastaddr:
226.94.1.1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>mcastport: 5405</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> ttl:
1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> }</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>}</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>logging {</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> fileline:
off</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> to_stderr:
no</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> to_logfile:
yes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> to_syslog:
yes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> logfile:
/var/log/cluster/corosync.log</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> debug: off</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> timestamp:
on</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
logger_subsys {</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
subsys: AMF</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
debug: off</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> }</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>}</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>amf {</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> mode:
disabled</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>}</i></div>
<br />
<b>- Replicate the configuration to neighbor node (dbmaster-02) and start corosync service.</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
corosync]# for f in /etc/corosync/corosync.conf
/etc/corosync/service.d/pcmk /etc/hosts; do scp $f dbmaster-02:$f ;
done</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
corosync]# /etc/init.d/corosync start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting Corosync
Cluster Engine (corosync): [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
corosync]# grep -e "corosync.*network interface" -e
"Corosync Cluster Engine" -e "Successfully read main
configuration file" /var/log/messages</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [MAIN ] Corosync Cluster Engine
('1.4.1'): started and ready to provide service.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [MAIN ] Successfully read main
configuration file '/etc/corosync/corosync.conf'.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [TOTEM ] The network interface
[192.168.0.11] is now up.
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
corosync]# grep TOTEM /var/log/messages</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [TOTEM ] Initializing transport
(UDP/IP Multicast).</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [TOTEM ] Initializing transmit/receive
security: libtomcrypt SOBER128/SHA1HMAC (mode 0).</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [TOTEM ] The network interface
[192.168.0.11] is now up.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [TOTEM ] A processor joined or left
the membership and a new membership was formed.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ssh dbmaster-02 -- /etc/init.d/corosync start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting Corosync
Cluster Engine (corosync): [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<b>- Monitoring startup status of corosync</b><br />
Make sure pacemaker module is loaded successfully.<br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
corosync]# grep pcmk_startup /var/log/messages</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: CRM:
Initialized</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [pcmk ] Logging: Initialized
pcmk_startup</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: Maximum
core file size is: 18446744073709551615</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: Service:
10</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:08:39
dbmaster-01 corosync[27718]: [pcmk ] info: pcmk_startup: Local
hostname: dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Startup pacemaker on both nodes</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# chown -R hacluster:haclient /var/log/cluster</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# /etc/init.d/pacemaker start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting Pacemaker
Cluster Manager: [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# grep -e pacemakerd.*get_config_opt -e pacemakerd.*start_child -e
"Starting Pacemaker" /var/log/messages</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found
'pacemaker' for option: name</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found '1' for
option: ver</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found
'pacemaker' for option: name</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found '1' for
option: ver</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Defaulting to
'no' for option: use_logd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Defaulting to
'no' for option: use_mgmtd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'off'
for option: debug</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'yes'
for option: to_logfile</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found
'/var/log/cluster/corosync.log' for option: logfile</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Found 'yes'
for option: to_syslog</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31333]: info: get_config_opt: Defaulting to
'daemon' for option: syslog_facility</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: main: Starting Pacemaker
1.1.6-3.el6 (Build: a02c0f19a00c1eb2527ad38f146ebc0834814558):
generated-manpages agent-manpages ascii-docs publican-docs ncurses
trace-logging cman corosync-quorum corosync</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child
31341 for process stonith-ng</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child
31342 for process cib</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child
31343 for process lrmd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child
31344 for process attrd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child
31345 for process pengine</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dec 29 03:29:05
dbmaster-01 pacemakerd: [31337]: info: start_child: Forked child
31346 for process crmd
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ssh dbmaster-02 -- chown -R hacluster:haclient /var/log/cluster</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ssh dbmaster-02 -- /etc/init.d/pacemaker start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting Pacemaker
Cluster Manager: [ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Verify if heartbeat processes are started</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ps axf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> PID TTY STAT
TIME COMMAND</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i> 2 ? S
0:00 [kthreadd]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>... lots of
processes ....
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>27718 ? Ssl
0:00 corosync</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31337 pts/0 S
0:00 pacemakerd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31341 ? Ss
0:00 \_ /usr/lib64/heartbeat/stonithd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31342 ? Ss
0:00 \_ /usr/lib64/heartbeat/cib</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31343 ? Ss
0:00 \_ /usr/lib64/heartbeat/lrmd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31344 ? Ss
0:00 \_ /usr/lib64/heartbeat/attrd</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31345 ? Ss
0:00 \_ /usr/lib64/heartbeat/pengine</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>31346 ? Ss
0:00 \_ /usr/lib64/heartbeat/crmd
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@test-db1
corosync]# grep ERROR: /var/log/messages | grep -v unpack_resources</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@test-db1
corosync]#</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<b>- Verify the HA cluster is running now</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# crm_mon</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last updated: Thu
Dec 29 05:19:52 2011</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Last change: Thu Dec
29 05:07:59 2011 via crmd on dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Stack: openais</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Current DC:
dbmaster-01.localdomain - partition with quorum</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Version:
1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>2 Nodes configured,
2 expected votes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>0 Resources
configured.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>============</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Online: [
dbmaster-01.localdomain dbmaster-02.localdomain ]</i></div>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-6472676926439076592012-06-14T21:54:00.000-07:002012-08-11T03:23:09.961-07:00HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 2<b>- OS Pre-configuration task</b><br />
Below listed items have to be done on both master nodes.<br />
<br />
<b>- Disable SELinux and iptables</b><br />
<i>[root@dbmaster-01~]# getenforce <br />Disabled<br />[root@dbmaster-01~]# service iptables stop<br />[root@dbmaster-01~]# chkconfig iptables off</i><br />
<br />
<b>- Network configuration</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="margin-bottom: 0in;">
Each master node has to be configured
with 2 network interfaces. One is external nic for external traffic
(e.g. MySQL traffic, Internet traffic … etc) while another one is
internal heartbeat nic which is used to connect 2 master nodes.<br />
<br />
In our scenario the master nodes will
have IP address like this.</div>
<br />
dbmaster-01: IP 192.168.0.11/24, Gateway 192.168.0.1<br />
dbmaster-02: IP 192.168.0.12/24, Gateway 192.168.0.1<br />
<br />
<div style="margin-bottom: 0in;">
</div>
<div style="margin-bottom: 0in;">
The VIP is 192.168.0.10/24 which will
be floating between 2 nodes, depending on the server availability.</div>
<br />
<b>- Generate SSH-key and allow key-based authentication</b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01~]#
ssh-keygen -t rsa
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Generating
public/private rsa key pair.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Enter file in which
to save the key (/root/.ssh/id_rsa):
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Enter passphrase
(empty for no passphrase):
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Enter same
passphrase again:
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Your identification
has been saved in /root/.ssh/id_rsa.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Your public key has
been saved in /root/.ssh/id_rsa.pub.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>The key fingerprint
is:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>c7:50:ac:a4:09:fb:5d:f3:1e:13:ed:2e:21:d4:a7:f7
root@dbmaster-01</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>The key's randomart
image is:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>+--[ RSA 2048]----+</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| .. |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| . ... |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| o +.. . . |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| . o .o+ o o |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| . .Sooo = |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| . ... * o |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| o * . |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| o . E|</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>| . |</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>+-----------------+
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01~]#
ssh-copy-id -i .ssh/id_rsa.pub root@dbmaster-02 <span style="color: red;">#####
replace dbmaster-02 with dbmaster-01 if you are doing it from
dbmaster-02 to dbmaster-01</span></i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>The authenticity of
host <span style="color: red;">dbmaster-02 </span>(192.168.0.12)'
can't be established.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>RSA key fingerprint
is 51:73:cb:48:c3:8e:9c:39:88:38:b8:a9:70:b8:fd:76.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Are you sure you
want to continue connecting (yes/no)? yes</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Warning: Permanently
added 'dbmaster-02' (RSA) to the list of known hosts.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>root@dbmaster-02's
password:
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Now try logging into
the machine, with "ssh 'root@dbmaster-02'", and check in:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
.ssh/authorized_keys</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>to make sure we
haven't added extra keys that you weren't expecting.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@ dbmaster-01
~]# ssh dbmaster-02 date<span style="color: red;">##### replace
dbmaster-02 with</span> <span style="color: red;">dbmaster-01 if you are
doing it from dbmaster-02 to dbmaster-01</span></i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Tue Dec 27 22:08:44
EST 2011</i> </div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
- Configure ntpd to make sure time are in sync<i><br /></i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# service ntpd stop</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Shutting down ntpd:
[ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ntpdate ntp.asia.pool.ntp.org</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>27 Dec 22:11:27
ntpdate[1796]: adjust time server x.x.x.x offset 0.000983 sec</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# service ntpd start</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Starting ntpd:
[ OK ]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Add host entries to host file to make sure both nodes see each other via dns name</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# cat /etc/hosts</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i># Do not remove the
following line, or various programs</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i># that require
network functionality will fail.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>127.0.0.1
localhost.localdomain localhost</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>::1
localhost6.localdomain6 localhost6</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>192.168.0.11
dbmaster-01 dbmaster-01.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>192.168.0.12
dbmaster-02 dbmaster-02.localdomain</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ping dbmaster-01</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>PING dbmaster-01
(192.168.0.11) 56(84) bytes of data.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>64 bytes from
dbmaster-01 (192.168.0.11): icmp_seq=1 ttl=64 time=0.017 ms</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>^C</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>--- dbmaster-01 ping
statistics ---</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>1 packets
transmitted, 1 received, 0% packet loss, time 588ms</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>rtt min/avg/max/mdev
= 0.017/0.017/0.017/0.000 ms</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# ping dbmaster-02</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>PING dbmaster-02
(192.168.0.12) 56(84) bytes of data.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>64 bytes from
dbmaster-02 (192.168.0.12): icmp_seq=1 ttl=64 time=0.017 ms</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>^C</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>--- dbmaster-01 ping
statistics ---</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>1 packets
transmitted, 1 received, 0% packet loss, time 588ms</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>rtt min/avg/max/mdev
= 0.017/0.017/0.017/0.000 ms</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Partition the shared-disk</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="margin-bottom: 0in;">
The shared disk will be used to store MySQL db and have to be partition
from master node (either one). If you are using vmware ESXi, you could simply mount the disk on both VM and the disk will probably be presented as second disk (e.g. /dev/sdb). Please be notice that no parallel run is
allowed which means only one node could access and write the shared-disk at the same time or otherwise it will crash the disk. In our example, we will assume /dev/sdb as our shared-disk.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# mkfs.ext4 /dev/sdb <span style="color: red;">#### /dev/sdb is the
shared disk in this example</span></i></div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Installing MySQL Packages.</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
This should be pretty striaght forward, given that your server could access the Internet with any issue.<i> </i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# yum -y install mysql-server mysql
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Setting up Install
Process</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Resolving
Dependencies</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>--> Running
transaction check</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>---> Package
mysql.x86_64 0:5.1.52-1.el6_0.1 will be installed</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>****************</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>**** skipped ****</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>****************</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Dependency
Installed:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>
perl-DBD-MySQL.x86_64 0:4.013-3.el6
perl-DBI.x86_64 0:1.609-4.el6
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Complete!</i></div>
<br />
<b>- Configure MySQL </b><br />
Edit /etc/my.cnf as below.<i><br /></i><br />
<style type="text/css"><!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# cat /etc/my.cnf</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[mysqld]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>datadir=/mysql</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>socket=/mysql/mysql.sock</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>user=mysql</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i># Disabling
symbolic-links is recommended to prevent assorted security risks</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>symbolic-links=0</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>innodb_rollback_on_timeout=1</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>innodb_lock_wait_timeout=600</i></div>
<i>log-bin=mysql-bin ## Just in case you want replication</i>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>binlog-format='ROW' ## Just in case you want replication</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[mysqld_safe]</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>log-error=/mysql/mysqld.log</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>pid-file=/var/run/mysqld/mysqld.pid</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<b>- Create the mount point and mount the shared disk</b></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# mkdir /mysql</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# mount /dev/sdb /mysql</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@dbmaster-01
~]# chown –R mysql:mysql /mysql</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i></div>
<b>- Run mysql_install_db script to install the base-db, make sure this will only be run once only. </b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@ dbmaster-01
~]# mysql_install_db
</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Installing MySQL
system tables...</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>OK</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Filling help
tables...</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>OK</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>To start mysqld at
boot time you have to copy</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>support-files/mysql.server
to the right place for your system</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>PLEASE REMEMBER TO
SET A PASSWORD FOR THE MySQL root USER !</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>To do so, start the
server, then issue the following commands:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>/usr/bin/mysqladmin
-u root password 'new-password'</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>/usr/bin/mysqladmin
-u root -h test-db1.dpcloud.local password 'new-password'</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Alternatively you
can run:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>/usr/bin/mysql_secure_installation</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>which will also give
you the option of removing the test</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>databases and
anonymous user created by default. This is</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>strongly recommended
for production servers.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>See the manual for
more instructions.</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>You can start the
MySQL daemon with:</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>cd /usr ;
/usr/bin/mysqld_safe &</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>You can test the
MySQL daemon with mysql-test-run.pl</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>cd /usr/mysql-test ;
perl mysql-test-run.pl</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i><br /></i>
</div>
<div style="line-height: 100%; margin-bottom: 0in;">
<i>Please report any
problems with the /usr/bin/mysqlbug script!</i></div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<div style="line-height: 100%; margin-bottom: 0in;">
[root@dbmaster-01 ~]#
chown –R mysql:mysql /mysql</div>
<br />
<b>- Remember to unmount the shared-disk /mysql after that. </b><br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="line-height: 100%; margin-bottom: 0in;">
<i>[root@ dbmaster-01
~]# umount /mysql</i></div>
<br />
<br />Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-19564049128550149502012-06-12T21:23:00.000-07:002012-08-11T03:19:48.968-07:00HA Active-Standby MySQL + Heartbeat 3.x + Coroysnc 1.x + Pacemaker 1.x on RHEL / CentOS - Section 1<b>- The HA Cluster design.</b><br />
<br />
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
<br />
<div style="margin-bottom: 0in;">
This HA MySQL clustering configuration will
be based on 2 servers which are in an active-backup relationship. Below diagram explain the logical
design of the setup.</div>
<div style="margin-bottom: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3EZLy90wFqNqkqabF1u9DYkNWzRT4OWr-Ao1hP5ID2xQnxU_mz5HghapXaqyHHPAOzsDdy6XsPXK2ksODOZLFQydvwn4cFKiC6fLZHu0dJjDMv9AEJvtHfqsCgxj2XjGsO8Q_FM_TpG8/s1600/ha-mysql-db.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3EZLy90wFqNqkqabF1u9DYkNWzRT4OWr-Ao1hP5ID2xQnxU_mz5HghapXaqyHHPAOzsDdy6XsPXK2ksODOZLFQydvwn4cFKiC6fLZHu0dJjDMv9AEJvtHfqsCgxj2XjGsO8Q_FM_TpG8/s320/ha-mysql-db.png" width="320" /></a></div>
<div style="margin-bottom: 0in;">
Both HA-master nodes are installed with heartbeat packages, an internal nic presented to each other with private ip. Heartbeat package of both nodes will check the status of remote node and take the “active node” role when remote node is down. Active node will be responsible to take over the VIP (Virtual IP, the IP floating between 2 HA master), MySQL DB store and spawn up MySQL process to serve mysql request. If replication slave is needed, a 3rd node could be added to the cluster as replication slave and this 3rd node could hooking to the VIP to fetch binary logs but the detail procedures for replication slave is out of the scope of this howto hence it should be pretty straight forward for those that familiar with mysql master-slave replication setup. </div>
<div style="margin-bottom: 0in;">
<br /></div>
<div style="margin-bottom: 0in;">
<b>- Hardware and software requirement of both nodes</b></div>
<div style="margin-bottom: 0in;">
So here are the ingredients for the HA setup.</div>
<div style="margin-bottom: 0in;">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
-->
</style>
</div>
<ul>
<li>2 server with
identical hardware configuration</li>
<li>Minimum
requirement (at least 1G memory, 20G root disk)</li>
<li>Each node will
need a pair of nic, public and internal heartbeat nic</li>
<li>Public nic will
be configured as public facing (or at least the nic is connected to internet to allow packages fetching)</li>
<li>Internal nic of
both nodes will be connected to each other via either crossover
cable or on same VLAN.</li>
<li>One shared LUN
(Logical Disk Unit) to be seen and mountable by both nodes. You may probably configure this by using iscsi (e.g. openfiler), vmware shared-disk , phyiscal fiber-channel storage or DRBD disk. In this example we used a pre-configured Vmware ESX shared-disk.</li>
<li>One shared
Virtual IP for IP failover in between 2 nodes. This IP will be on the same segment of public nic's IP.</li>
<li>The OS will be RHEL/CentOS 6 64bit with minimal installation. Additional software repositories have to be added so that we could fetch and install Heartbeat 3.x, Corosync 1.x and Pacemaker 1.1.x. </li>
</ul>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-25495663430759587422012-06-12T21:12:00.000-07:002014-06-12T21:15:58.381-07:00Good reference to improve your slide.<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWEeVAWzjdEbnPqu1JdhXl27p-RhwvOnNQqlgv3SW2AZ6ZF2P32y_X8joc9zYKvfe9DgCf2zj4nteUHqOfoHewmsrG5dazYA-LbwgcnMGZ-khCWyMUCqiVOGD9UAmL3K-CULT7gfN4_2k/s1600/slideology.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWEeVAWzjdEbnPqu1JdhXl27p-RhwvOnNQqlgv3SW2AZ6ZF2P32y_X8joc9zYKvfe9DgCf2zj4nteUHqOfoHewmsrG5dazYA-LbwgcnMGZ-khCWyMUCqiVOGD9UAmL3K-CULT7gfN4_2k/s1600/slideology.gif" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
Note: It was quite sometime since I got this book through the O'Reilly Blogger Review Program and I am just about to write the review just recently.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
So basically I am a technical geek with no concept of how a presentation should be. After reading this book this does show me how one create an interesting presentation. It is so well written and with lots of example demonstrate the idea. it is definitely a good reference for those that look forward to improve their presentation skill. </div>
Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-33912640541875037702012-05-24T21:15:00.000-07:002012-07-10T21:16:24.639-07:00Allowing ddclient on vyatta to push NAT outside IP to DDNS providerI have a vyatta vpn appliance sit behind NAT and have a need to use dynamic DNS to update its public IP to dynamic dns provide like no-ip.com. So I ran the suggested commands mentioned in their doc <br />
<br />
<i>vyatta@vyatta# set service dns dynamic interface eth0 service dyndns host-name myvyattatestbox.no-ip.org <br />[edit]<br />vyatta@vyatta# set service dns dynamic interface eth0 service dyndns server dynupdate.no-ip.com<br />[edit]<br />vyatta@vyatta# set service dns dynamic interface eth0 service dyndns login myusername<br />[edit]<br />vyatta@vyatta# set service dns dynamic interface eth0 service dyndns password mypassword<br />[edit]</i><br />
<i>vyatta@vyatta# commit</i><br />
<i>[edit]</i><br />
<br />
However, somehow it updated its internal IP of the nic instead of the NAT outside public IP address.<br />
<br />
<i>$ show dns dynamic status<br />interface : eth0<br />ip address : 192.168.0.80<br />host-name : myvyattatestbox.no-ip.org<br />last update : Wed May 11 04:07:20 2012<br />update-status: good</i><br />
<br />
It looks like the way that vyatta will update the IP binded to interface though I would expect it to update with the NAT outside address. To let vyatta to update with the NAT outside, we could make a trick on /opt/vyatta/sbin/vyatta-dynamic-dns.pl, replace the line from<br />
<br />
<i>97 #$output .= "use=if, if=$interface\n\n\n";</i><br />
<br />
to<br /> 98 <i>$output .= "use=web, web=checkip.dyndns.com/, web-skip='IP Address: '\n";</i><br />
<br />
By replacing the line, vyatta will query the NAT outside IP against checkip.dyndns.com and then use the polled IP to update against the dynamic DNS provider. <br />
<br />
<i>$ show dns dynamic status<br />
interface : eth0<br />
ip address : 1.2.3.4<br />
host-name : myvyattatestbox.no-ip.org<br />
last update : Wed May 11 05:07:20 2012<br />
update-status: good</i><br />
<br />
<br />Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-72835348926837946872012-05-15T01:00:00.000-07:002012-06-12T01:17:32.207-07:00Using kpartx to mount partition(s) from disk image.Scenario:<br />
<br />
You got a disk image which was just dumped via dd (e.g. <i>dd if=/dev/hda of=/disk.img</i>). There are 3 partitions on your source disk /dev/hda and they looks like this,<br />
<br />
<br />
<i>[root@localhost /]# fdisk -l</i><br />
<i><br /></i><br />
<i>Disk /dev/hda: 75.1GB, 75161927680 bytes</i><br />
<i>255 heads, 63 sectors/track, 9137 cylinders</i><br />
<i>Units = cylinders of 16065 * 512 = 8225280 bytes</i><br />
<i><br /></i><br />
<i> Device Boot Start End Blocks Id System</i><br />
<i>/dev/hda1 * 1 25 200781 83 Linux</i><br />
<i>/dev/hda2 26 1330 10482412+ 82 Linux swap / Solaris</i><br />
<i>/dev/hda3 1331 9137 62709727+ 83 Linux</i><br />
<br />
Now, as long as your disk image /disk.img is a complete dump of /dev/hda, you should be able to mount the partitions within the disk image via loopback device with proper offset value. For e.g.<br />
<br />
#### Create the mount point<br />
<i>[root@localhost /]# mkdir -p /chroot/boot</i><br />
<br />
#### Mount the first partition with offset (Start * sectors/track * 512, i.e. 1*63*512)<br />
<i>[root@localhost /]# mount -o loop,offset=$((1*63*512)) /disk.img /chroot/boot</i><br />
<i>[root@localhost /]# mount | grep /chroot/boot</i><br />
<i>/disk.img on /chroot/boot type ext3 (rw,loop=/dev/loop1,offset=32256)</i><br />
<br />
So you successfully mounted the first partition (/dev/hda1) and then planning to mount the 3rd partition (/dev/hda3) to /chroot<br />
<br />
#### Try mounting the 3rd partition with offset 1331<br />
<i>[root@localhost /]# mount -o loop,offset=$((1331*63*512)) /disk.img /chroot</i><br />
<i>hfs: unable to find HFS+ superblock</i><br />
<i>mount: you must specify the filesystem type</i><br />
<br />
Apparently there is something wrong with mount and for some reason it didn't handle offset very well. The util-linux on my box is ver 2.13 which claims to support offset higher than 32bit (well i didn't do any deeper for this specific matters though) but unfortunately it didn't help. As what I want is a quick fix, I come across a tools called "kpartx"which is actually a swiss knife to mount partitions within a disk image file. So here is a demo of how it works.<br />
<br />
Solutions: <br />
<br />
To list partitions on a disk image. So in this example kpartx see 3 partitions from image disk.img<br />
<i>[root@localhost /]# kpartx -l /disk.img</i><br />
<i>loop0p1 : 0 401562 /dev/loop 63</i><br />
<i>loop0p2 : 0 20964825 /dev/loop 401625</i><br />
<i>loop0p3 : 0 21366450 /dev/loop 21366450</i><br />
<br />
To activate these partitons, we can run kpartx with option -av <br />
<i>[root@localhost /]# kpartx -av /disk.img</i><br />
<i>add map loop0p1 : 0 401562 linear /dev/loop 63</i><br />
<i>add map loop0p2 : 0 20964825 linear /dev/loop 401625</i><br />
<i>add map loop0p3 : 0 21366450 linear /dev/loop 21366450</i><br />
<br />
We can see the device is now mounted as loopback device and presented via /dev/mapper<br />
<i>[root@localhost /]# losetup -a</i><br />
<i>/dev/loop0: [0800]:49154 (/disk.img)</i><br />
<i>[root@localhost /]# ls /dev/mapper/loop0p*</i><br />
<i>/dev/mapper/loop0p1 /dev/mapper/loop0p2 /dev/mapper/loop0p3</i><br />
<br />
<br />
Lets see if we could mount these loopback partitions.<br />
<br />
<i>[root@localhost /]# mount /dev/mapper/loop0p1 /chroot/boot</i><br />
<i>[root@localhost /]# mount /dev/mapper/loop0p3 /chroot</i> <br />
<i>[root@localhost /]# mount | grep chroot</i><br />
<i>/dev/mapper/loop0p1 on /chroot/boot type ext3 (rw)</i><br />
<i>/dev/mapper/loop0p3 on /chroot type ext3 (rw)</i><br />
<br />
So it looks like the partitions are mounted and file systems on it are recognized without any issue. Let say we finished with the operations on these partitions and now we plan to unmount it and clean it up.<br />
<br />
<i>[root@localhost /]# umount /chroot/boot /chroot</i><br />
<i>[root@localhost /]# losetup -d /dev/mapper/loop0p1</i><br />
<i>[root@localhost /]# losetup -d /dev/mapper/loop0p2 </i><br />
<i>[root@localhost /]# losetup -d /dev/mapper/loop0p3</i><br />
<i>[root@localhost /]# kpartx -d /test.img</i><br />
<br />
Pretty much it. <i><br /></i>Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0tag:blogger.com,1999:blog-8392021181198238917.post-60085991443722298242012-05-13T20:00:00.000-07:002012-08-05T20:32:29.317-07:00Nice webcast, Diagnosing Intermittent MySQL Problems<a href="http://event.on24.com/r.htm?e=440046&s=1&k=4E1095FD06B3CA383BC5CD02DC93714C" target="_blank">http://event.on24.com/r.htm?e=440046&s=1&k=4E1095FD06B3CA383BC5CD02DC93714C</a><br />
<br />
<br />Philip's Tech Bloghttp://www.blogger.com/profile/01560753940856560992noreply@blogger.com0