Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N

So,  because of my job duty and I have to deal with Citrix Cloudstack day by day. Recently we are deploying a new advanced zone and for some reason we are seeing errors like this during deploy of our first VM instance.


2013-07-22 22:09:27,625 WARN  [api.commands.DeployVMCmd] (Job-Executor-50:job-534828) Exception:
com.cloud.exception.AgentUnavailableException: Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N
................
Caused by: com.cloud.utils.exception.CloudRuntimeException: Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N
................
2013-07-22 22:09:27,626 WARN  [cloud.api.ApiDispatcher] (Job-Executor-50:job-534828) class com.cloud.api.ServerApiException : Resource [Host:N] is unreachable: Host N: Unable to start instance due to Template systemvm-kvm-3.0.0 has not been completely downloaded to zone N

So, basically, what Cloudstack doing is to
1. check if there is any valid systemvm template (in this case systemvm-kvm-3.0.0) deployed to the zone.
2. If things works as it should, you should be able to find the installed/downloaded template from table cloud.vm_template, cloud.template_zone_ref and template_host_ref. Hence, if you scan through the template list from the Web GUI, you should be able to see the template be downloaded.

In my case, the template was not downloaded as it should (or marked as downloaded at DB layer), and if you look at the table cloud.template_host_ref, there is some abnormality here.

mysql> select * from  template_host_ref where id=11111\G
*************************** 1. row ***************************
            id: 11111
       host_id: *masked*
   template_id: *masked*
       created: 2013-07-18 17:50:43
  last_updated: 2013-07-22 20:04:52
        job_id: 75a75e55-5280-4ba5-b823-cadbcbe2cc7a
  download_pct: 0
          size: 0
 physical_size: 0
download_state: DOWNLOAD_ERROR
     error_str: No route to host
    local_path: /mnt/SecStorage/04ab8f0b-c4e0-34a4-80b3-457c433acde3/template/tmpl/2/1686/dnld6951269530983090325tmp_
  install_path: NULL
           url: http://download.cloud.com/templates/acton/acton-systemvm-02062012.qcow2.bz2
     destroyed: 0
       is_copy: 0

So, basically the things are 1) download_pct is 0 (while it should be 100 if download succeed), 2) download_state is DOWNLOAD_ERROR (while it should be DOWNLOADED if download successed and 3) error_str is "No route to host".

In my case, the template installation procedures was not completed (though I have completed the cloud-install-sys-tmplt script per official installation guide), at least at DB layer.

So I double checked the secondary storage to make sure the template file is completely downloaded (IMPORTANT!!!, if the file is not there, go through installation guide and re-run cloud-install-sys-tmplt script) and hacked the DB by updating the cloud.template_host_ref table. (Replace "N" with the correct account id and template id respectively)

mysql> updated template_host_ref set download_pct=100, download_state='DOWNLOADED', error_str=NULL, localpath='template/tmpl/N/N' where id=11111\G
*************************** 1. row ***************************
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Now cloudstack could launch VM as it should.

Using kpartx to mount partition(s) from disk image.

Scenario:

You got a disk image which was just dumped via dd (e.g. dd if=/dev/hda of=/disk.img). There are 3 partitions on your source disk /dev/hda and they looks like this,


[root@localhost /]# fdisk -l


Disk /dev/hda: 75.1GB, 75161927680 bytes
255 heads, 63 sectors/track, 9137 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes


   Device  Boot      Start       End       Blocks   Id   System
/dev/hda1 *                  1         25       200781  83   Linux
/dev/hda2                   26     1330  10482412+ 82   Linux swap / Solaris
/dev/hda3               1331     9137  62709727+ 83   Linux
 
Now, as long as your disk image /disk.img is a complete dump of /dev/hda, you should be able to mount the partitions within the disk image via loopback device with proper offset value. For e.g.

#### Create the mount point
[root@localhost /]# mkdir -p /chroot/boot

#### Mount the first partition with offset (Start * sectors/track * 512, i.e. 1*63*512)
[root@localhost /]# mount -o loop,offset=$((1*63*512)) /disk.img /chroot/boot
[root@localhost /]# mount | grep /chroot/boot
/disk.img on /chroot/boot type ext3 (rw,loop=/dev/loop1,offset=32256)

So you successfully mounted the first partition (/dev/hda1) and then planning to mount the 3rd partition (/dev/hda3) to /chroot

#### Try mounting the 3rd partition with offset 1331
[root@localhost /]# mount -o loop,offset=$((1331*63*512)) /disk.img /chroot
hfs: unable to find HFS+ superblock
mount: you must specify the filesystem type

Apparently there is something wrong with mount and for some reason it didn't handle offset very well. The util-linux on my box is ver 2.13 which claims to support offset higher than 32bit (well i didn't do any deeper for this specific matters though) but unfortunately it didn't help. As what I want is a quick fix, I come across a tools called "kpartx"which is actually a swiss knife to mount partitions within a disk image file. So here is a demo of how it works.

Solutions:

To list partitions on a disk image. So in this example kpartx see 3 partitions from image disk.img
[root@localhost /]# kpartx -l /disk.img
loop0p1 : 0 401562 /dev/loop 63
loop0p2 : 0 20964825 /dev/loop 401625
loop0p3 : 0 21366450 /dev/loop 21366450

To activate these partitons, we can run kpartx with option -av
[root@localhost /]# kpartx -av /disk.img
add map loop0p1 : 0 401562 linear /dev/loop 63
add map loop0p2 : 0 20964825 linear /dev/loop 401625
add map loop0p3 : 0 21366450 linear /dev/loop 21366450

We can see the device is now mounted as loopback device and presented via /dev/mapper
[root@localhost /]# losetup -a
/dev/loop0: [0800]:49154 (/disk.img)
[root@localhost /]# ls /dev/mapper/loop0p*
/dev/mapper/loop0p1  /dev/mapper/loop0p2  /dev/mapper/loop0p3


Lets see if we could mount these loopback partitions.

[root@localhost /]# mount /dev/mapper/loop0p1 /chroot/boot
[root@localhost /]# mount /dev/mapper/loop0p3 /chroot
[root@localhost /]# mount | grep chroot
/dev/mapper/loop0p1 on /chroot/boot type ext3 (rw)
/dev/mapper/loop0p3 on /chroot type ext3 (rw)

So it looks like the partitions are mounted and file systems on it are recognized without any issue. Let say we finished with the operations on these partitions and now we plan to unmount it and clean it up.

[root@localhost /]# umount /chroot/boot /chroot
[root@localhost /]# losetup -d /dev/mapper/loop0p1
[root@localhost /]# losetup -d /dev/mapper/loop0p2
[root@localhost /]# losetup -d /dev/mapper/loop0p3
[root@localhost /]# kpartx -d /test.img

Pretty much it.

Reinstalling GRUB after upgrade from Ubuntu 9 to Ubuntu 10.04

So I just upgraded my Ubuntu 9.10 desktop (well, EOL for longtime) to a more recent release, Ubuntu 10.04 LTS Lucid Lynx. So far the upgrade was pretty smooth except it took me like 3 hours to download and complete all the installation files.

Unfortunately it shows something like this after the reboot.

GRUB loading.
error: the symbol 'grub_puts' not found

grub rescue>

So I thought there is something going on with the GRUB during the upgrade, I grep a Ubuntu 10.04 Desktop iso (make sure it is Desktop iso instead of server iso, so that we could boot it up to Live CD mode. ) and boot it up to perform system rescue.

Once the Live CD is booted, I mounted / to somewhere under /mnt

/dev/sda1 on /mnt type ext4 (rw)

I tried chroot to /mnt and run grub-install from there, no dice.

root@ubuntu:/# grub-install  --force /dev/sda
/usr/sbin/grub-probe: error: cannot find a device for /boot/grub (is /dev mounted?).
No path or device is specified.
Try `/usr/sbin/grub-probe --help' for more information.
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.

So it is apparently the grub-installation on the disk is corrupted or something, I quited from chroot mode and simply run grub-install from the Live CD.

root@ubuntu:~# grub-install --root-directory=/mnt/ /dev/sda
Installation finished. No error reported.

After a reboot my machine is booting without any issue.

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ABC at /etc/puppet/manifests/nodes.pp:14 on node XYZ

So this error is pretty confusing to understand at first glance,

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ABC at /etc/puppet/manifests/nodes.pp:14 on node XYZ

Basically it is kind of complaining on missing of specific modules ABC. There are 2 things could be checked here,

1. Make sure you have define the modulepath parameter per the document
http://docs.puppetlabs.com/puppet/2.7/reference/modules_fundamentals.html

2. Make sure you have properly define the class name within ABC/manifests/init.pp (well it happens to me couples of time and it turns out there was a typo within the init.pp)

So below init.pp is deemed to see the declared class not found error.


# cat /etc/puppet/modules/ABC/manifests/init.pp
class ABCtypo {

  exec { "blah ....":
       }

}

How to add a puppet client to puppet master.

Before a new puppet client be allowed to fetch manifest from puppet server, the client will have to be signed and below command would do the job

[root@puppetclient ~]# puppet agent --server puppetmaster --test --waitforcert 30



The above command will execute puppet as agent mode and connect to server puppetmaster (** remote server name in here have to match the remote server hostname or otherwise client agent will come up with error "err: Could not retrieve catalog from remote server: hostname was not match with the server certificate"). Option "--test" means the agent will be executed in test mode and then --waitforcert 30 means the puppet client will wait for 30 seconds for server to sign up the certificate. If 30 seconds passed and the client certificate is still not signed, the client agent will stop and exit.

So on server, below command would list out the certs pending for approval

root@puppetmaster:~# puppetca --list
  puppetclient
(CC:2B:2B:9D:4A:EF:3F:15:EF:60:C7:73:C9:18:FF:D1)


root@puppetmaster:~# puppetca --sign puppetclient
notice: Signed certificate request for puppetclient
notice: Removing file Puppet::SSL::CertificateRequest puppetclient at '/var/lib/puppet/ssl/ca/requests/puppetclient'

AWS Storage Gateway, can it be seated behind NAT?

Recently I was testing the AWS Storage Gateway. AWS storage gateway is a product combing AWS console frontend and a ESXi-based Linux VM (we call it storage gateway VM). The AWS console is responsible to handle user instruction on the storage gateway and pass the instruction to storage VM (e.g. create iscsi target on Storage VM, take or restore snapshot ... etc) while the ESXi-based storage VM is the actual host handling the instruction and storage thing.

From observation, there will be 2 ports be listening on the storage gateway VM, they are TCP port 80 and 3260. Port 80 is actually a java instance which is responsible to serve API call (user submit the request via AWS console or AWS API, and then AWS pass the request to the API handler on port 80 of Storage VM). Port 3260 is the ISCSI target which is responsible to handle all ISCSI request..

So, i was asked if it is possible to run this storage gateway VM behind NAT, i.e. sitting the VM in private network. With this subjective, there are 2 possible scenarios,

 - one is sitting the VM behind NAT but without any port mapping on port 80 and 3260
 - while another scenario is putting the VM behind NAT but with port mapping enabled (i.e. exposing and mapping the port 80 and 3260 on wan outside to port 80 and 3260 on the VM with private IP address).


Unfortunately both scenario wouldn't work. For first scenario, though the storage VM could be activated without problem, AWS just not able to communicate with the storage gateway VM therefore all instruction from users wouldn't be passed over to storage gateway VM at all. No ISCSI target could be created, no snapshot could be created or restored as all instructions are pended and timed out. For second scenario, we could activate the storage gateway VM, create volume, create and restore snapshot but the ISCSI target is just not working at all due to the ISCSI implementation restriction. ISCSI initiator (or ISCSI guest) could discover the ISCSI target via port 3260 but it just couldn't login to the resources.
 
To explain why it wont work behind NAT, we will have to go through the process in connecting or mapping an ISCSI target.

The ISCSI connection establishment process is actually a two-step process. The first step would be the iscsi initiator to scan and discover iscsi remote resource on the remote iscsi target. During the test, we could successfully perform this step as we do see the iscsi target during iscsi discovery. However, on the 2nd step when we tried to login into the ISCSI resource and then we see issue. The situation is that, iscsi resource is presented with a combination of on-host ip and iqn, e.g "10.1.1.1, iqn-name". The ip address here is the ip on the host, therefore that is NATTed private address. Once iscsi initiator(the guest VM) try to map the remote resource, due to implementation restriction it will connect to the ip address being presented, therefore the private ip address. As the IP address presented is private, VM initiator from public network wouldn't be able to talk to that IP address and connecting to the target would lead to request timeout.

The only possible workaround we could think of right now is to create a VPN tunnel between Initiator and the storage gateway VM behind NAT. In this way the AWS storage VM's iSCSI targets can be seen as they would be on the same LAN segment.  However with this approach it would definitely add extra overhead on the ISCSI's I/O performance.

BTW, this storage appliance is designed for access from on-premise device (i.e., natively they should be on the same network segemtn) and which means the ISCSI traffic should not really need to get through public network. In situation like this the appliance should be good to use.

err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

So I am seeing below captioned error when I am trying to connect to a puppet master.


err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

The reason for above error is because the agent node is trying to connect to a different master and then it failed to validate the certificate. To solve the problem, we have to execute below command and retry.

find /var/lib/puppet -type f | xargs rm -rf

Generate puppet server certificate

So I am getting error "err: Could not call sign: Could not find certificate request for puppetmaster" when I try to startup puppet server. I have to generate a SSL cert for the puppet server before going on.

root@puppetmaster:/etc/puppet# puppet cert generate puppetmaster
notice: puppetmaster has a waiting certificate request
notice: Signed certificate request for puppetmaster
notice: Removing file Puppet::SSL::CertificateRequest puppetmaster at '/var/lib/puppet/ssl/ca/requests/puppetmaster.pem'
notice: Removing file Puppet::SSL::CertificateRequest puppetmaster at '/var/lib/puppet/ssl/certificate_requests/puppetmaster.pem'

Ubuntu, E: Unable to locate package

So I am trying to install packages on a newly installed Ubuntu box from aptitude but somehow it failed to locate the package.

# apt-get install gcc
Reading package lists... Done
Building dependency tree      
Reading state information... Done
E: Unable to locate package gcc
# aptitude search gcc
#

I am pretty sure the box could connect to the internet so it is quite weird it failed to locate the package.

In fact the issue is pretty straight forward, the local aptitude database didn't contain the software entries and this require an update of the database.

# apt-get update
Ign http://us.archive.ubuntu.com precise InRelease                                                        
Ign http://us.archive.ubuntu.com precise-updates InRelease                                                
Ign http://us.archive.ubuntu.com precise-backports InRelease
Ign http://security.ubuntu.com precise-security InRelease
Get:1 http://us.archive.ubuntu.com precise Release.gpg [198 B]
Get:2 http://us.archive.ubuntu.com precise-updates Release.gpg [198 B]
Get:3 http://security.ubuntu.com precise-security Release.gpg [198 B]
...
...
Get:87 http://us.archive.ubuntu.com precise-backports/restricted Translation-en [14 B]                                                        
Get:88 http://us.archive.ubuntu.com precise-backports/universe Translation-en [8,555 B]                                                       
Fetched 24.8 MB in 32s (769 kB/s)                                                                                                             
Reading package lists... Done

Now the issue is resolved. :-)

# apt-get install gcc
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  binutils cpp cpp-4.6 gcc-4.6 libc-dev-bin libc6-dev libgomp1 libmpc2 libmpfr4 libquadmath0 linux-libc-dev manpages-dev
Suggested packages:
... 
...

Openstack Keystone (diablo): Got: ImportError('No module named MySQLdb',)

If one is seeing this on a Ubuntu / Debian box during start of keystone after migrating DB from sqlite to MySQL, simply installing the associated python libraries would fix the issue.

root@keystone:~/openstack-keystone-79a9fde# ERROR: Unable to load keystone-legacy-auth from configuration file /etc/keystone/keystone.conf.
Got: ImportError('No module named MySQLdb',)


root@keystone:~/openstack-keystone-79a9fde# apt-get install python-mysqldb
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  python-support
Suggested packages:
  python-egenix-mxdatetime python-mysqldb-dbg
The following NEW packages will be installed:
  python-mysqldb python-support
0 upgraded, 2 newly installed, 0 to remove and 58 not upgraded.
Need to get 109 kB of archives.
After this operation, 578 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://us.archive.ubuntu.com/ubuntu/ oneiric/main python-support all 1.0.13ubuntu1 [26.6 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ oneiric/main python-mysqldb amd64 1.2.3-0ubuntu1 [82.5 kB]
Fetched 109 kB in 0s (184 kB/s)     
Selecting previously deselected package python-support.
(Reading database ... 55750 files and directories currently installed.)
Unpacking python-support (from .../python-support_1.0.13ubuntu1_all.deb) ...
Selecting previously deselected package python-mysqldb.
Unpacking python-mysqldb (from .../python-mysqldb_1.2.3-0ubuntu1_amd64.deb) ...
Processing triggers for man-db ...
Setting up python-support (1.0.13ubuntu1) ...
Setting up python-mysqldb (1.2.3-0ubuntu1) ...
Processing triggers for python-support ...

root@keystone:~/openstack-keystone-79a9fde# keystone &
[1] 13882
root@keystone:~/openstack-keystone-79a9fde# Starting the RAX-KEY extension
Starting the Legacy Authentication component
Service API listening on 0.0.0.0:5000
Admin API listening on 0.0.0.0:35357

AWS storage gateway: WORKING STORAGE NOT CONFIGURED

As continuing the test on AWS Storage gateway, I found that there is an implicit requirement of the AWS storage VM, i.e. the VM have to be assigned with a publicly accessible IP address, or at least the IP address could be reached by AWS network.

The logic behind is that when someone trying to manage the AWS storage VM via AWS web console, the instruction will have to be passed over to the VM (possibly via port 80 of the AWS VM, but I didnt confirm it yet) via public network. In any case AWS failed to reach the VM, it will not able to proceed with the instruction.

The above idea was tested against an internal VM I was playing with yesterday. The VM is sit on private network (e..g 192.168.x.x) with outgoing NAT enable but not incoming NAT enable. I could successfully proceed with the VM activation but no volumes could be added from AWS console. The newly added volumes keep showing "WORKING STORAGE NOT CONFIGURED" on AWS console which basically means that it is not creating at all. Usually, creating a new volume should not take too long at all.

Here is the screenshot though,

Apart from volumes creation failure, I also tried adding new virtual disk to the storage VM and see if AWS could see the new virtual disk. However, the answer is no. So what I could pretty sure here is that AWS will have to talk to VM and it just wont be able to put the Storage VM on an internal network segment which is not accessible from public.

Openstack Swift with swauth, getting "Account creation failed: 500 Server Error" when adding account

During testing of Swift with swauth, I was trying to add account to swauth database with swauth-add-account but it was failed out with "Account creation failed: 500 Server Error"

root@proxy:~# swauth-add-account -A https://1.2.3.4:8080/auth -K swauthkey testgp
Account creation failed: 500 Server Error


With further checking, it looks like "allow_account_management = true" have to be added under [app:proxy-server] tag of proxy-server.conf like this.






[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
account_autocreate = true
Once above line is added to configuration file, followed by proxy restart and that should fix the problem.

root@proxy:~s3-curl# swauth-add-account -A https://1.2.3.4:8080/auth -K swauthkey testgp
root@proxy:~s3-curl# swauth-list -A https://1.2.3.4:8080/auth -K swauthkey
{"accounts": [{"name": "system"}, {"name": "testgp"}]}

AWS storage gateway port 80 connection refused

Was testing on AWS storage gateway services in my ESXi host.  So what is AWS storage gateway? Basically it is a AWS service to host behind your on-premise firewall. An AWS customized VM living on ESXi will be put on-premise and allow remote storage management from AWS console. The VM on ESXi will present storage via iscsi which allow remote read-write access..

I was following this documents but unfortunately I jump into the condition while I was trying to activate my storage gateway VM. It looks like activation require access on port 80 but unfortunately I see nothing on port 80. Running nmap against the VM but I see nothing coming up.


root@localhost:~$ nmap -sT 1.2.3.4


Starting Nmap 5.00 ( http://nmap.org ) at 2012-03-29 16:13 HKT
Interesting ports on 1.2.3.4:
Not shown: 996 filtered ports
PORT     STATE  SERVICE
22/tcp   closed ssh
80/tcp   closed http
631/tcp  closed ipp
3260/tcp closed iscsi

 I tried to get into the storage VM as root (prior to the that, I booted the VM to single user mode and reset the password, for details see here) and found that no services are running on those ports.

With further checking, it seems that it have to do with the time service on the VM. The storage VM will need an very accurate time or otherwise it will not be coming up. As per the suggestion from setup guide, I enabled ntp service on ESXi and then enable the "Synchronize guest time with host" option on VM followed by VM restart. I hope it would work but somehow it doesnt, looks like time didnt catch up still.

Turn out I manually setup the time by logging into the storage gateway VM and set it up. After that I proceed with VM restart and then port 80 is coming up. Now I could activate the VM gateway.

Reset root password for AWS Storage Gateway VM

I am not sure if that would violate the terms and conditions of using AWS Storage Gateway VM, what I know is that the default sguser is pretty restrictive and not much troubleshoot could be done from that user (well, I kept seeing activation failure when tried to activate the Storage Gateway service and it looks like port 80 was rejecting the requests for some reasons).

So, as a last resort I tried to "break in" the VM by booting it into single user mode so that I could reset the password there (thanks God, as long as it is still a general Linux).

To reset the password, hit "e" at the Grub menu, use the up-down cursor to scroll to the line start with "Kernel". Once you are on that line, press "e" and then append "boot single" to the end of the line and press enter and "b" to boot. After that the VM will be booted to single user mode and you could simply type "password root" to reset the root password.

So what makes it good to reset the root password? You could get into this VM and do further troubleshooting. In my case, I get into the AWS Storage Gateway VM and found that port 80 didnt come up at all for some reason so that I could check on something else.(though, with further investigation, it looks like i didnt configure proper ntp options for the VM and ESX host)

Tricks to avoid DHCP client to override /etc/resolv.conf

I have a laptop installed with Ubuntu and is using DHCP client to connect to the Internet in couples of locations. Most of the DHCP server love to offer DHCP IP bundled with DNS addresses which is kind of convenience if one dont have their own DNS. For some reason, I have to use my own DNS server to perform DNS lookup and this DHCP kindness is getting annoying as I have to update the resolv.conf everytime I got the DHCP IP.

Just think of a trick to lock the /etc/resolv.conf from overwriting by doing chattr +i, i.e.

[root@ ~]# lsattr /etc/resolv.conf
------------- /etc/resolv.conf
[root@ ~]# chattr +i /etc/resolv.conf
[root@ ~]# lsattr /etc/resolv.conf
----i-------- /etc/resolv.conf

After that the file /etc/resolv.conf would be locked from writing until removal of this tag. I tested it by appending some crap to the /etc/resolv.conf but it doesnt allow me to write over.

[root@ ~]# echo some-crap >> /etc/resolv.conf
-bash: /etc/resolv.conf: Permission denied

Now I could keep using my own DNS and no need to update the file all the time.

Falling back is easy.

[root@ ~]# chattr -i /etc/resolv.conf
[root@ ~]# lsattr /etc/resolv.conf
------------- /etc/resolv.conf

Turn on bash history timestamp

Bash history time-stamping is not something new but it is not enabled as default in most Linux distro and hence not much people really know about it. I found this is really useful especially when you want to trace back user activities on server. So, it really worth a minute to turn it on. 

To turn it on, you can either add the below parameter to systems' bashrc (i.e. /etc/bashrc in CentOS/Fedora/RHEL, /etc/bash.bashrc in Ubuntu/Debian) or your own bashrc (~/.bashrc)

export HISTTIMEFORMAT="%d.%m.%y %T "


Once you added above parameter to bashrc, logout and login and issues some command and then check back the history, you will see the timestamp is added.

# history
...
  127  21.05.11 22:10:56 uptime
  128  21.05.11 22:11:12 su - admin
  129  21.05.11 22:11:15 exit
  130  21.05.11 22:12:19 su - admin
  132  21.05.11 22:12:33 exit
  133  21.05.11 22:13:56 ps auxww
  134  21.05.11 22:15:43 pwd
  135  21.05.11 22:17:56 ls
  136  21.05.11 22:20:56 sudo su -
  137  21.05.11 22:23:56 exit
...

So, the magic here is the option HISTTIMEFORMAT. This option making use of strftime format. so%d %m %y %T means

%d - Day
%m - Month
%y - Year
%T - Time

To know more, one can always type "help history", "man bash" and "man strftime".

Howto install python easy_install on ubuntu 11.10

easy_install will help installing python modules and here is how we can install it on Ubuntu 11.10

root@ubuntu:~# apt-get install python-setuptools
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  python-pkg-resources
Suggested packages:
  python-distribute python-distribute-doc
The following NEW packages will be installed:
  python-setuptools
The following packages will be upgraded:
  python-pkg-resources
1 upgraded, 1 newly installed, 0 to remove and 61 not upgraded.
Need to get 274 kB of archives.
After this operation, 1,061 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://us.archive.ubuntu.com/ubuntu/ oneiric-updates/main python-pkg-resources all 0.6.16-1ubuntu0.1 [62.7 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu/ oneiric-updates/main python-setuptools all 0.6.16-1ubuntu0.1 [212 kB]
Fetched 274 kB in 0s (393 kB/s)       
(Reading database ... 52429 files and directories currently installed.)
Preparing to replace python-pkg-resources 0.6.16-1 (using .../python-pkg-resources_0.6.16-1ubuntu0.1_all.deb) ...
Unpacking replacement python-pkg-resources ...
Selecting previously deselected package python-setuptools.
Unpacking python-setuptools (from .../python-setuptools_0.6.16-1ubuntu0.1_all.deb) ...
Setting up python-pkg-resources (0.6.16-1ubuntu0.1) ...
Setting up python-setuptools (0.6.16-1ubuntu0.1) ...

So how to use easy_install? We are going to install python module paste to illustrate the idea.


root@ubuntu:~# easy_install -U paste
Searching for paste
Reading http://pypi.python.org/simple/paste/
Reading http://pythonpaste.org
Best match: Paste 1.7.5.1
Downloading http://pypi.python.org/packages/source/P/Paste/Paste-1.7.5.1.tar.gz#md5=7ea5fabed7dca48eb46dc613c4b6c4ed
Processing Paste-1.7.5.1.tar.gz
Running Paste-1.7.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-I0cKPn/Paste-1.7.5.1/egg-dist-tmp-cdqwYD
warning: no previously-included files matching '*' found under directory 'docs/_build/_sources'
Adding Paste 1.7.5.1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/Paste-1.7.5.1-py2.7.egg
Processing dependencies for paste
Finished processing dependencies for paste

Reset root password for Ubuntu 10.04 LTS

Oops, I am trying to reset root password for a server with Ubuntu 10.04 LTS but the grub menu just doesn't come up. So it turns out the interactive grub menu is  disabled since GRUB2. And what we need to do to get into the GRUB menu is to hold the shift key during boot. According official documentation, pressing the ESC key may also display the menu sometime too (Reference here)

So once you get into the menu, move your text cursor to the end of the line starting with "LINUX", append "init=/bin/bash panic=3" and press with CTRL+X to start booting to single user mode.

Once you are booted to single user mode, you will be shown wilt login prompt 

root@(none)~# 

You could now type "mount -o remount,rw /" to remount / directory to change the root mount from read-only to read-writable. Now you could issue "password" command to update the password on this server.

Recover corrupted LVM root partition by fsck

So you have a Linux machine that is installed with LVM and you have assigned root partition to sit on that LVM pool. One day your machine is crashed and the root partition is corrupted. It failed to boot properly and asked you to fill in root password to run fsck. Sadly, you lost the root password and you need a recovery CD (or installation CD) to boot and recover the disk.

Now your machine is booted and you found that your just couldn't run fsck against a LVM partition.

root@test:/# fdisk -l

Disk /dev/vda: 21.5 GB, 21474836480 bytes
16 heads, 63 sectors/track, 41610 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00014ef8

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3         389      194560   83  Linux
/dev/vda2             391       41609    20773888   8e  Linux LVM
So rescue CD by default just won't automatically activate the LVM (and its underlaying volumes). What you need to do is to activate the LVM partition. and then run fsck on the volumes.

### Run lvm from rescue CD
bash-4.1# lvm

### This will scan and list the PV
lvm> pvscan 
   PV /dev/vda2   VG volgroup01   lvm2 [19.78GiB/ 0     free]
   Total: 1 [19.78 GiB] / in use: 1 [19.78 GiB] / in no VG: 0 [0    ]

### This will scan and list the VG
lvm> vgscan
   Reading all physical volumes. This may take a while...
   Found volume group "volgroup01" using metadata type lvm2

### This will list and scan the LV (the meat is here)
lvm> lvscan
   inactive                 '/dev/volgroup01/root'    [11.78 GiB] inherit
   inactive                 '/dev/volgroup01/swap'  [8.00 GiB] inherit

### And then execute lvchange
bash-4.1# lvchange -ay /dev/volgroup01/root

### So quit the lvm
lvm > exit
Adding dirhash hint to filesystem

### now time to run fsck
bash-4.1# fsck -y /dev/volgroup01/root