Using kpartx to mount partition(s) from disk image.

Scenario:

You got a disk image which was just dumped via dd (e.g. dd if=/dev/hda of=/disk.img). There are 3 partitions on your source disk /dev/hda and they looks like this,


[root@localhost /]# fdisk -l


Disk /dev/hda: 75.1GB, 75161927680 bytes
255 heads, 63 sectors/track, 9137 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes


   Device  Boot      Start       End       Blocks   Id   System
/dev/hda1 *                  1         25       200781  83   Linux
/dev/hda2                   26     1330  10482412+ 82   Linux swap / Solaris
/dev/hda3               1331     9137  62709727+ 83   Linux
 
Now, as long as your disk image /disk.img is a complete dump of /dev/hda, you should be able to mount the partitions within the disk image via loopback device with proper offset value. For e.g.

#### Create the mount point
[root@localhost /]# mkdir -p /chroot/boot

#### Mount the first partition with offset (Start * sectors/track * 512, i.e. 1*63*512)
[root@localhost /]# mount -o loop,offset=$((1*63*512)) /disk.img /chroot/boot
[root@localhost /]# mount | grep /chroot/boot
/disk.img on /chroot/boot type ext3 (rw,loop=/dev/loop1,offset=32256)

So you successfully mounted the first partition (/dev/hda1) and then planning to mount the 3rd partition (/dev/hda3) to /chroot

#### Try mounting the 3rd partition with offset 1331
[root@localhost /]# mount -o loop,offset=$((1331*63*512)) /disk.img /chroot
hfs: unable to find HFS+ superblock
mount: you must specify the filesystem type

Apparently there is something wrong with mount and for some reason it didn't handle offset very well. The util-linux on my box is ver 2.13 which claims to support offset higher than 32bit (well i didn't do any deeper for this specific matters though) but unfortunately it didn't help. As what I want is a quick fix, I come across a tools called "kpartx"which is actually a swiss knife to mount partitions within a disk image file. So here is a demo of how it works.

Solutions:

To list partitions on a disk image. So in this example kpartx see 3 partitions from image disk.img
[root@localhost /]# kpartx -l /disk.img
loop0p1 : 0 401562 /dev/loop 63
loop0p2 : 0 20964825 /dev/loop 401625
loop0p3 : 0 21366450 /dev/loop 21366450

To activate these partitons, we can run kpartx with option -av
[root@localhost /]# kpartx -av /disk.img
add map loop0p1 : 0 401562 linear /dev/loop 63
add map loop0p2 : 0 20964825 linear /dev/loop 401625
add map loop0p3 : 0 21366450 linear /dev/loop 21366450

We can see the device is now mounted as loopback device and presented via /dev/mapper
[root@localhost /]# losetup -a
/dev/loop0: [0800]:49154 (/disk.img)
[root@localhost /]# ls /dev/mapper/loop0p*
/dev/mapper/loop0p1  /dev/mapper/loop0p2  /dev/mapper/loop0p3


Lets see if we could mount these loopback partitions.

[root@localhost /]# mount /dev/mapper/loop0p1 /chroot/boot
[root@localhost /]# mount /dev/mapper/loop0p3 /chroot
[root@localhost /]# mount | grep chroot
/dev/mapper/loop0p1 on /chroot/boot type ext3 (rw)
/dev/mapper/loop0p3 on /chroot type ext3 (rw)

So it looks like the partitions are mounted and file systems on it are recognized without any issue. Let say we finished with the operations on these partitions and now we plan to unmount it and clean it up.

[root@localhost /]# umount /chroot/boot /chroot
[root@localhost /]# losetup -d /dev/mapper/loop0p1
[root@localhost /]# losetup -d /dev/mapper/loop0p2
[root@localhost /]# losetup -d /dev/mapper/loop0p3
[root@localhost /]# kpartx -d /test.img

Pretty much it.

Nice webcast, Diagnosing Intermittent MySQL Problems

http://event.on24.com/r.htm?e=440046&s=1&k=4E1095FD06B3CA383BC5CD02DC93714C

Reinstalling GRUB after upgrade from Ubuntu 9 to Ubuntu 10.04

So I just upgraded my Ubuntu 9.10 desktop (well, EOL for longtime) to a more recent release, Ubuntu 10.04 LTS Lucid Lynx. So far the upgrade was pretty smooth except it took me like 3 hours to download and complete all the installation files.

Unfortunately it shows something like this after the reboot.

GRUB loading.
error: the symbol 'grub_puts' not found

grub rescue>

So I thought there is something going on with the GRUB during the upgrade, I grep a Ubuntu 10.04 Desktop iso (make sure it is Desktop iso instead of server iso, so that we could boot it up to Live CD mode. ) and boot it up to perform system rescue.

Once the Live CD is booted, I mounted / to somewhere under /mnt

/dev/sda1 on /mnt type ext4 (rw)

I tried chroot to /mnt and run grub-install from there, no dice.

root@ubuntu:/# grub-install  --force /dev/sda
/usr/sbin/grub-probe: error: cannot find a device for /boot/grub (is /dev mounted?).
No path or device is specified.
Try `/usr/sbin/grub-probe --help' for more information.
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.

So it is apparently the grub-installation on the disk is corrupted or something, I quited from chroot mode and simply run grub-install from the Live CD.

root@ubuntu:~# grub-install --root-directory=/mnt/ /dev/sda
Installation finished. No error reported.

After a reboot my machine is booting without any issue.

Good site for python egg explanation.

Just so much better than perl's CPAN.

http://peak.telecommunity.com/DevCenter/PythonEggs

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ABC at /etc/puppet/manifests/nodes.pp:14 on node XYZ

So this error is pretty confusing to understand at first glance,

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class ABC at /etc/puppet/manifests/nodes.pp:14 on node XYZ

Basically it is kind of complaining on missing of specific modules ABC. There are 2 things could be checked here,

1. Make sure you have define the modulepath parameter per the document
http://docs.puppetlabs.com/puppet/2.7/reference/modules_fundamentals.html

2. Make sure you have properly define the class name within ABC/manifests/init.pp (well it happens to me couples of time and it turns out there was a typo within the init.pp)

So below init.pp is deemed to see the declared class not found error.


# cat /etc/puppet/modules/ABC/manifests/init.pp
class ABCtypo {

  exec { "blah ....":
       }

}

How to add a puppet client to puppet master.

Before a new puppet client be allowed to fetch manifest from puppet server, the client will have to be signed and below command would do the job

[root@puppetclient ~]# puppet agent --server puppetmaster --test --waitforcert 30



The above command will execute puppet as agent mode and connect to server puppetmaster (** remote server name in here have to match the remote server hostname or otherwise client agent will come up with error "err: Could not retrieve catalog from remote server: hostname was not match with the server certificate"). Option "--test" means the agent will be executed in test mode and then --waitforcert 30 means the puppet client will wait for 30 seconds for server to sign up the certificate. If 30 seconds passed and the client certificate is still not signed, the client agent will stop and exit.

So on server, below command would list out the certs pending for approval

root@puppetmaster:~# puppetca --list
  puppetclient
(CC:2B:2B:9D:4A:EF:3F:15:EF:60:C7:73:C9:18:FF:D1)


root@puppetmaster:~# puppetca --sign puppetclient
notice: Signed certificate request for puppetclient
notice: Removing file Puppet::SSL::CertificateRequest puppetclient at '/var/lib/puppet/ssl/ca/requests/puppetclient'

AWS Storage Gateway, can it be seated behind NAT?

Recently I was testing the AWS Storage Gateway. AWS storage gateway is a product combing AWS console frontend and a ESXi-based Linux VM (we call it storage gateway VM). The AWS console is responsible to handle user instruction on the storage gateway and pass the instruction to storage VM (e.g. create iscsi target on Storage VM, take or restore snapshot ... etc) while the ESXi-based storage VM is the actual host handling the instruction and storage thing.

From observation, there will be 2 ports be listening on the storage gateway VM, they are TCP port 80 and 3260. Port 80 is actually a java instance which is responsible to serve API call (user submit the request via AWS console or AWS API, and then AWS pass the request to the API handler on port 80 of Storage VM). Port 3260 is the ISCSI target which is responsible to handle all ISCSI request..

So, i was asked if it is possible to run this storage gateway VM behind NAT, i.e. sitting the VM in private network. With this subjective, there are 2 possible scenarios,

 - one is sitting the VM behind NAT but without any port mapping on port 80 and 3260
 - while another scenario is putting the VM behind NAT but with port mapping enabled (i.e. exposing and mapping the port 80 and 3260 on wan outside to port 80 and 3260 on the VM with private IP address).


Unfortunately both scenario wouldn't work. For first scenario, though the storage VM could be activated without problem, AWS just not able to communicate with the storage gateway VM therefore all instruction from users wouldn't be passed over to storage gateway VM at all. No ISCSI target could be created, no snapshot could be created or restored as all instructions are pended and timed out. For second scenario, we could activate the storage gateway VM, create volume, create and restore snapshot but the ISCSI target is just not working at all due to the ISCSI implementation restriction. ISCSI initiator (or ISCSI guest) could discover the ISCSI target via port 3260 but it just couldn't login to the resources.
 
To explain why it wont work behind NAT, we will have to go through the process in connecting or mapping an ISCSI target.

The ISCSI connection establishment process is actually a two-step process. The first step would be the iscsi initiator to scan and discover iscsi remote resource on the remote iscsi target. During the test, we could successfully perform this step as we do see the iscsi target during iscsi discovery. However, on the 2nd step when we tried to login into the ISCSI resource and then we see issue. The situation is that, iscsi resource is presented with a combination of on-host ip and iqn, e.g "10.1.1.1, iqn-name". The ip address here is the ip on the host, therefore that is NATTed private address. Once iscsi initiator(the guest VM) try to map the remote resource, due to implementation restriction it will connect to the ip address being presented, therefore the private ip address. As the IP address presented is private, VM initiator from public network wouldn't be able to talk to that IP address and connecting to the target would lead to request timeout.

The only possible workaround we could think of right now is to create a VPN tunnel between Initiator and the storage gateway VM behind NAT. In this way the AWS storage VM's iSCSI targets can be seen as they would be on the same LAN segment.  However with this approach it would definitely add extra overhead on the ISCSI's I/O performance.

BTW, this storage appliance is designed for access from on-premise device (i.e., natively they should be on the same network segemtn) and which means the ISCSI traffic should not really need to get through public network. In situation like this the appliance should be good to use.