AWS Storage Gateway, can it be seated behind NAT?

Recently I was testing the AWS Storage Gateway. AWS storage gateway is a product combing AWS console frontend and a ESXi-based Linux VM (we call it storage gateway VM). The AWS console is responsible to handle user instruction on the storage gateway and pass the instruction to storage VM (e.g. create iscsi target on Storage VM, take or restore snapshot ... etc) while the ESXi-based storage VM is the actual host handling the instruction and storage thing.

From observation, there will be 2 ports be listening on the storage gateway VM, they are TCP port 80 and 3260. Port 80 is actually a java instance which is responsible to serve API call (user submit the request via AWS console or AWS API, and then AWS pass the request to the API handler on port 80 of Storage VM). Port 3260 is the ISCSI target which is responsible to handle all ISCSI request..

So, i was asked if it is possible to run this storage gateway VM behind NAT, i.e. sitting the VM in private network. With this subjective, there are 2 possible scenarios,

 - one is sitting the VM behind NAT but without any port mapping on port 80 and 3260
 - while another scenario is putting the VM behind NAT but with port mapping enabled (i.e. exposing and mapping the port 80 and 3260 on wan outside to port 80 and 3260 on the VM with private IP address).


Unfortunately both scenario wouldn't work. For first scenario, though the storage VM could be activated without problem, AWS just not able to communicate with the storage gateway VM therefore all instruction from users wouldn't be passed over to storage gateway VM at all. No ISCSI target could be created, no snapshot could be created or restored as all instructions are pended and timed out. For second scenario, we could activate the storage gateway VM, create volume, create and restore snapshot but the ISCSI target is just not working at all due to the ISCSI implementation restriction. ISCSI initiator (or ISCSI guest) could discover the ISCSI target via port 3260 but it just couldn't login to the resources.
 
To explain why it wont work behind NAT, we will have to go through the process in connecting or mapping an ISCSI target.

The ISCSI connection establishment process is actually a two-step process. The first step would be the iscsi initiator to scan and discover iscsi remote resource on the remote iscsi target. During the test, we could successfully perform this step as we do see the iscsi target during iscsi discovery. However, on the 2nd step when we tried to login into the ISCSI resource and then we see issue. The situation is that, iscsi resource is presented with a combination of on-host ip and iqn, e.g "10.1.1.1, iqn-name". The ip address here is the ip on the host, therefore that is NATTed private address. Once iscsi initiator(the guest VM) try to map the remote resource, due to implementation restriction it will connect to the ip address being presented, therefore the private ip address. As the IP address presented is private, VM initiator from public network wouldn't be able to talk to that IP address and connecting to the target would lead to request timeout.

The only possible workaround we could think of right now is to create a VPN tunnel between Initiator and the storage gateway VM behind NAT. In this way the AWS storage VM's iSCSI targets can be seen as they would be on the same LAN segment.  However with this approach it would definitely add extra overhead on the ISCSI's I/O performance.

BTW, this storage appliance is designed for access from on-premise device (i.e., natively they should be on the same network segemtn) and which means the ISCSI traffic should not really need to get through public network. In situation like this the appliance should be good to use.

AWS storage gateway: WORKING STORAGE NOT CONFIGURED

As continuing the test on AWS Storage gateway, I found that there is an implicit requirement of the AWS storage VM, i.e. the VM have to be assigned with a publicly accessible IP address, or at least the IP address could be reached by AWS network.

The logic behind is that when someone trying to manage the AWS storage VM via AWS web console, the instruction will have to be passed over to the VM (possibly via port 80 of the AWS VM, but I didnt confirm it yet) via public network. In any case AWS failed to reach the VM, it will not able to proceed with the instruction.

The above idea was tested against an internal VM I was playing with yesterday. The VM is sit on private network (e..g 192.168.x.x) with outgoing NAT enable but not incoming NAT enable. I could successfully proceed with the VM activation but no volumes could be added from AWS console. The newly added volumes keep showing "WORKING STORAGE NOT CONFIGURED" on AWS console which basically means that it is not creating at all. Usually, creating a new volume should not take too long at all.

Here is the screenshot though,

Apart from volumes creation failure, I also tried adding new virtual disk to the storage VM and see if AWS could see the new virtual disk. However, the answer is no. So what I could pretty sure here is that AWS will have to talk to VM and it just wont be able to put the Storage VM on an internal network segment which is not accessible from public.

AWS storage gateway port 80 connection refused

Was testing on AWS storage gateway services in my ESXi host.  So what is AWS storage gateway? Basically it is a AWS service to host behind your on-premise firewall. An AWS customized VM living on ESXi will be put on-premise and allow remote storage management from AWS console. The VM on ESXi will present storage via iscsi which allow remote read-write access..

I was following this documents but unfortunately I jump into the condition while I was trying to activate my storage gateway VM. It looks like activation require access on port 80 but unfortunately I see nothing on port 80. Running nmap against the VM but I see nothing coming up.


root@localhost:~$ nmap -sT 1.2.3.4


Starting Nmap 5.00 ( http://nmap.org ) at 2012-03-29 16:13 HKT
Interesting ports on 1.2.3.4:
Not shown: 996 filtered ports
PORT     STATE  SERVICE
22/tcp   closed ssh
80/tcp   closed http
631/tcp  closed ipp
3260/tcp closed iscsi

 I tried to get into the storage VM as root (prior to the that, I booted the VM to single user mode and reset the password, for details see here) and found that no services are running on those ports.

With further checking, it seems that it have to do with the time service on the VM. The storage VM will need an very accurate time or otherwise it will not be coming up. As per the suggestion from setup guide, I enabled ntp service on ESXi and then enable the "Synchronize guest time with host" option on VM followed by VM restart. I hope it would work but somehow it doesnt, looks like time didnt catch up still.

Turn out I manually setup the time by logging into the storage gateway VM and set it up. After that I proceed with VM restart and then port 80 is coming up. Now I could activate the VM gateway.

Reset root password for AWS Storage Gateway VM

I am not sure if that would violate the terms and conditions of using AWS Storage Gateway VM, what I know is that the default sguser is pretty restrictive and not much troubleshoot could be done from that user (well, I kept seeing activation failure when tried to activate the Storage Gateway service and it looks like port 80 was rejecting the requests for some reasons).

So, as a last resort I tried to "break in" the VM by booting it into single user mode so that I could reset the password there (thanks God, as long as it is still a general Linux).

To reset the password, hit "e" at the Grub menu, use the up-down cursor to scroll to the line start with "Kernel". Once you are on that line, press "e" and then append "boot single" to the end of the line and press enter and "b" to boot. After that the VM will be booted to single user mode and you could simply type "password root" to reset the root password.

So what makes it good to reset the root password? You could get into this VM and do further troubleshooting. In my case, I get into the AWS Storage Gateway VM and found that port 80 didnt come up at all for some reason so that I could check on something else.(though, with further investigation, it looks like i didnt configure proper ntp options for the VM and ESX host)