This blog post is part one of a 3-part series describing a technique for how to export and import Amazon Web Services (AWS) instance volume images. Read the full introduction for more details.
In this post, I’ll demonstrate a technique using standard Linux tools to export an AWS volume to another cloud provider or on-prem storage. For the demonstration, I created an image of a decommissioned ownCloud server and then transferred this image to Google Drive for long-term archiving.
This blog series assumes familiarity with Linux and Amazon AWS.
I used a Linux server running in EC2 to export and import the images. Any flavor of Linux works; I used Ubuntu 16.04. I refer to this Linux server running in EC2 as the “build” server. On the build server, you’ll need at least twice as much free storage space as the size of the EC2 volume image.
This technique works for both decommissioned servers (stopped) and servers actively running in EC2.
How to Create an Image
Step 1: Take a Snapshot
Locate the EC2 instance to be imaged, identify its boot volume, and create a snapshot.
Step 2: Create a Volume
Create a volume from the snapshot. Be sure that the volume is created in the same availability zone as your build server, otherwise your server won’t be able to mount it.
Step 3: Attach the Volume
The new, just-created volume will now appear as “available” in the Volumes tab. Attach this volume to your build server.
Step 4: Locate the Volume
Log into your build server, become root, and locate the volume you just attached:
# fdisk -l
In this example, the volume to be imaged is the 8GB volume which is now connected as device /dev/xvdf.
Note that this device name doesn’t match what was specified in the “Attach Volume” window earlier; the actual device name is provided by your Linux kernel. I have never seen an attached volume show up with the “/dev/sdx” device name, but your experience may differ.
Step 5: Image the Volume
Use the “dd” command to make an exact copy of this disk to a file on the build server. Make sure the partition where the file is to be stored has enough free space!
# dd if=/dev/xvdf of=/mnt/scratch/owncloud_image_23Jan19.img
Depending on the size of your disk, this can take a while.
When the image is complete, compare the image file’s size to the size of the disk as reported by fdisk. In this example, both report a size of 8589934592 bytes.
Step 6: Test the Image
Use fdisk to read the contents of the image file and display the partition table.
# fdisk -lu filename
The size of the disk and other details of the partition should match the source EC2 volume.
Step 7: Double-Check the Image
Next, mount the image and explore the file system to ensure the expected data is present.
Since we created a disk image and not a partition image, we must first determine the byte offset that corresponds to the start of the partition contained within the image file. Look at the output of the “fdisk -lu” command and note the sector size and the starting sector of the partition you would like to mount.
In this example, the sector size is reported as “512 bytes” and the start of the first partition is “2048.” So, 512 bytes per sector multiplied by 2048 sectors means that the beginning of the partition is at a byte offset of 1048576 bytes.
Now that the offset is known, prepare to mount the partition.
# mount -o loop,offset=1048576 owncloud_image_23Jan19.img /mnt/test
The partition will now be mounted under the specified mount point.
By default, Linux will refuse to mount a dirty NTFS filesystem. If you created a snapshot from a running Windows server, your image will be in this state. To bypass this restriction, mount the filesystem as read-only. You can then explore the filesystem to verify the image.
# mount -o ro,force,loop,offset=1084875 FireOakDC04.24Jan2019.img /mnt/test
Step 8: Compress the Image
Your image likely contains a significant amount of free space, so compress it using gzip, bzip2, or a compressor of your choice. In this example, a 20GB image compressed to 13GB using gzip. Be patient, this will take a while to complete.
Step 9: Copy the Image out of AWS
The compressed volume image is now ready for long-term storage. You can now transfer it to an archival storage repository of your choice.
To transfer to other cloud services, I highly recommend the excellent rclone utility. After you’ve installed rclone and have configured a remote, transferring the file is straightforward.
Rclone compares file checksums during copy operations, so you can be confident that the file has been transferred correctly. If you have a high-value image, be sure to test the image before removing the volumes and associated snapshots from your AWS account.
In my next post, I’ll demonstrate how to restore an image to create a bootable AWS instance. In part 3 of this series, I’ll demonstrate how to automate the export process.
Eric Smith is the FireOak Strategies Chief Technologist and Information Security Officer.
