Web Analytics

FireOak Strategies Blog

Insights and articles related to knowledge management, information security, technology, data and analytics, business process automation, platform management, and other related topics, from our experienced team of consultants.

Part 1: Create and Export an AWS EC2 Volume Image

In this post, I’ll demonstrate a technique using standard Linux tools to export an AWS volume to another cloud provider or on-prem storage.


AWS EC2

This blog post is part one of a 3-part series describing a technique for how to export and import Amazon Web Services (AWS) instance volume images. Read the full introduction for more details.

In this post, I’ll demonstrate a technique using standard Linux tools to export an AWS volume to another cloud provider or on-prem storage. For the demonstration, I created an image of a decommissioned ownCloud server and then transferred this image to Google Drive for long-term archiving.

This blog series assumes familiarity with Linux and Amazon AWS.

I used a Linux server running in EC2 to export and import the images. Any flavor of Linux works; I used Ubuntu 16.04. I refer to this Linux server running in EC2 as the “build” server. On the build server, you’ll need at least twice as much free storage space as the size of the EC2 volume image.

This technique works for both decommissioned servers (stopped) and servers actively running in EC2.

How to Create an Image

Step 1: Take a Snapshot

Locate the EC2 instance to be imaged, identify its boot volume, and create a snapshot.

Take a Snapshot of Server Image

Step 2: Create a Volume

Create a volume from the snapshot.  Be sure that the volume is created in the same availability zone as your build server, otherwise your server won’t be able to mount it.

Step 3: Attach the Volume

The new, just-created volume will now appear as “available” in the Volumes tab. Attach this volume to your build server.

Attach the Volume in AWS

Step 4: Locate the Volume

Log into your build server, become root, and locate the volume you just attached:

# fdisk -l

In this example, the volume to be imaged is the 8GB volume which is now connected as device /dev/xvdf.

Locate the Volume

Note that this device name doesn’t match what was specified in the “Attach Volume” window earlier; the actual device name is provided by your Linux kernel. I have never seen an attached volume show up with the “/dev/sdx” device name, but your experience may differ.

Step 5: Image the Volume

Use the “dd” command to make an exact copy of this disk to a file on the build server.  Make sure the partition where the file is to be stored has enough free space!

# dd if=/dev/xvdf of=/mnt/scratch/owncloud_image_23Jan19.img 
Image the Volume

Depending on the size of your disk, this can take a while.    

When the image is complete, compare the image file’s size to the size of the disk as reported by fdisk.  In this example, both report a size of 8589934592 bytes.

Review the Image

Step 6: Test the Image

Use fdisk to read the contents of the image file and display the partition table.

 # fdisk -lu filename
Test the Image

The size of the disk and other details of the partition should match the source EC2 volume.

Step 7: Double-Check the Image

Next, mount the image and explore the file system to ensure the expected data is present.

Since we created a disk image and not a partition image, we must first determine the byte offset that corresponds to the start of the partition contained within the image file.   Look at the output of the “fdisk -lu” command and note the sector size and the starting sector of the partition you would like to mount. 

In this example, the sector size is reported as “512 bytes” and the start of the first partition is “2048.” So, 512 bytes per sector multiplied by 2048 sectors means that the beginning of the partition is at a byte offset of 1048576 bytes.

Now that the offset is known, prepare to mount the partition. 

 # mount -o loop,offset=1048576 owncloud_image_23Jan19.img /mnt/test

The partition will now be mounted under the specified mount point. 

Mount the Partition

By default, Linux will refuse to mount a dirty NTFS filesystem.  If you created a snapshot from a running Windows server, your image will be in this state.  To bypass this restriction, mount the filesystem as read-only.  You can then explore the filesystem to verify the image.

 # mount -o ro,force,loop,offset=1084875 FireOakDC04.24Jan2019.img /mnt/test
Mount Windows Image

Step 8: Compress the Image

Your image likely contains a significant amount of free space, so compress it using gzip, bzip2, or a compressor of your choice.   In this example, a 20GB image compressed to 13GB using gzip.  Be patient, this will take a while to complete.

Compress the Image

Step 9: Copy the Image out of AWS

The compressed volume image is now ready for long-term storage. You can now transfer it to an archival storage repository of your choice. 

To transfer to other cloud services, I highly recommend the excellent rclone utility.    After you’ve installed rclone and have configured a remote, transferring the file is straightforward.

Transfer the Image File

Rclone compares file checksums during copy operations, so you can be confident that the file has been transferred correctly.  If you have a high-value image, be sure to test the image before removing the volumes and associated snapshots from your AWS account.

In my next post, I’ll demonstrate how to restore an image to create a bootable AWS instance. In part 3 of this series, I’ll demonstrate how to automate the export process.

Eric Smith is the FireOak Strategies Chief Technologist and Information Security Officer.


From the FireOak Blog

News and insights from the FireOak team about managing, securing, and sharing knowledge