This is a blog post originally from Jeff Slapp, director, systems engineering and solution architecture at DataCore Software. See the full post here:
https://www.linkedin.com/pulse/datacore-storage-tiering-amazon-s3-jeffrey-slapp?trk=prof-post
INTRODUCTION
One of the strengths of software-driven architectures is how software interoperates (or should interoperate) across various hardware platforms. This should
also be equally true when it comes to interoperating with other software-driven architectures.
This multi-part series will cover how DataCore interoperates with various layers within the Amazon Web Services (AWS) platform. Part One of the story will
begin with linking DataCore’s auto-tiering feature with Amazon S3 via the AWS Storage Gateway. This overview is not intended to be a comprehensive
technical implementation guide, but rather a brief introduction to how these two technologies work together based on well-established industry standards.
For additional background information on DataCore’s auto-tiering feature, please visit DataCore’s website, or view this short three minute video.
ARCHITECTURE OVERVIEW
Let’s begin by understanding the relationship between the various components in the stack. Below is a basic architecture view leveraging Hyper-V to host
the guest AWS Storage Gateway VM directly on the DataCore storage engine.
From DataCore’s perspective, the AWS Storage Gateway is simply an iSCSI storage target presenting the raw volume(s) referred to as Cached Volumes. Once the
iSCSI target volumes are seen by DataCore and are made members of the DataCore pool, the volumes become a tier of storage like any other disks attached to
the engine.
Below is a more detailed view of a full enterprise deployment. Any internal or external system utilizing storage volumes from DataCore can take advantage
of the AWS storage tier.
FIRST THINGS FIRST
Before we begin, you will need to have a DataCore engine up and running and an active Amazon AWS account.
We will start with the AWS side of things. When you log into your AWS account, click on the orange cube in the upper left-hand corner. This will bring you
to the full Amazon Web Services menu. From here, click on the Storage Gateway option under the Storage and Content Delivery section.
The first step is to select the Gateway Type. Since it will be necessary to maintain a local copy of the data for high-speed access (referred to as a
cached volume), the gateway type will need to be set to Cached.
The next step is to select the type of virtual machine which will act as the storage gateway and iSCSI target for the local DataCore engine.
Click on the appropriate virtual machine type and select the Download button. This article will show the AWS Storage Gateway VM running within Hyper-V
since Hyper-V can reside directly on the DataCore engine for low latency access. Once downloaded, import the VM into Hyper-V using the Hyper-V Manager.
FINAL STEPS
Once imported, modify the VM to include at least two additional virtual disks. One disk will be used for the Upload Buffer and the other for the Cache
Volume (where the data will locally reside). Once this is set, start the VM. The VM operating system will boot and load all the necessary services to
connect to AWS and provide iSCSI target access to the initiator used later to mount the storage volume on the DataCore engine.
When you see the login prompt appear in the VM console, login using the default credentials (sguser, sgpassword). You will need to either configure a
static IP address for the gateway or retrieve the DHCP address which has been assigned. This will be used in the next step.
In the AWS Web Interface, enter the IP address of the AWS Storage Gateway VM. This address will most likely be a private address behind your corporate
firewall. To continue, click the Being Activation button.
Once activated you will need to specify the disks to use for the Upload Buffer and Cache Volumes. The choices presented for the available disks should
match the two volumes you created earlier when the VM was first imported.
iSCSI CHAP authentication passwords can also be configured for the Cached Volume, if required.
At this point the gateway’s iSCSI target and Cached Volume(s) should now be ready for use. To attach the volume, simply open the iSCSI control panel and
connect to the gateway’s local IP address.
When the iSCSI interface shows as connected, perform a rescan from within the Disk Management interface on the DataCore engine. The volume should now be
visible to DataCore. The final step is to add the AWS Cached Volume to a disk pool.
Once the Cached Volume is in the pool, set the appropriate disk tiering value respective of the other disks in the pool (typically this will be the
last tier in your pool). The new AWS S3 tier is now ready for production use.
If using a single DataCore engine, you can add Cached Volume redundancy by using Disk Pool Mirroring. This will require another Cached Volume of the
exact same size as the original from another AWS Storage Gateway VM. If using HA DataCore engines, you can introduce one gateway to each engine and
DataCore will synchronously mirror across them. In either case, the two gateways can be linked with different Region and Availability Zones for service
diversity.
CONCLUSION
This concludes the introductory overview. For more detailed information related to configuring the storage gateway, please see the AWS Getting Started
Guide located here.
