Points in waveform

Enable mass migrations from on-premises to AWS

How to enable mass migrations to AWS in an enterprise environment with AWS Transfer Family and AWS Service Catalog

February 05 2021 Artur Schneider

One of the challenges of migrating data to AWS is that you need to use the AWS APIs that might not be natively supported by an application or less known to the application team than traditional protocols like SFTP. AWS has resolved this limitation by introducing the AWS Transfer Family service. It is an SFTP or FTP gateway for S3 buckets or EFS, fully managed by AWS and integrated into the platform, esp. regarding access, permissions or security.

Advantages of AWS Transfer Family service:

  • Fully managed and serverless AWS Service for file transfer to S3 or EFS
  • Supports traditional widely used protocols like SFTP, FTPS or FTP
  • Key Based Authentication (No IAM Accounts required)
  • Access through private VPC network possible (multi AZ configuration supported)
  • Easy access to almost unlimited and low cost cloud storage (when using the S3 as destination)
  • Connect through custom DNS name
  • Comes with security features such as folder restriction and event logging

Security:

Eye with binary code

SFTP stands for Secure File Transfer Protocol and indeed in contrast to FTP the transfer of commands and data is cryptographically secured. Another advantage - especially compared to FTPS or Secure FTP - is that no problems can arise with regard to FTP passive and active mode if the communication has to pass through firewalls or NAT routers.  In addition to the secure SFTP protocol, this service comes with AWS native access control features like IAM roles and policies, logical directories for S3 and Security Groups. 

The service allows you to create custom user accounts, there is no dependency on the IAM accounts. Each user needs to generate the SSH key pair and share only the public key with the SFTP server. But the access to S3 is completely governed by the IAM role & policy, which is the standard in accessing AWS resources.

Network:

There are several ways to deploy the SFTP server, including publicly accessible or VPC hosted. The VPC hosted option enables the SFTP server to access the service from the internal network (including on-prem datacenter) via endpoint ENIs in the respective VPC subnet. This means that communication is not routed over the public network, but instead over AWS VPC endpoint.

Since AWS Transfer Family supports hosted server endpoints in centrally managed as well as shared Amazon VPC environments, which will become important next when we get to our client-specific implementation.

Solution design implemented by T-Systems

Grafik

Our client here is from the enterprise segment and Telco domain currently undergoing a large-scale cloud transformation. The client has centrally maintained VPCs on AWS and uses the VPC sharing feature of AWS for making those VPCs accessible to project accounts. Also, Service Catalog is used as part of the landing zone of the client.

So, the idea was to offer a fully integrated data transfer area as a product via the service catalog, which consists of an AWS Transfer for SFTP server within the shared corporate VPC, a secured S3, the corresponding roles and a security group. 

The SFTP server receives IP addresses from the subnet through the corresponding VPC endpoint ENIs, which are connected through Direct Connect with the on-premises datacenter. The user connects via the corresponding IP address to the SFTP server and starts the data migration. The data is securely stored on an S3 Bucket which comes with features like versioning, lifecycle management, replication or event-based notification. Subsequently the data can be automatically processed and loaded into AWS native services like for example RDS or Glacier. 

Below figure shows the solution architecture. The whole solution is realized as a Cloudformation template (Infrastructure as Code), which is the most straightforward way to integrate a product into the service catalog. Maintenance of the solution is supported by a CI/CD pipeline.

Conclusion:

With the possibility to exchange data over known protocols like SFTP, migrations to AWS can be started more easily for many users. This simplifies migration scenarios and provides on-premise access to virtually unlimited cloud storage.

To make it even more user-friendly, the SFTP server can also be addressed using a custom DNS name, for this purpose all IP addresses of the SFTP server can be linked to a DNS A record in a private hosted zone.

For more information on how to set it up you can follow the AWS documentation.

Some additional use cases:

  • If you want to give other applications access to the data of your classic application, you can store the application data through traditional protocols in S3 and give access to other applications through modern channels like API gateway or Athena.
  • Let´s say you run out of storage space or your data center can´t keep up with the storage capacity requirements of your application, then you can take advantage of the almost unlimited capacity of S3 at low costs.
  • If you need to store your data for compliance reasons for longer periods of time you can upload them to S3 and automatically transition it to Glacier using a lifecycle rule.
  • AWS Transfer for SFTP uses SFTP protocol, which is a widely used protocol, so it's easier to adopt for traditional applications and platforms. And it comes with all the functionalities that users are familiar with, for example resumption of interrupted data transfer.
  • If you need a file-based interface from a legacy application on-prem to an AWS based application, you can set it up with very few efforts using this product. Further advantage: The availability of the SFTP server does not depend on the availability of any component of your AWS application.

Good to know:

AWS PrivateLink is now available for S3 as well and offers a good addition to AWS Transfer Family 

About the author
Profil von Artur Schneider, schwarz/weiß

Artur Schneider

Senior Cloud Consultant, T-Systems International

Show profile and articles
Do you visit t-systems.com outside of Germany? Visit the local website for more information and offers for your country.