Quick Navigation:

How Can I Migrate Files from S3 to Another S3 Bucket?

 

Migrate 5 TB S3 bucket from one AWS account to another

Hello People, My team is working on migrating the S3 bucket from another AWS account to our Account. The size of the bucket is 5 TB and there are approx. 100 Million objects in the bucket.

As far as I have read, DataSync is the recommended approach to achieve our goal. But it comes with its own limitations like we cannot transfer more than 25 Million objects in a single task. 

Also based on a demo task we did, the time estimate for the task to finish would be around 30 hrs. Is there a better/faster way to do this? Please help me find options or suggestions to overcome the challenges. Thank you all for your Help!!

- Question from reddit.com

It’s a common need to move files between different AWS accounts especially when you have more than one team to manage. However, many people find that S3 to S3 data transfer speed is quite slow, and has limitations for objects to migrate while transferring large numbers of data from S3 to S3.

How to migrate files from Amazon S3 to another account easily and quickly? Today, we will show you the top effective ways to perform S3 to S3 data transfer.

Limitations of Amazon Built-in Features to Migrate from S3 to S3

Although you can use the below solutions to transfer data from S3 to S3:

  • Run parallel uploads using the AWS CLI
  • Use an AWS SDK
  • Use cross-Region replication or same-Region replication
  • Use Amazon S3 batch operations
  • Use S3DistCp with Amazon EMR
  • Use AWS DataSync

They have their shortcomings, and we will show you:

  • Run parallel uploads using the AWS CLI. An older version of AWS CLI can show you errors while migrating files from one S3 bucket to another S3 account. It requires you to use a command line to finish this job. It might be slow to move a large number of data from S3 to S3.
  • Use an AWS SDK. You have to build a custom application for this S3 to S3 data transfer.
  • Use cross-Region replication or same-Region replication. Only new objects can be replicated to the destination other than existing objects.
  • Use Amazon S3 batch operations. It’s designed for you to copy multiple objects instead of hundreds of millions of objects.
  • Use S3DistCp with Amazon EMR. Amazon EMR is a premium feature that you have to pay for.

Use AWS DataSync. AWS DataSync is quite expensive when you migrate a large number of files from S3 to S3 and needs the ability for coding and complex commands to complete the S3 to S3 file transfer before migrating, or it will fail.

Thus, Is there any easy free way to do S3 to S3 data transfer and do it quickly? Definitely, you can try a professional cloud data migration tool to transfer files between S3 buckets, even different accounts.

Best Free Way S3 to S3 Data Migration Tool - MultCloud

Here comes the powerful web-based multi-cloud storage manager - MultCloud to help. You can get the below benefits from this S3 data migration tool.

  • It is compatible with 30+ famous clouds all over the world, like Amazon S3, Dropbox, Dropbox Business, OneDrive, OneDrive for Business, Google Drive, Google Workspace, Box, Box for Business, Shared Drive, Shared with me, iCloud Drive, iCloud Photos, Wasabi, Backblaze, MEGA, MediaFire, MySQL, and many more.
  • Offers high-speed transfer between clouds, up to 10 threads so you can migrate a large number of objects from one S3 bucket to another S3 bucket quickly.
  • Migrate everything from one cloud to another directly without downloading.
  • Provides flexible settings, like Email, Filter, and Schedule to run S3 to S3 data transfer personalized.
  • You can perform S3 to S3 data migration in real time with Real Time Sync if you needed.
  • It’s very easy to configure and transfer compared with Amazon built-in functions.

AWS S3 to S3 Data Transfer in 3 Effective Ways

In fact, MultCloud provides you multiple ways to move files between Amazon S3 buckets, let’s find out one by one.

Way 1. AWS S3 to S3 Data Transfer via Cloud Transfer

To transfer everything from one S3 buckets to another bucket between AWS accounts, check the following steps with Cloud Transfer in MultCloud.

1. Please sign up for a MultCloud account, and sign in to MultCloud Dashboard.

2. Hit Add Cloud under My Online Services, then choose Amazon S3, then give your bucket name, access key ID, and secret Access Key to add to MultCloud. Repeat to add another AWS S3 account.

Add Amazon S3
Add Amazon S3

3. Select Cloud Transfer, tap FROM to choose an Amazon S3 bucket as the source, and hit TO to select a bucket in another Amazon S3 account.

Transfer Amazon S3 to Another S3
Transfer Amazon S3 to Another S3

Tips:

  • You can set up Transfer methods, Email and Filter in Options. If you want to free up space on Amazon S3, you can hit Options > Transfer > Delete all source files after transfer is complete to configure, it automatically deletes files from Amazon S3 once the S3 to S3 data transfer is completed. Use Filter to only transfer certain files you’d like to move.
  • Automatic transfer can be done with Schedule, enable it and you can run the task daily, weekly, monthly, even the time you specified.
  • Unlock to enjoy all the premium features, including Filter, Schedule, much more data traffic, fast transfer speed, and more.

4. Press Transfer Now to perform S3 to S3 data transfer quickly.

Way 2. Sync Files from S3 to Another S3 in Real Time

Team collaboration can be configured with Real Time Sync between Amazon S3 accounts for different teams. Thus, check the following steps to sync files from S3 to another S3 account in real time if required, and always keep them the same:

1. Please ensure you have added Amazon S3 accounts to MultCloud by clicking Add Cloud.

2. Hit Cloud Sync, then change to Real Time Sync, then choose one Amazon S3 account as the source, and another Amazon S3 account as the target.

Real Time Sync Between S3 Accounts
Real Time Sync Between S3 Accounts

3. Press the Sync Now button to sync files between Amazon S3 accounts in real time.

Tip: Except for Email and Filter you can configure in Options, you can also set up up to 10 sync modes, like two-way sync, one-way sync, move sync, mirror sync, cumulative sync, update sync, incremental sync, etc. Upgrade to enjoy all of them now.

Way 3. Copy S3 Bucket Objects across AWS Accounts via Copy to

Besides, you can use Copy to to copy S3 bucket objects across AWS accounts, check how it works:

1. After adding Amazon S3 accounts to MultCloud, tap the source Amazon S3 from the left side to open.

2. Choose a bucket from the source S3 account, and choose Copy to from the top menu.

3. Select a bucket in another S3 account in the pop-up window, and hit OK to copy files between S3 buckets.

Copy Files from Amazon S3 to Another Account
Copy Files from Amazon S3 to Another Account
  • Smart Share: Share cloud files via public, private, or source mode.
  • One-key Move: Quick data transfer, sync, and backup among clouds.
  • Email Archive: Protect emails to other clouds or local disks.
  • Remote upload: Effectively save web files to clouds via URL.
  • Safe: Full-time 256-bit AES and OAuth authorization protections.
  • Easy: Access and manage all clouds in one place with one login.
  • Extensive: 30+ clouds and on-premise services are supported.

The End

Now, S3 to S3 data transfer is a piece of cake for MultCloud, even hundreds of millions of objects to be migrated. Just find the way that suits you best to move files between Amazon S3 accounts with the detailed guide.

Moreover, you can also move files from Amazon S3 to Google Workspace, OneDrive for Business, or other clouds directly, share files publicly or privately, backup Dropbox to Google Drive, move files between business clouds with Team Transfer, and so on. So, why not give it a try?

MultCloud Logo

Manage 30+ popular clouds in one platform.

Sign up Free

100% secure

New Release

Save Email as PDF

After adding Gmail or Outlook to MultCloud, it can convert emails to PDF and save them on your device or cloud storage.

Get Started for Free