S3 Sync Helper: A Shell Script for Incremental S3 Bucket Syncs
I've released S3 Sync Helper, a new shell script to simplify data synchronization between S3 buckets. This script is intended for developers and ops professionals who need a straightforward way to manage S3 content.
The core functionality includes:
- Incremental Uploads: It only transfers files that are new or have changed, reducing transfer times and costs.
- Deletion Handling: The script can mirror deletions from the source bucket to the destination, keeping the two in sync.
- Prefix Filtering: An optional feature allows you to specify a prefix, so only objects within that path are considered for synchronization. This is useful for syncing specific subdirectories or datasets within a larger bucket.
This tool is built as a shell script, making it easy to integrate into existing CI/CD pipelines, cron jobs, or other automation workflows. It relies on standard AWS CLI commands, so if you have the AWS CLI configured, you can use this script.
It's designed to be a functional, no-frills utility for common S3 sync tasks.