Seamless Cross-Account, Cross-Region Replication of Encrypted Objects in AWS S3: Simplified Data Protection

Photo by Thomas Habr on Unsplash

In today’s digital landscape, data protection is paramount for organizations handling sensitive information. Amazon Simple Storage Service (S3) offers a robust solution for storing and managing data in the cloud. One of the powerful features provided by S3 is Cross-Region Replication, which allows for automatic and asynchronous replication of objects between different AWS regions. This feature enhances data protection, ensuring data durability, availability, and compliance with data residency requirements. Furthermore, S3 Cross-Account Replication extends the capabilities by enabling replication of objects between different AWS accounts, adding an extra layer of security and flexibility to the data protection strategy.

Scenario:

You are an IT professional working for a global organization that deals with sensitive customer data. Data availability, security, and compliance are of utmost importance. To address these requirements, your organization has implemented a robust data protection strategy utilizing Amazon S3 and Amazon Key Management Service (KMS). Your objective is to replicate encrypted objects from a source S3 bucket in one AWS account and region to a destination S3 bucket in a different AWS account and region.

In this scenario, for the successful replication of objects between the source and destination S3 buckets, the following requirements must be met:

1. Enable AWS Regions: The source bucket owner must have the source and destination AWS Regions enabled for their account. Similarly, the destination bucket owner must have the destination Region enabled for their account. Enabling or disabling an AWS Region can be managed using the AWS Management Console or APIs.
2. Enable Versioning: Both the source and destination buckets must have versioning enabled. Versioning keeps track of multiple versions of an object in the bucket, ensuring data integrity and facilitating the replication process. You can enable versioning for a bucket through the AWS Management Console or via the AWS CLI/APIs.
3. Set Up Replication Permissions: Amazon S3 must have the necessary permissions to replicate objects from the source bucket to the destination bucket(s) on your behalf. These permissions involve configuring appropriate access policies and roles. More information on setting up permissions can be found in the AWS documentation.
4. Set Up Bucket Policy Permissions: The owner of the destination buckets must grant the owner of the source bucket the necessary permissions to replicate objects. This can be achieved by configuring a bucket policy that allows the source bucket owner to replicate objects to the destination bucket.
5. Configure KMS Key Permissions: If you intend to use an AWS Key Management Service (KMS) key from the destination account to encrypt the replicated objects, the destination account must grant the replication role permissions in the KMS key policy. Specifically, the permission required is kms:Encrypt.

Solution:

The replication process can be summarized into the following diagram:

Here’s an outline for the steps involved in setting up the KMS, IAM role, S3 buckets, and replication:

1. Set up the KMS for the source bucket
2. Set up the KMS the destination bucket
3. Generate the IAM role for the source bucket replication
4. Establish the KMS key policy in the destination account
5. Create the source S3 bucket
6. Create the destination S3 bucket
7. Configure the bucket policy for the destination bucket
8. Configure the replication for the source S3 bucket

By following this outline, I will provide a comprehensive and structured guide for you to understand the process of setting up and configuring the necessary components for replicating encrypted objects between AWS accounts and regions.

Let’s start by defining and setting up some global variables that will be used throughout this demonstration.

Source Bucket Definition

src_bkt_region="us-east-1"
src_bkt="source-bucket-replication-2023"
src_replication_role_name="bucket-replication-role"
src_replication_role_policy="replication"
src_account="0*********"

Destination Bucket Definition

dest_bkt_region="us-east-2"
dest_bkt="destination-bucket-replication-2023"
dest_account="8********"

These variables will be used throughout the demonstration to reference the specific AWS accounts, regions, and bucket names involved in the replication process.

When performing operations on the source bucket, we will use the src profile, which is associated with the necessary credentials and permissions for the source AWS account. Similarly, for operations on the destination bucket, we will use the dest profile, which is associated with the appropriate credentials and permissions for the destination AWS account.

Once all the prerequisites have been prepared, I will meticulously delineate the commands utilized throughout the process.

1. Create KMS for server-side encryption of the source bucket

1–1: Create the source bucket’s KMS Key

Use the command aws kms create-key to generate a new KMS key. This command will return the Key ID of the newly created key and use the command aws kms describe-key to retrieve the ARN (Amazon Resource Name) of the created key by providing the Key ID.

src_key_id=$(aws kms create-key --query 'KeyMetadata.KeyId'  --output text --profile src)
​
src_key_arn=$(aws kms describe-key --key-id $src_key_id --query 'KeyMetadata.Arn'  --output text --profile src)

By executing these commands, you will successfully create the KMS key and obtain its ARN for further use in the replication process.

Make sure to replace src in the --profile option with the appropriate AWS CLI profile if you are using a named profile.

1–2: Add tags to KMS key of the source bucket

To add tags to the source bucket’s KMS key, you can use the following command:

aws kms tag-resource --key-id $src_key_id --tags TagKey=Name,TagValue=source_data --profile src

This command associates tags with the KMS key identified by $src_key_id. In this example, we are adding a tag with the key Name and the value source_data. You can modify the tag key and value as per your requirements.

1–3: Enable the key rotation period for the source bucket’s KMS key

Enabling key rotation ensures that the KMS key automatically generates a new cryptographic key at regular intervals. This practice enhances the security of the key and helps protect the encrypted data within the source bucket.

To enable the key rotation period for the source bucket’s KMS key, you can use the following command:

aws kms enable-key-rotation --key-id $src_key_id --profile src

This command enables automatic key rotation for the KMS key identified by $src_key_id.

1–4: Associate alias to the source bucket’s KMS

To associate an alias with the source bucket’s KMS key, you can use the following command:

aws kms create-alias \
  --alias-name "alias/source" \
  --target-key-id $src_key_id --profile src

This command creates an alias with the name “alias/source” and associates it with the KMS key identified by $src_key_id. The alias provides a more user-friendly and memorable name to reference the KMS key.

2. Create KMS for server-side encryption of the destination bucket

The steps to create and configure the KMS key for server-side encryption of the destination bucket are similar to the ones in source bucket.

2–1: Create the destination bucket’s KMS key

dest_key_id=$(aws kms create-key --query 'KeyMetadata.KeyId'  --output text --profile dest)
​
dest_key_arn=$(aws kms describe-key --key-id $dest_key_id --query 'KeyMetadata.Arn'  --output text --profile dest)

These commands create a new KMS key for the destination bucket and retrieve its Key ID (dest_key_id) and ARN (dest_key_arn) for further use.

2-2: Add Tags to the destination bucket's KMS key

aws kms tag-resource --key-id $dest_key_id --tags TagKey=Name,TagValue=destination_data --profile dest

This command associates tags with the KMS key associated with the destination bucket. You can specify the desired key-value pairs for the tags.

2-3: Set the key rotation period for the destination bucket's KMS key

aws kms enable-key-rotation --key-id $dest_key_id --profile dest

This command enables automatic key rotation for the KMS key associated with the destination bucket. Automatic key rotation periodically generates a new cryptographic key, enhancing the security of the key.

2-4: Create an alias for the destination bucket's KMS key

aws --profile dest kms create-alias --alias-name alias/destination --target-key-id $dest_key_id

This command creates an alias, alias/destination, and associates it with the KMS key of the destination bucket. The alias provides a more user-friendly and memorable name to refer to the KMS key.

3. Generate IAM role for the replication of the source bucket

3-1: Create a role

Create the role using the aws iam create-role command. This command creates an IAM role with a specified name, description, and assume role policy document. The assume role policy document defines the trusted entity that can assume the role. In this case, the trusted entity is the Amazon S3 service.

aws iam create-role --role-name $src_replication_role_name --description "Allow S3 to assume the role for replication" --assume-role-policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "s3ReplicationAssume",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}' --profile src

Retrieve the ARN of the created IAM role using the aws iam get-role command. This command retrieves information about the specified IAM role, including the ARN.

src_replication_role_arn=$(aws iam get-role --role-name "$src_replication_role_name" --query 'Role.Arn' --output text --profile src )

3-2: Create a policy

Create a policy that allows the IAM role to perform the necessary actions for S3 replication.

aws iam create-policy \
  --policy-name $src_replication_role_policy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": [
          "s3:GetReplicationConfiguration",
          "s3:ListBucket"
        ],
        "Effect": "Allow",
        "Resource": [
          "arn:aws:s3:::'${src_bkt}'"
        ]
      },
      {
        "Action": [
          "s3:GetObjectVersion",
          "s3:GetObjectVersionForReplication",
          "s3:GetObjectVersionAcl"
        ],
        "Effect": "Allow",
        "Resource": [
          "arn:aws:s3:::'${src_bkt}'/*"
        ]
      },
      {
        "Action": [
          "s3:ReplicateObject",
          "s3:ReplicateDelete"
        ],
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::'${dest_bkt}'/*"
      },
      { 
        "Action": [
          "kms:GenerateDataKey",
          "kms:Decrypt"
        ],
        "Effect": "Allow",
        "Condition": {
          "StringLike": {
            "kms:ViaService": "s3.'${src_bkt_region}'.amazonaws.com",
            "kms:EncryptionContext:aws:s3:arn": [
              "arn:aws:s3:::'${src_bkt}'/*"
            ]
          }
        },
        "Resource": [
          "'${src_key_arn}'"
        ]
      },
      { 
        "Action": [
          "kms:GenerateDataKey",
          "kms:Encrypt"
        ],
        "Effect": "Allow",
        "Condition": {
          "StringLike": {
            "kms:ViaService": "s3.'${dest_bkt_region}'.amazonaws.com",
            "kms:EncryptionContext:aws:s3:arn": [
              "arn:aws:s3:::'${dest_bkt}'/*"
            ]
          }
        },
        "Resource": [
          "'${dest_key_arn}'"
        ]
      }
    ]
  }' --profile src
​

3-3: Attach the policy to the role

Use the aws iam attach-role-policy command to attach the policy to the IAM role. Specify the ARN of the policy and the name of the IAM role.

aws iam attach-role-policy \
  --role-name  $src_replication_role_name \
  --policy-arn arn:aws:iam::$src_account:policy/$src_replication_role_policy --profile src

4. Establish the KMS key policy in the destination account

After we create the IAM role for the replication, the next step is to configure the KMS key policy for the destination bucket. This policy allows the IAM role associated with the source bucket to perform encryption actions using the destination bucket's KMS key.

aws --profile dest kms put-key-policy --key-id $dest_key_id --policy-name default --policy '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::'${dest_account}':root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Enable cross account encrypt access for S3 Cross Region Replication",
      "Effect": "Allow",
      "Principal": {
        "AWS": "'${src_replication_role_arn}'"
      },
      "Action": [
        "kms:Encrypt"
      ],
      "Resource": "*"
    }
  ]
}' --profile dest

This command sets the KMS key policy for the destination bucket's KMS key. The policy allows the root user of the destination AWS account (arn:aws:iam::'${dest_account}':root) to perform all KMS actions. Additionally, it enables cross-account encryption access for S3 Cross Region Replication, granting permission to the IAM role ('${src_replication_role_arn}') associated with the source bucket in the source AWS account.

5. Create the source S3 bucket

5-1: Create the S3 bucket

Let's create the source S3 bucket. The following command is used for creating the bucket:

# for s3 in  us-east-1 don't need locationConstraint
  aws s3api create-bucket \
  --bucket $src_bkt \
  --acl private --profile src

Make sure to retrieve the S3 bucket ARN using the following format:

arn:aws:s3:::<bucket-name>

5-2: Enable the versioning for the S3 bucket

To enable the versioning for the S3 bucket, execute the following command:

aws s3api put-bucket-versioning \
  --bucket $src_bkt \
  --versioning-configuration Status=Enabled --profile src

5-3: Configure server-side encryption for the S3 bucket

aws s3api put-bucket-encryption \
  --bucket  $src_bkt \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "'${src_key_arn}'"
        }
      }
    ]
  }' --profile src

By running this code, you are configuring server-side encryption for the source S3 bucket using AWS Key Management Service (KMS) for encryption, and associating it with the specified KMS master key.

6. Create the destination S3 bucket

6-1: create the S3 bucket

Let's proceed with the creation of the destination S3 bucket. Use the following command:

aws s3api create-bucket \
  --bucket $dest_bkt \
  --acl private \
  --create-bucket-configuration LocationConstraint=$dest_bkt_region \
  --profile dest

6-2: Enable versioning on the S3 bucket

To enable the versioning for the destination S3 bucket, execute the following command:

aws s3api put-bucket-versioning \
  --bucket $dest_bkt \
  --versioning-configuration Status=Enabled \
  --profile dest

6-3: Configure server-side encryption for the S3 bucket:

Next, let's configure server-side encryption for the destination S3 bucket. Use the following command:

aws s3api put-bucket-encryption \
  --bucket  $dest_bkt \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "'${dest_key_arn}'"
        }
      }
    ]
  }' --profile dest

7. Set the bucket policy to allow replication from the source account

To enable replication from the source account to the destination S3 bucket, we need to set up a bucket policy of the destination bucket. Use the following command:

aws s3api put-bucket-policy --bucket $dest_bkt --policy '{
  "Version": "2008-10-17",
  "Id": "",
  "Statement": [
    {
      "Sid": "AllowReplication",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::'${src_account}':root"
      },
      "Action": [
        "s3:GetBucketVersioning",
        "s3:PutBucketVersioning",
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ObjectOwnerOverrideToBucketOwner"
      ],
      "Resource": [
        "arn:aws:s3:::'${dest_bkt}'",
        "arn:aws:s3:::'${dest_bkt}'/*"
      ]
    },
    {
      "Sid": "AllowRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::'${src_account}':root"
      },
      "Action": [
        "s3:List*",
        "s3:Get*"
      ],
      "Resource": [
        "arn:aws:s3:::'${dest_bkt}'",
        "arn:aws:s3:::'${dest_bkt}'/*"
      ]
    }
  ]
}
' --profile dest

This command sets up a bucket policy with two statements. The first statement, with the AllowReplication SID, allows the source account to perform replication-related actions such as getting and putting bucket versioning, replicating objects, replicating deletions, and overriding object ownership to the bucket owner. The resource field specifies the ARN of the destination bucket and its objects.

The second statement, with the AllowRead SID, grants the source account permission to list and get objects from the destination bucket. This allows the source account to access the destination bucket for replication purposes.

8. Set up the replication configuration for the source S3 bucket

To configure replication for the source S3 bucket, execute the following command:

aws s3api put-bucket-replication \
  --bucket $src_bkt \
  --replication-configuration '{
    "Role": "'${src_replication_role_arn}'",
    "Rules": [
      {
        "ID": "ReplicationRule",
        "Status": "Enabled",
        "Prefix": "",
        "Destination": {
          "Bucket": "arn:aws:s3:::'${dest_bkt}'",
          "EncryptionConfiguration": {
            "ReplicaKmsKeyID": "'${dest_key_arn}'"
          }
        },
        "SourceSelectionCriteria": {
          "SseKmsEncryptedObjects": {
            "Status": "Enabled"
          }
        }
      }
    ]
  }' --profile src
​

In this command, the Role field specifies the ARN of the IAM role (src_replication_role_arn) that grants permissions for the replication process to access the source bucket and replicate objects to the destination bucket. Ensure that the IAM role has the necessary permissions to read the source bucket, replicate objects, delete objects, and apply encryption settings.

Testing time

Once you have set up live replication for both the source and destination buckets, you can proceed to test the configuration by uploading a file, and ensure that it is replicated to the destination bucket as per your configured replication settings.

Final thought

By following the step-by-step process outlined in this blog, you can confidently set up live replication, enabling you to maintain data consistency, improve disaster recovery readiness, and meet compliance requirements.

After implementing this replication strategy, you can enhance your data protection and resilience, ensuring that your valuable assets are securely stored and accessible across multiple AWS accounts and regions. Leverage the power of AWS S3's replication capabilities to take control of your data and achieve peace of mind in today's dynamic and interconnected digital landscape.