Tackle Anonymous Ownership in AWS S3 Bucket Object

Tackle Anonymous Ownership in AWS S3 Bucket Object

November 8, 2019 / Nirav Shah

When object uploaded to s3 from third-party media, sometimes the object ownership considered as an anonymous. It’s difficult to do any object level operations on it when the ownership is anonymous and consider you have a huge amount of objects like in TB, it’s very, very difficult to change anonymous permission one by one for all the objects, it’s near to impossible. So what’s the solution !

The Issue that we faced.

• The code used to crawl images from third party websites was uploading to the bucket under anonymous ownership which resulted in basic administrative functions like edit/delete fails.
• The size of the bucket was 1.5 TB approx with every individual image having the size of approx 150 KB.
• We wanted to secure the bucket & make it private but the anonymous ownership was not allowing us to make the bucket private as every image url would give a 403 Access Denied error if we make the bucket private.

The Solution

• We searched through the web for a solution but the senorio was too unique for anybody to provide general solution.
• Aws also didn’t have the proper solution as they suggested to delete all the existing images and reupload those.
• which was not applicable to our scenario as there was no way to reupload the deleted images.
• So, we had to come up with a custom script that would run on every individual image and change its ownership to aws account owner.
• Implementation
• We created a simple bash script that would take an object name from a list of object names in the provided file and execute 2 AWS-CLI commands.
• First Command
• It would change the ownership of a particular object that was in process from anonymous to AWS account owner.
• Second Command
• It would copy the above inprocess object into the same bucket to mimic the effect of reuploading to the bucket.
• We had to buy a spot instance T3A.2xlarge with 8 vCPU and 32 GB Ram and ran for at least a month with the below mentioned script.

Link to install AWS cli
https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html

Script to list and change ownership of objects in s3 bucket

First of all run a command that will output only the names of the object in a particular directory in s3 bucket and pipe its output into a file.

$ aws s3 ls s3://BUCKET_NAME/FOLDER_NAME/ | perl -pe 's/^(?:\S+\s+){3}//' >> FILE_NAME.txt

Now, we have a file(FILE_NAME.txt) which contains the list containing only names of the objects that are in a buckets’ sub-directory.

Now, we use that files’s list of names as an input for our script that will change the ownership.

$ vim script.sh

#!/bin/bash

input="/FILE_PATH/FILE_NAME.txt"

while IFS= read -r line

do

aws s3 ls s3://BUCKET_NAME/FOLDER_NAME/$line --recursive | awk
'{cmd="aws s3api put-object-acl --acl bucket-owner-full-control
--no-sign-request --bucket BUCKET_NAME --key "$4; system(cmd)}' &&\

aws s3 cp --acl bucket-owner-full-control
s3://BUCKET_NAME/FOLDER_NAME/$line
s3://BUCKET_NAME/FOLDER_NAME/$line --storage-class STANDARD

Save the above lines in a file and make it executable.

Create another file which will store logs of the above script.

$ touch task.log

Now, execute the command as follows.

$ nice -n 15 sh script.sh &>> task.log &\

> disown -h %1

The Conclusion

• We learnt AWS-CLI Commands to administer S3 buckets without GUI.
• We learnt how to make custom bucket policy to make it secure as well as not lose the required functionality.
• We learnt to use spot instance request and how spot instance lifecycle works.