Quick Links
If you want to allow servers in your network access to internal S3 buckets, without making the objects within them open to the internet, whitelisting access with a bucket policy is a simple solution to allow downloading files from an internal bucket.
Accessing an S3 Bucket Over the Internet
The most ideal method for interfacing with S3 from Linux is to just install the AWS CLI, and run commands like
get-object
to fetch files directly, or use the API or SDK for the language of your choice. If you're running on EC2, it's fairly trivial to update the IAM role for the EC2 instance, and attach a policy giving it access to the bucket. As long as the AWS CLI is installed, you can use it with the instance role without managing keys.
However, if you're not accessing S3 from EC2, or from another AWS resource that can assume a role, you'll have to manage keys. The AWS CLI will need an ID and secret to authenticate, which depending on the complexity of your deployment may be an issue. You might be tempted to hardcode it in your build script, but that's a messy solution that we don't advise. You can just set the bucket or object to public, but that won't work for sensitive data that you don't want others downloading.
The best option then is just whitelisting IP addresses. This manages access implicitly---if the request is coming from the IP address of your server, it will be allowed. This can be used to very easily allow downloading files from their endpoint URL, as if the bucket was running in a private subnet (though it's still going over the internet).
https://s3.amazonaws.com/bucketname/object.zip
Without a bucket policy in place though, you'll get a 403 error. Head over to the bucket settings, and find the bucket policy editor under Permissions > Bucket Policy.
Enter in the following policy, replacing the IP addresses and bucket name with your own:
{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::bucketname/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"192.168.1.1",
"192.168.1.2",
"192.168.1.3"
]
}
}
}
]
}
This just allows the downloading of files from the bucket (GetObject). If you want to enable other API actions, you can either set it to something specific with AWS's Bucket Policy generator, or allow every action on the bucket with a wildcard (probably a bad idea):
{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "IPAllow",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::bucketname/*",
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"192.168.1.1",
"192.168.1.2",
"192.168.1.3"
]
}
}
}
]
}