Skip to main content
Version: v4.5

AWS S3 Requirements

AWS S3 Overview#

Amazon Simple Storage Service (S3) is a scalable storage service offered by AWS that provides object storage through a web service interface.

Setting up AWS S3 for SMI#

To set up AWS S3 for SMI, do the following:

  1. Set up four buckets using your own naming scheme (the names you choose can be configured in the Helm Chart file values.yaml). We suggest using a format such as: ${COMPANY_NAME}-dtplatform-${ENVIRONMENT_SHORTNAME}-${BUCKET_TYPE}-${CLOUD_REGION}

  2. Ensure that the required four buckets are set up as follows:

  • kafka - For the Kafka request reply system (communication between services)
  • filesvc - For the File Service storage
  • scriptmanager - For the Script Manager/Worker log storage
  • datasourcesvc - For the Datasources Service file storage

For example, bucket names for a company called “ACME” in an an environment called prod1 in the us-west-2 region, would be as follows:

  • acme-dtplatform-prod1-kafka-us-west-2
  • acme-dtplatform-prod1-filesvc-us-west-2
  • acme-dtplatform-prod1-scriptmanager-us-west-2
  • acme-dtplatform-prod1-datasourcesvc-us-west-2
  1. Ensure that the following bucket security requirements are met:
  • The buckets should perform server-side encryption.
  • The buckets can enable versioning if required by a customer policy. However, no custom data files are overwritten by design.
  • The buckets should block all public access.
  • The buckets object ownership should have “bucket owner enforced”.
  • The bucket should not allow another account access except as required by the customer security policy.
  • The Cross-Origin Resource Sharing (CORS) policy should be broad unless restrictions are required by the customer’s security policy. See example below.
[    {        "AllowedHeaders": [            "*"        ],        "AllowedMethods": [            "GET"        ],        "AllowedOrigins": [            "*"        ],        "ExposeHeaders": [],        "MaxAgeSeconds": 1728000    }]

  • The FileService bucket will have a bucket policy to allow the CloudFront Distribution to access objects. Refer to AWS documentation for more information. See sample script below. 

    
{  "Version": "2012-10-17",  "Statement": [      {          "Sid": "CloudFrontCanGetObjects",          "Effect": "Allow",          "Principal": {              "Service": "cloudfront.amazonaws.com"          },          "Action": "s3:GetObject",          "Resource": "${FILESVC_BUCKET_ARN}/*",          "Condition": {              "StringEquals": {                  "aws:SourceArn": "${CLOUDFRONT_DISTRO_ARN}"              }          }      }  ]}