AWS S3 Requirements

AWS S3 Overview

Amazon Simple Storage Service (S3) is a scalable storage service offered by AWS that provides object storage through a web service interface.

Setting up AWS S3

To set up AWS S3 for a self-hosted installation, do the following:

1. Set up the buckets using your own naming scheme (the names you choose can be configured in the Helm Chart values.yaml file). We suggest using a format such as: ${COMPANY_NAME}-dtplatform-${ENVIRONMENT_SHORTNAME}-${BUCKET_TYPE}-${CLOUD_REGION}
2. Ensure that the following buckets are set up:

• kafka - For the Kafka request/reply system (communication between services)
• filesvc - For the File Service storage
• scriptmanager - For the Script Manager/Worker log storage
• datasourcesvc - For the Datasource Service file storage
• workflowsvc - For the Workflow Service storage

3. For example, the bucket names for a company called “ACME” in an environment called prod1 in the us-west-2 region would be:

• acme-dtplatform-prod1-kafka-us-west-2
• acme-dtplatform-prod1-filesvc-us-west-2
• acme-dtplatform-prod1-scriptmanager-us-west-2
• acme-dtplatform-prod1-datasourcesvc-us-west-2
• acme-dtplatform-prod1-workflowsvc-us-west-2

4. Ensure that the following bucket security requirements are met:

• The buckets should perform server-side encryption.
• The buckets can enable versioning if required by customer policy. By design, no Platform-written objects are overwritten.
• The buckets should block all public access.
• The buckets' object ownership should be set to “bucket owner enforced”.
• The buckets should not allow access from other accounts except as required by customer security policy.
• The Cross-Origin Resource Sharing (CORS) policy should be broad unless restrictions are required by customer security policy. See the example below.

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 1728000
    }
]

• The FileService bucket needs a bucket policy that allows the CloudFront distribution to access objects. Refer to the AWS documentation for more information. See the sample policy below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudFrontCanGetObjects",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "${FILESVC_BUCKET_ARN}/*",
      "Condition": {
        "StringEquals": {
            "aws:SourceArn": "${CLOUDFRONT_DISTRO_ARN}"
        }
      }
    }
  ]
}